[ovirt-users] qemu cgroup_controllers

[Martin Polednik]

On 03/06/16 14:05 +0300, Дмитрий Глушенок wrote:
>Thank you Martin!
>
>Actually I tried the workaround hook, provided in [2], but then VDSM (oVirt 3.6.6) tries to interpret hostdev in XML as PCI device, which leads to:
>
>::The vm start process failed
>Traceback (most recent call last):
>  File "/usr/share/vdsm/virt/vm.py", line 703, in _startUnderlyingVm
>    self._run()
>  File "/usr/share/vdsm/virt/vm.py", line 1949, in _run
>    self._domDependentInit()
>  File "/usr/share/vdsm/virt/vm.py", line 1797, in _domDependentInit
>    self._getUnderlyingVmDevicesInfo()
>  File "/usr/share/vdsm/virt/vm.py", line 1738, in _getUnderlyingVmDevicesInfo
>    self._getUnderlyingHostDeviceInfo()
>  File "/usr/share/vdsm/virt/vm.py", line 4277, in _getUnderlyingHostDeviceInfo
>    **self._getUnderlyingDeviceAddress(source))
>TypeError: pci_address_to_name() got an unexpected keyword argument 'target'
>
>XML part was:
><hostdev managed="no" mode="subsystem" rawio="yes" type="scsi">
>	<source>
>		<adapter name="scsi_host2"/>
>		<address bus="0" target="1" unit="0"/>
>	</source>
></hostdev>
><hostdev managed="no" mode="subsystem" rawio="yes" type="scsi">
>	<source>
>		<adapter name="scsi_host2"/>
>		<address bus="0" target="2" unit="0"/>
>	</source>
></hostdev>
>
>
>As of creating custom partition - by default machine.slice has "a *:* rwm" in devices.list. But for every new VM libvirt removes *:* mask and fills the list with actually needed devices (as I understand the process). For example:
>
>c 136:* rw
>c 1:3 rw
>c 1:7 rw
>c 1:5 rw
>c 1:8 rw
>c 1:9 rw
>c 5:2 rw
>c 10:232 rw
>c 253:0 rw
>c 10:228 rw
>c 10:196 rw
>
>What I'm looking for is a way to tell libvirt about my additional devices without breaking oVirt.

The solution would be creating your own partition and somehow (e.g.
VDSM hook) appending

  <resource>
    <partition>/machine/custom</partition>
  </resource>

to the libvirt's <domain> element. I'm not sure how feasible creating
your own partition is though. I've tried the process as follows:

$ cat ~/create-partition.sh
# sh ~/create-partition.sh
for i in blkio cpu,cpuacct cpuset devices freezer memory net_cls perf_event
do
    mkdir /sys/fs/cgroup/$i/machine.slice/custom.partition
done

for i in cpuset.cpus cpuset.mems
do
    cat /sys/fs/cgroup/cpuset/machine.slice/$i > /sys/fs/cgroup/cpuset/machine.slice/custom.partition/$i
done

(creates /machine/custom partition).
Now, we can create vdsm before_vm_start hook that will set given partition
for the VM:

$ pwd /usr/libexec/vdsm/hooks/before_vm_start
$ cat 10_cgroups

#!/usr/bin/python

import hooking


def custom_partition(domxml):
    resource = domxml.createElement('resource')
    partition = domxml.createElement('partition')
    partition_text = domxml.createTextNode('/machine/custom')
    partition.appendChild(partition_text)
    resource.appendChild(partition)

    return resource

domxml = hooking.read_domxml()
domain = domxml.getElementsByTagName('domain')[0]
domain.appendChild(custom_partition(domxml))
hooking.write_domxml(domxml)

>--
>Dmitry Glushenok
>Jet Infosystems
>http://www.jet.msk.su
>+7-495-411-7601 (ext. 1237)
>
>> 3 июня 2016 г., в 12:24, Martin Polednik <mpolednik at redhat.com> написал(а):
>>
>> On 03/06/16 11:48 +0300, Дмитрий Глушенок wrote:
>>> Hello!
>>>
>>> Is it possible to tell libvirt to add specific devices to qemu cgroup? By somehow enumerating the devices in XML using a hook for example.
>>> I'm passing scsi-generic disks (/dev/sgX) to VM using qemucmdline hook and it doesn't work until I remove "devices" from cgroup_controllers in qemu.conf.
>>
>> One way to achieve this is creating a hook to generate the scsi device
>> XML instead of modifying qemu cmdline directly. Libvirt assumes
>> ownership of all devices created in the XML and therefore adds them to
>> the machine cgroup.
>>
>> Example of the XML taken from [1]:
>> <devices>
>>   <hostdev mode='subsystem' type='scsi' sgio='filtered' rawio='yes'>
>>       <source>
>>           <adapter name='scsi_host0'/>
>>           <address bus='0' target='0' unit='0'/>
>>       </source>
>>       <readonly/>
>>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>   </hostdev>
>> </devices>
>>
>> There is slight issue with this approach outlined in [2].
>>
>> If you want to keep the qemu approach, I think creating a custom
>> partition and moving devices there would be the cleanest approach. In
>> this case, [3] could help but I'm not entirely sure if that would
>> solve the issue.
>>
>> [1] https://libvirt.org/formatdomain.html
>> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1325485
>> [3] https://libvirt.org/cgroups.html
>>> --
>>> Dmitry Glushenok
>>> Jet Infosystems
>>> http://www.jet.msk.su
>>> +7-495-411-7601 (ext. 1237)
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>