[ovirt-users] qemu cgroup_controllers

Mon Jun 6 10:35:39 UTC 2016

On 06/06/16 13:26 +0300, Дмитрий Глушенок wrote:
>Hello Martin,
>
>Thank you for your time. It is clear how to create partition and assign it to a VM. But libvirt manipulates with contents of devices.list and I didn't find a way to put my devices into devices.list and stop libvirt from removing them.

That is, by the way, pretty interesting. I have tried it right now and
when using custom partition, libvirt doesn't touch my devices.list.

It might make sense to file a libvirt bug.

>Anyway, it looks like that the best solution will be using oVirt 4.0 where scsi pass-through have been fixed.

Hoping so!

>--
>Dmitry Glushenok
>Jet Infosystems
>http://www.jet.msk.su <http://www.jet.msk.su/>+7-495-411-7601 (ext. 1237)
>
>> 6 июня 2016 г., в 12:33, Martin Polednik <mpolednik at redhat.com> написал(а):
>>
>> On 03/06/16 14:05 +0300, Дмитрий Глушенок wrote:
>>> Thank you Martin!
>>>
>>> Actually I tried the workaround hook, provided in [2], but then VDSM (oVirt 3.6.6) tries to interpret hostdev in XML as PCI device, which leads to:
>>>
>>> ::The vm start process failed
>>> Traceback (most recent call last):
>>> File "/usr/share/vdsm/virt/vm.py", line 703, in _startUnderlyingVm
>>>   self._run()
>>> File "/usr/share/vdsm/virt/vm.py", line 1949, in _run
>>>   self._domDependentInit()
>>> File "/usr/share/vdsm/virt/vm.py", line 1797, in _domDependentInit
>>>   self._getUnderlyingVmDevicesInfo()
>>> File "/usr/share/vdsm/virt/vm.py", line 1738, in _getUnderlyingVmDevicesInfo
>>>   self._getUnderlyingHostDeviceInfo()
>>> File "/usr/share/vdsm/virt/vm.py", line 4277, in _getUnderlyingHostDeviceInfo
>>>   **self._getUnderlyingDeviceAddress(source))
>>> TypeError: pci_address_to_name() got an unexpected keyword argument 'target'
>>>
>>> XML part was:
>>> <hostdev managed="no" mode="subsystem" rawio="yes" type="scsi">
>>> 	<source>
>>> 		<adapter name="scsi_host2"/>
>>> 		<address bus="0" target="1" unit="0"/>
>>> 	</source>
>>> </hostdev>
>>> <hostdev managed="no" mode="subsystem" rawio="yes" type="scsi">
>>> 	<source>
>>> 		<adapter name="scsi_host2"/>
>>> 		<address bus="0" target="2" unit="0"/>
>>> 	</source>
>>> </hostdev>
>>>
>>>
>>> As of creating custom partition - by default machine.slice has "a *:* rwm" in devices.list. But for every new VM libvirt removes *:* mask and fills the list with actually needed devices (as I understand the process). For example:
>>>
>>> c 136:* rw
>>> c 1:3 rw
>>> c 1:7 rw
>>> c 1:5 rw
>>> c 1:8 rw
>>> c 1:9 rw
>>> c 5:2 rw
>>> c 10:232 rw
>>> c 253:0 rw
>>> c 10:228 rw
>>> c 10:196 rw
>>>
>>> What I'm looking for is a way to tell libvirt about my additional devices without breaking oVirt.
>>
>> The solution would be creating your own partition and somehow (e.g.
>> VDSM hook) appending
>>
>> <resource>
>>   <partition>/machine/custom</partition>
>> </resource>
>>
>> to the libvirt's <domain> element. I'm not sure how feasible creating
>> your own partition is though. I've tried the process as follows:
>>
>> $ cat ~/create-partition.sh
>> # sh ~/create-partition.sh
>> for i in blkio cpu,cpuacct cpuset devices freezer memory net_cls perf_event
>> do
>>   mkdir /sys/fs/cgroup/$i/machine.slice/custom.partition
>> done
>>
>> for i in cpuset.cpus cpuset.mems
>> do
>>   cat /sys/fs/cgroup/cpuset/machine.slice/$i > /sys/fs/cgroup/cpuset/machine.slice/custom.partition/$i
>> done
>>
>> (creates /machine/custom partition).
>> Now, we can create vdsm before_vm_start hook that will set given partition
>> for the VM:
>>
>> $ pwd /usr/libexec/vdsm/hooks/before_vm_start
>> $ cat 10_cgroups
>>
>> #!/usr/bin/python
>>
>> import hooking
>>
>>
>> def custom_partition(domxml):
>>   resource = domxml.createElement('resource')
>>   partition = domxml.createElement('partition')
>>   partition_text = domxml.createTextNode('/machine/custom')
>>   partition.appendChild(partition_text)
>>   resource.appendChild(partition)
>>
>>   return resource
>>
>> domxml = hooking.read_domxml()
>> domain = domxml.getElementsByTagName('domain')[0]
>> domain.appendChild(custom_partition(domxml))
>> hooking.write_domxml(domxml)
>>
>>> --
>>> Dmitry Glushenok
>>> Jet Infosystems
>>> http://www.jet.msk.su
>>> +7-495-411-7601 (ext. 1237)
>>>
>>>> 3 июня 2016 г., в 12:24, Martin Polednik <mpolednik at redhat.com> написал(а):
>>>>
>>>> On 03/06/16 11:48 +0300, Дмитрий Глушенок wrote:
>>>>> Hello!
>>>>>
>>>>> Is it possible to tell libvirt to add specific devices to qemu cgroup? By somehow enumerating the devices in XML using a hook for example.
>>>>> I'm passing scsi-generic disks (/dev/sgX) to VM using qemucmdline hook and it doesn't work until I remove "devices" from cgroup_controllers in qemu.conf.
>>>>
>>>> One way to achieve this is creating a hook to generate the scsi device
>>>> XML instead of modifying qemu cmdline directly. Libvirt assumes
>>>> ownership of all devices created in the XML and therefore adds them to
>>>> the machine cgroup.
>>>>
>>>> Example of the XML taken from [1]:
>>>> <devices>
>>>>  <hostdev mode='subsystem' type='scsi' sgio='filtered' rawio='yes'>
>>>>      <source>
>>>>          <adapter name='scsi_host0'/>
>>>>          <address bus='0' target='0' unit='0'/>
>>>>      </source>
>>>>      <readonly/>
>>>>      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>>>  </hostdev>
>>>> </devices>
>>>>
>>>> There is slight issue with this approach outlined in [2].
>>>>
>>>> If you want to keep the qemu approach, I think creating a custom
>>>> partition and moving devices there would be the cleanest approach. In
>>>> this case, [3] could help but I'm not entirely sure if that would
>>>> solve the issue.
>>>>
>>>> [1] https://libvirt.org/formatdomain.html
>>>> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1325485
>>>> [3] https://libvirt.org/cgroups.html
>>>>> --
>>>>> Dmitry Glushenok
>>>>> Jet Infosystems
>>>>> http://www.jet.msk.su
>>>>> +7-495-411-7601 (ext. 1237)
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/users
>