qemu cgroup_controllers
Hello! Is it possible to tell libvirt to add specific devices to qemu cgroup? By somehow enumerating the devices in XML using a hook for example. I'm passing scsi-generic disks (/dev/sgX) to VM using qemucmdline hook and it doesn't work until I remove "devices" from cgroup_controllers in qemu.conf. -- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su +7-495-411-7601 (ext. 1237)
On 03/06/16 11:48 +0300, Дмитрий Глушенок wrote:
Hello!
Is it possible to tell libvirt to add specific devices to qemu cgroup? By somehow enumerating the devices in XML using a hook for example. I'm passing scsi-generic disks (/dev/sgX) to VM using qemucmdline hook and it doesn't work until I remove "devices" from cgroup_controllers in qemu.conf.
One way to achieve this is creating a hook to generate the scsi device XML instead of modifying qemu cmdline directly. Libvirt assumes ownership of all devices created in the XML and therefore adds them to the machine cgroup. Example of the XML taken from [1]: <devices> <hostdev mode='subsystem' type='scsi' sgio='filtered' rawio='yes'> <source> <adapter name='scsi_host0'/> <address bus='0' target='0' unit='0'/> </source> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </hostdev> </devices> There is slight issue with this approach outlined in [2]. If you want to keep the qemu approach, I think creating a custom partition and moving devices there would be the cleanest approach. In this case, [3] could help but I'm not entirely sure if that would solve the issue. [1] https://libvirt.org/formatdomain.html [2] https://bugzilla.redhat.com/show_bug.cgi?id=1325485 [3] https://libvirt.org/cgroups.html
-- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su +7-495-411-7601 (ext. 1237)
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Thank you Martin! Actually I tried the workaround hook, provided in [2], but then VDSM (oVirt 3.6.6) tries to interpret hostdev in XML as PCI device, which leads to: ::The vm start process failed Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 703, in _startUnderlyingVm self._run() File "/usr/share/vdsm/virt/vm.py", line 1949, in _run self._domDependentInit() File "/usr/share/vdsm/virt/vm.py", line 1797, in _domDependentInit self._getUnderlyingVmDevicesInfo() File "/usr/share/vdsm/virt/vm.py", line 1738, in _getUnderlyingVmDevicesInfo self._getUnderlyingHostDeviceInfo() File "/usr/share/vdsm/virt/vm.py", line 4277, in _getUnderlyingHostDeviceInfo **self._getUnderlyingDeviceAddress(source)) TypeError: pci_address_to_name() got an unexpected keyword argument 'target' XML part was: <hostdev managed="no" mode="subsystem" rawio="yes" type="scsi"> <source> <adapter name="scsi_host2"/> <address bus="0" target="1" unit="0"/> </source> </hostdev> <hostdev managed="no" mode="subsystem" rawio="yes" type="scsi"> <source> <adapter name="scsi_host2"/> <address bus="0" target="2" unit="0"/> </source> </hostdev> As of creating custom partition - by default machine.slice has "a *:* rwm" in devices.list. But for every new VM libvirt removes *:* mask and fills the list with actually needed devices (as I understand the process). For example: c 136:* rw c 1:3 rw c 1:7 rw c 1:5 rw c 1:8 rw c 1:9 rw c 5:2 rw c 10:232 rw c 253:0 rw c 10:228 rw c 10:196 rw What I'm looking for is a way to tell libvirt about my additional devices without breaking oVirt. -- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su +7-495-411-7601 (ext. 1237)
3 июня 2016 г., в 12:24, Martin Polednik <mpolednik@redhat.com> написал(а):
On 03/06/16 11:48 +0300, Дмитрий Глушенок wrote:
Hello!
Is it possible to tell libvirt to add specific devices to qemu cgroup? By somehow enumerating the devices in XML using a hook for example. I'm passing scsi-generic disks (/dev/sgX) to VM using qemucmdline hook and it doesn't work until I remove "devices" from cgroup_controllers in qemu.conf.
One way to achieve this is creating a hook to generate the scsi device XML instead of modifying qemu cmdline directly. Libvirt assumes ownership of all devices created in the XML and therefore adds them to the machine cgroup.
Example of the XML taken from [1]: <devices> <hostdev mode='subsystem' type='scsi' sgio='filtered' rawio='yes'> <source> <adapter name='scsi_host0'/> <address bus='0' target='0' unit='0'/> </source> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </hostdev> </devices>
There is slight issue with this approach outlined in [2].
If you want to keep the qemu approach, I think creating a custom partition and moving devices there would be the cleanest approach. In this case, [3] could help but I'm not entirely sure if that would solve the issue.
[1] https://libvirt.org/formatdomain.html [2] https://bugzilla.redhat.com/show_bug.cgi?id=1325485 [3] https://libvirt.org/cgroups.html
-- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su +7-495-411-7601 (ext. 1237)
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On 03/06/16 14:05 +0300, Дмитрий Глушенок wrote:
Thank you Martin!
Actually I tried the workaround hook, provided in [2], but then VDSM (oVirt 3.6.6) tries to interpret hostdev in XML as PCI device, which leads to:
::The vm start process failed Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 703, in _startUnderlyingVm self._run() File "/usr/share/vdsm/virt/vm.py", line 1949, in _run self._domDependentInit() File "/usr/share/vdsm/virt/vm.py", line 1797, in _domDependentInit self._getUnderlyingVmDevicesInfo() File "/usr/share/vdsm/virt/vm.py", line 1738, in _getUnderlyingVmDevicesInfo self._getUnderlyingHostDeviceInfo() File "/usr/share/vdsm/virt/vm.py", line 4277, in _getUnderlyingHostDeviceInfo **self._getUnderlyingDeviceAddress(source)) TypeError: pci_address_to_name() got an unexpected keyword argument 'target'
XML part was: <hostdev managed="no" mode="subsystem" rawio="yes" type="scsi"> <source> <adapter name="scsi_host2"/> <address bus="0" target="1" unit="0"/> </source> </hostdev> <hostdev managed="no" mode="subsystem" rawio="yes" type="scsi"> <source> <adapter name="scsi_host2"/> <address bus="0" target="2" unit="0"/> </source> </hostdev>
As of creating custom partition - by default machine.slice has "a *:* rwm" in devices.list. But for every new VM libvirt removes *:* mask and fills the list with actually needed devices (as I understand the process). For example:
c 136:* rw c 1:3 rw c 1:7 rw c 1:5 rw c 1:8 rw c 1:9 rw c 5:2 rw c 10:232 rw c 253:0 rw c 10:228 rw c 10:196 rw
What I'm looking for is a way to tell libvirt about my additional devices without breaking oVirt.
The solution would be creating your own partition and somehow (e.g. VDSM hook) appending <resource> <partition>/machine/custom</partition> </resource> to the libvirt's <domain> element. I'm not sure how feasible creating your own partition is though. I've tried the process as follows: $ cat ~/create-partition.sh # sh ~/create-partition.sh for i in blkio cpu,cpuacct cpuset devices freezer memory net_cls perf_event do mkdir /sys/fs/cgroup/$i/machine.slice/custom.partition done for i in cpuset.cpus cpuset.mems do cat /sys/fs/cgroup/cpuset/machine.slice/$i > /sys/fs/cgroup/cpuset/machine.slice/custom.partition/$i done (creates /machine/custom partition). Now, we can create vdsm before_vm_start hook that will set given partition for the VM: $ pwd /usr/libexec/vdsm/hooks/before_vm_start $ cat 10_cgroups #!/usr/bin/python import hooking def custom_partition(domxml): resource = domxml.createElement('resource') partition = domxml.createElement('partition') partition_text = domxml.createTextNode('/machine/custom') partition.appendChild(partition_text) resource.appendChild(partition) return resource domxml = hooking.read_domxml() domain = domxml.getElementsByTagName('domain')[0] domain.appendChild(custom_partition(domxml)) hooking.write_domxml(domxml)
-- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su +7-495-411-7601 (ext. 1237)
3 июня 2016 г., в 12:24, Martin Polednik <mpolednik@redhat.com> написал(а):
On 03/06/16 11:48 +0300, Дмитрий Глушенок wrote:
Hello!
Is it possible to tell libvirt to add specific devices to qemu cgroup? By somehow enumerating the devices in XML using a hook for example. I'm passing scsi-generic disks (/dev/sgX) to VM using qemucmdline hook and it doesn't work until I remove "devices" from cgroup_controllers in qemu.conf.
One way to achieve this is creating a hook to generate the scsi device XML instead of modifying qemu cmdline directly. Libvirt assumes ownership of all devices created in the XML and therefore adds them to the machine cgroup.
Example of the XML taken from [1]: <devices> <hostdev mode='subsystem' type='scsi' sgio='filtered' rawio='yes'> <source> <adapter name='scsi_host0'/> <address bus='0' target='0' unit='0'/> </source> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </hostdev> </devices>
There is slight issue with this approach outlined in [2].
If you want to keep the qemu approach, I think creating a custom partition and moving devices there would be the cleanest approach. In this case, [3] could help but I'm not entirely sure if that would solve the issue.
[1] https://libvirt.org/formatdomain.html [2] https://bugzilla.redhat.com/show_bug.cgi?id=1325485 [3] https://libvirt.org/cgroups.html
-- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su +7-495-411-7601 (ext. 1237)
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_64C09473-EDEA-4FA9-8643-DC7E66D3674C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hello Martin, Thank you for your time. It is clear how to create partition and assign = it to a VM. But libvirt manipulates with contents of devices.list and I = didn't find a way to put my devices into devices.list and stop libvirt = from removing them. Anyway, it looks like that the best solution will be using oVirt 4.0 = where scsi pass-through have been fixed. -- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su <http://www.jet.msk.su/>+7-495-411-7601 (ext. = 1237)
6 =D0=B8=D1=8E=D0=BD=D1=8F 2016 =D0=B3., =D0=B2 12:33, Martin Polednik = <mpolednik@redhat.com> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB(=D0=B0):=
=20 On 03/06/16 14:05 +0300, =D0=94=D0=BC=D0=B8=D1=82=D1=80=D0=B8=D0=B9 = =D0=93=D0=BB=D1=83=D1=88=D0=B5=D0=BD=D0=BE=D0=BA wrote:
Thank you Martin! =20 Actually I tried the workaround hook, provided in [2], but then VDSM = (oVirt 3.6.6) tries to interpret hostdev in XML as PCI device, which = leads to: =20 ::The vm start process failed Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 703, in _startUnderlyingVm self._run() File "/usr/share/vdsm/virt/vm.py", line 1949, in _run self._domDependentInit() File "/usr/share/vdsm/virt/vm.py", line 1797, in _domDependentInit self._getUnderlyingVmDevicesInfo() File "/usr/share/vdsm/virt/vm.py", line 1738, in = _getUnderlyingVmDevicesInfo self._getUnderlyingHostDeviceInfo() File "/usr/share/vdsm/virt/vm.py", line 4277, in = _getUnderlyingHostDeviceInfo **self._getUnderlyingDeviceAddress(source)) TypeError: pci_address_to_name() got an unexpected keyword argument = 'target' =20 XML part was: <hostdev managed=3D"no" mode=3D"subsystem" rawio=3D"yes" type=3D"scsi">=
<source> <adapter name=3D"scsi_host2"/> <address bus=3D"0" target=3D"1" unit=3D"0"/> </source> </hostdev> <hostdev managed=3D"no" mode=3D"subsystem" rawio=3D"yes" type=3D"scsi">=
<source> <adapter name=3D"scsi_host2"/> <address bus=3D"0" target=3D"2" unit=3D"0"/> </source> </hostdev> =20 =20 As of creating custom partition - by default machine.slice has "a *:* = rwm" in devices.list. But for every new VM libvirt removes *:* mask and = fills the list with actually needed devices (as I understand the = process). For example: =20 c 136:* rw c 1:3 rw c 1:7 rw c 1:5 rw c 1:8 rw c 1:9 rw c 5:2 rw c 10:232 rw c 253:0 rw c 10:228 rw c 10:196 rw =20 What I'm looking for is a way to tell libvirt about my additional = devices without breaking oVirt. =20 The solution would be creating your own partition and somehow (e.g. VDSM hook) appending =20 <resource> <partition>/machine/custom</partition> </resource> =20 to the libvirt's <domain> element. I'm not sure how feasible creating your own partition is though. I've tried the process as follows: =20 $ cat ~/create-partition.sh # sh ~/create-partition.sh for i in blkio cpu,cpuacct cpuset devices freezer memory net_cls = perf_event do mkdir /sys/fs/cgroup/$i/machine.slice/custom.partition done =20 for i in cpuset.cpus cpuset.mems do cat /sys/fs/cgroup/cpuset/machine.slice/$i > = /sys/fs/cgroup/cpuset/machine.slice/custom.partition/$i done =20 (creates /machine/custom partition). Now, we can create vdsm before_vm_start hook that will set given = partition for the VM: =20 $ pwd /usr/libexec/vdsm/hooks/before_vm_start $ cat 10_cgroups =20 #!/usr/bin/python =20 import hooking =20 =20 def custom_partition(domxml): resource =3D domxml.createElement('resource') partition =3D domxml.createElement('partition') partition_text =3D domxml.createTextNode('/machine/custom') partition.appendChild(partition_text) resource.appendChild(partition) =20 return resource =20 domxml =3D hooking.read_domxml() domain =3D domxml.getElementsByTagName('domain')[0] domain.appendChild(custom_partition(domxml)) hooking.write_domxml(domxml) =20 -- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su +7-495-411-7601 (ext. 1237) =20
Hello! =20 Is it possible to tell libvirt to add specific devices to qemu = cgroup? By somehow enumerating the devices in XML using a hook for = example. I'm passing scsi-generic disks (/dev/sgX) to VM using qemucmdline = hook and it doesn't work until I remove "devices" from = cgroup_controllers in qemu.conf. =20 One way to achieve this is creating a hook to generate the scsi = device XML instead of modifying qemu cmdline directly. Libvirt assumes ownership of all devices created in the XML and therefore adds them = to
3 =D0=B8=D1=8E=D0=BD=D1=8F 2016 =D0=B3., =D0=B2 12:24, Martin = Polednik <mpolednik@redhat.com> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB= (=D0=B0): =20 On 03/06/16 11:48 +0300, =D0=94=D0=BC=D0=B8=D1=82=D1=80=D0=B8=D0=B9 = =D0=93=D0=BB=D1=83=D1=88=D0=B5=D0=BD=D0=BE=D0=BA wrote: the machine cgroup. =20 Example of the XML taken from [1]: <devices> <hostdev mode=3D'subsystem' type=3D'scsi' sgio=3D'filtered' = rawio=3D'yes'> <source> <adapter name=3D'scsi_host0'/> <address bus=3D'0' target=3D'0' unit=3D'0'/> </source> <readonly/> <address type=3D'drive' controller=3D'0' bus=3D'0' target=3D'0' = unit=3D'0'/> </hostdev> </devices> =20 There is slight issue with this approach outlined in [2]. =20 If you want to keep the qemu approach, I think creating a custom partition and moving devices there would be the cleanest approach. = In this case, [3] could help but I'm not entirely sure if that would solve the issue. =20 [1] https://libvirt.org/formatdomain.html [2] https://bugzilla.redhat.com/show_bug.cgi?id=3D1325485 [3] https://libvirt.org/cgroups.html
-- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su +7-495-411-7601 (ext. 1237) =20 _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_64C09473-EDEA-4FA9-8643-DC7E66D3674C Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div class=3D"">Hello Martin,</div><div class=3D""><br = class=3D""></div><div class=3D"">Thank you for your time. It is clear = how to create partition and assign it to a VM. But libvirt manipulates = with contents of devices.list and I didn't find a way to put my devices = into devices.list and stop libvirt from removing them.</div><div = class=3D""><br class=3D""></div><div class=3D"">Anyway, it looks like = that the best solution will be using oVirt 4.0 where scsi pass-through = have been fixed.</div><br class=3D""><div class=3D""> <div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; orphans: = auto; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div class=3D"">--</div><div class=3D""><div class=3D"" = style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;"><div class=3D"" = style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;"><div class=3D"" = style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;"><div class=3D"">Dmitry = Glushenok</div><div class=3D"">Jet Infosystems</div><div class=3D""><a = href=3D"http://www.jet.msk.su" = class=3D"">http://www.jet.msk.su</a></div></div></div></div>+7-495-411-760= 1 (ext. 1237)</div></div></div></div></div></div> </div> <br class=3D""><div><blockquote type=3D"cite" class=3D""><div class=3D"">6= =D0=B8=D1=8E=D0=BD=D1=8F 2016 =D0=B3., =D0=B2 12:33, Martin Polednik = <<a href=3D"mailto:mpolednik@redhat.com" = class=3D"">mpolednik@redhat.com</a>> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0= =D0=BB(=D0=B0):</div><br class=3D"Apple-interchange-newline"><div = class=3D""><span style=3D"font-family: Menlo-Regular; font-size: 13px; = font-style: normal; font-variant: normal; font-weight: normal; = letter-spacing: normal; line-height: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: = none; display: inline !important;" class=3D"">On 03/06/16 14:05 +0300, = =D0=94=D0=BC=D0=B8=D1=82=D1=80=D0=B8=D0=B9 =D0=93=D0=BB=D1=83=D1=88=D0=B5=D0= =BD=D0=BE=D0=BA wrote:</span><br style=3D"font-family: Menlo-Regular; = font-size: 13px; font-style: normal; font-variant: normal; font-weight: = normal; letter-spacing: normal; line-height: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: = 0px;" class=3D""><blockquote type=3D"cite" style=3D"font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant: = normal; font-weight: normal; letter-spacing: normal; line-height: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px;" class=3D"">Thank you Martin!<br = class=3D""><br class=3D"">Actually I tried the workaround hook, provided = in [2], but then VDSM (oVirt 3.6.6) tries to interpret hostdev in XML as = PCI device, which leads to:<br class=3D""><br class=3D"">::The vm start = process failed<br class=3D"">Traceback (most recent call last):<br = class=3D"">File "/usr/share/vdsm/virt/vm.py", line 703, in = _startUnderlyingVm<br class=3D""> self._run()<br = class=3D"">File "/usr/share/vdsm/virt/vm.py", line 1949, in _run<br = class=3D""> self._domDependentInit()<br class=3D"">File = "/usr/share/vdsm/virt/vm.py", line 1797, in _domDependentInit<br = class=3D""> self._getUnderlyingVmDevicesInfo()<br = class=3D"">File "/usr/share/vdsm/virt/vm.py", line 1738, in = _getUnderlyingVmDevicesInfo<br = class=3D""> self._getUnderlyingHostDeviceInfo()<br = class=3D"">File "/usr/share/vdsm/virt/vm.py", line 4277, in = _getUnderlyingHostDeviceInfo<br = class=3D""> **self._getUnderlyingDeviceAddress(source))<br = class=3D"">TypeError: pci_address_to_name() got an unexpected keyword = argument 'target'<br class=3D""><br class=3D"">XML part was:<br = class=3D""><hostdev managed=3D"no" mode=3D"subsystem" rawio=3D"yes" = type=3D"scsi"><br class=3D""><span class=3D"Apple-tab-span" = style=3D"white-space: pre;"> </span><source><br class=3D""><span = class=3D"Apple-tab-span" style=3D"white-space: pre;"> </span><span = class=3D"Apple-tab-span" style=3D"white-space: pre;"> = </span><adapter name=3D"scsi_host2"/><br class=3D""><span = class=3D"Apple-tab-span" style=3D"white-space: pre;"> </span><span = class=3D"Apple-tab-span" style=3D"white-space: pre;"> = </span><address bus=3D"0" target=3D"1" unit=3D"0"/><br = class=3D""><span class=3D"Apple-tab-span" style=3D"white-space: pre;"> = </span></source><br class=3D""></hostdev><br = class=3D""><hostdev managed=3D"no" mode=3D"subsystem" rawio=3D"yes" = type=3D"scsi"><br class=3D""><span class=3D"Apple-tab-span" = style=3D"white-space: pre;"> </span><source><br class=3D""><span = class=3D"Apple-tab-span" style=3D"white-space: pre;"> </span><span = class=3D"Apple-tab-span" style=3D"white-space: pre;"> = </span><adapter name=3D"scsi_host2"/><br class=3D""><span = class=3D"Apple-tab-span" style=3D"white-space: pre;"> </span><span = class=3D"Apple-tab-span" style=3D"white-space: pre;"> = </span><address bus=3D"0" target=3D"2" unit=3D"0"/><br = class=3D""><span class=3D"Apple-tab-span" style=3D"white-space: pre;"> = </span></source><br class=3D""></hostdev><br class=3D""><br = class=3D""><br class=3D"">As of creating custom partition - by default = machine.slice has "a *:* rwm" in devices.list. But for every new VM = libvirt removes *:* mask and fills the list with actually needed devices = (as I understand the process). For example:<br class=3D""><br class=3D"">c= 136:* rw<br class=3D"">c 1:3 rw<br class=3D"">c 1:7 rw<br class=3D"">c = 1:5 rw<br class=3D"">c 1:8 rw<br class=3D"">c 1:9 rw<br class=3D"">c 5:2 = rw<br class=3D"">c 10:232 rw<br class=3D"">c 253:0 rw<br class=3D"">c = 10:228 rw<br class=3D"">c 10:196 rw<br class=3D""><br class=3D"">What = I'm looking for is a way to tell libvirt about my additional devices = without breaking oVirt.<br class=3D""></blockquote><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><span= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D"">The solution would be creating = your own partition and somehow (e.g.</span><br style=3D"font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant: = normal; font-weight: normal; letter-spacing: normal; line-height: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px;" class=3D""><span = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D"">VDSM hook) appending</span><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><span= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D""><resource></span><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><span= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" = class=3D""> <partition>/machine/custom</partition><= /span><br style=3D"font-family: Menlo-Regular; font-size: 13px; = font-style: normal; font-variant: normal; font-weight: normal; = letter-spacing: normal; line-height: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" = class=3D""><span style=3D"font-family: Menlo-Regular; font-size: 13px; = font-style: normal; font-variant: normal; font-weight: normal; = letter-spacing: normal; line-height: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: = none; display: inline !important;" class=3D""></resource></span><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><span= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D"">to the libvirt's <domain> = element. I'm not sure how feasible creating</span><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><span= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D"">your own partition is though. = I've tried the process as follows:</span><br style=3D"font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant: = normal; font-weight: normal; letter-spacing: normal; line-height: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px;" class=3D""><br style=3D"font-family:= Menlo-Regular; font-size: 13px; font-style: normal; font-variant: = normal; font-weight: normal; letter-spacing: normal; line-height: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px;" class=3D""><span = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D"">$ cat = ~/create-partition.sh</span><br style=3D"font-family: Menlo-Regular; = font-size: 13px; font-style: normal; font-variant: normal; font-weight: = normal; letter-spacing: normal; line-height: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: = 0px;" class=3D""><span style=3D"font-family: Menlo-Regular; font-size: = 13px; font-style: normal; font-variant: normal; font-weight: normal; = letter-spacing: normal; line-height: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: = none; display: inline !important;" class=3D""># sh = ~/create-partition.sh</span><br style=3D"font-family: Menlo-Regular; = font-size: 13px; font-style: normal; font-variant: normal; font-weight: = normal; letter-spacing: normal; line-height: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: = 0px;" class=3D""><span style=3D"font-family: Menlo-Regular; font-size: = 13px; font-style: normal; font-variant: normal; font-weight: normal; = letter-spacing: normal; line-height: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: = none; display: inline !important;" class=3D"">for i in blkio cpu,cpuacct = cpuset devices freezer memory net_cls perf_event</span><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><span= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D"">do</span><br style=3D"font-family:= Menlo-Regular; font-size: 13px; font-style: normal; font-variant: = normal; font-weight: normal; letter-spacing: normal; line-height: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px;" class=3D""><span = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D""> mkdir = /sys/fs/cgroup/$i/machine.slice/custom.partition</span><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><span= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D"">done</span><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><span= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D"">for i in cpuset.cpus = cpuset.mems</span><br style=3D"font-family: Menlo-Regular; font-size: = 13px; font-style: normal; font-variant: normal; font-weight: normal; = letter-spacing: normal; line-height: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" = class=3D""><span style=3D"font-family: Menlo-Regular; font-size: 13px; = font-style: normal; font-variant: normal; font-weight: normal; = letter-spacing: normal; line-height: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: = none; display: inline !important;" class=3D"">do</span><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><span= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D""> cat = /sys/fs/cgroup/cpuset/machine.slice/$i > = /sys/fs/cgroup/cpuset/machine.slice/custom.partition/$i</span><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><span= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D"">done</span><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><span= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D"">(creates /machine/custom = partition).</span><br style=3D"font-family: Menlo-Regular; font-size: = 13px; font-style: normal; font-variant: normal; font-weight: normal; = letter-spacing: normal; line-height: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" = class=3D""><span style=3D"font-family: Menlo-Regular; font-size: 13px; = font-style: normal; font-variant: normal; font-weight: normal; = letter-spacing: normal; line-height: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: = none; display: inline !important;" class=3D"">Now, we can create vdsm = before_vm_start hook that will set given partition</span><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><span= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D"">for the VM:</span><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><span= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D"">$ pwd = /usr/libexec/vdsm/hooks/before_vm_start</span><br style=3D"font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant: = normal; font-weight: normal; letter-spacing: normal; line-height: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px;" class=3D""><span = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D"">$ cat 10_cgroups</span><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><span= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D"">#!/usr/bin/python</span><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><span= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D"">import hooking</span><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><span= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D"">def = custom_partition(domxml):</span><br style=3D"font-family: Menlo-Regular; = font-size: 13px; font-style: normal; font-variant: normal; font-weight: = normal; letter-spacing: normal; line-height: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: = 0px;" class=3D""><span style=3D"font-family: Menlo-Regular; font-size: = 13px; font-style: normal; font-variant: normal; font-weight: normal; = letter-spacing: normal; line-height: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: = none; display: inline !important;" class=3D""> resource =3D = domxml.createElement('resource')</span><br style=3D"font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant: = normal; font-weight: normal; letter-spacing: normal; line-height: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px;" class=3D""><span = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D""> partition =3D = domxml.createElement('partition')</span><br style=3D"font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant: = normal; font-weight: normal; letter-spacing: normal; line-height: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px;" class=3D""><span = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D""> partition_text =3D = domxml.createTextNode('/machine/custom')</span><br style=3D"font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant: = normal; font-weight: normal; letter-spacing: normal; line-height: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px;" class=3D""><span = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" = class=3D""> partition.appendChild(partition_text)</span><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><span= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" = class=3D""> resource.appendChild(partition)</span><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><span= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" class=3D""> return = resource</span><br style=3D"font-family: Menlo-Regular; font-size: 13px; = font-style: normal; font-variant: normal; font-weight: normal; = letter-spacing: normal; line-height: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" = class=3D""><br style=3D"font-family: Menlo-Regular; font-size: 13px; = font-style: normal; font-variant: normal; font-weight: normal; = letter-spacing: normal; line-height: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" = class=3D""><span style=3D"font-family: Menlo-Regular; font-size: 13px; = font-style: normal; font-variant: normal; font-weight: normal; = letter-spacing: normal; line-height: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: = none; display: inline !important;" class=3D"">domxml =3D = hooking.read_domxml()</span><br style=3D"font-family: Menlo-Regular; = font-size: 13px; font-style: normal; font-variant: normal; font-weight: = normal; letter-spacing: normal; line-height: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: = 0px;" class=3D""><span style=3D"font-family: Menlo-Regular; font-size: = 13px; font-style: normal; font-variant: normal; font-weight: normal; = letter-spacing: normal; line-height: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: = none; display: inline !important;" class=3D"">domain =3D = domxml.getElementsByTagName('domain')[0]</span><br style=3D"font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant: = normal; font-weight: normal; letter-spacing: normal; line-height: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px;" class=3D""><span = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" = class=3D"">domain.appendChild(custom_partition(domxml))</span><br = style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><span= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; = display: inline !important;" = class=3D"">hooking.write_domxml(domxml)</span><br style=3D"font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant: = normal; font-weight: normal; letter-spacing: normal; line-height: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px;" class=3D""><br style=3D"font-family:= Menlo-Regular; font-size: 13px; font-style: normal; font-variant: = normal; font-weight: normal; letter-spacing: normal; line-height: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px;" class=3D""><blockquote type=3D"cite"= style=3D"font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: = auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D"">--<br= class=3D"">Dmitry Glushenok<br class=3D"">Jet Infosystems<br = class=3D""><a href=3D"http://www.jet.msk.su" = class=3D"">http://www.jet.msk.su</a><br class=3D"">+7-495-411-7601 (ext. = 1237)<br class=3D""><br class=3D""><blockquote type=3D"cite" class=3D"">3 = =D0=B8=D1=8E=D0=BD=D1=8F 2016 =D0=B3., =D0=B2 12:24, Martin Polednik = <mpolednik@redhat.com> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB(=D0= =B0):<br class=3D""><br class=3D"">On 03/06/16 11:48 +0300, = =D0=94=D0=BC=D0=B8=D1=82=D1=80=D0=B8=D0=B9 =D0=93=D0=BB=D1=83=D1=88=D0=B5=D0= =BD=D0=BE=D0=BA wrote:<br class=3D""><blockquote type=3D"cite" = class=3D"">Hello!<br class=3D""><br class=3D"">Is it possible to tell = libvirt to add specific devices to qemu cgroup? By somehow enumerating = the devices in XML using a hook for example.<br class=3D"">I'm passing = scsi-generic disks (/dev/sgX) to VM using qemucmdline hook and it = doesn't work until I remove "devices" from cgroup_controllers in = qemu.conf.<br class=3D""></blockquote><br class=3D"">One way to achieve = this is creating a hook to generate the scsi device<br class=3D"">XML = instead of modifying qemu cmdline directly. Libvirt assumes<br = class=3D"">ownership of all devices created in the XML and therefore = adds them to<br class=3D"">the machine cgroup.<br class=3D""><br = class=3D"">Example of the XML taken from [1]:<br = class=3D""><devices><br class=3D""> <hostdev = mode=3D'subsystem' type=3D'scsi' sgio=3D'filtered' rawio=3D'yes'><br = class=3D""> <source><br = class=3D""> <adapt= er name=3D'scsi_host0'/><br = class=3D""> <addre= ss bus=3D'0' target=3D'0' unit=3D'0'/><br = class=3D""> </source><br = class=3D""> <readonly/><br = class=3D""> <address type=3D'drive' = controller=3D'0' bus=3D'0' target=3D'0' unit=3D'0'/><br = class=3D""> </hostdev><br class=3D""></devices><br = class=3D""><br class=3D"">There is slight issue with this approach = outlined in [2].<br class=3D""><br class=3D"">If you want to keep the = qemu approach, I think creating a custom<br class=3D"">partition and = moving devices there would be the cleanest approach. In<br class=3D"">this= case, [3] could help but I'm not entirely sure if that would<br = class=3D"">solve the issue.<br class=3D""><br class=3D"">[1] = https://libvirt.org/formatdomain.html<br class=3D"">[2] = https://bugzilla.redhat.com/show_bug.cgi?id=3D1325485<br class=3D"">[3] = https://libvirt.org/cgroups.html<br class=3D""><blockquote type=3D"cite" = class=3D"">--<br class=3D"">Dmitry Glushenok<br class=3D"">Jet = Infosystems<br class=3D"">http://www.jet.msk.su<br = class=3D"">+7-495-411-7601 (ext. 1237)<br class=3D""><br = class=3D"">_______________________________________________<br = class=3D"">Users mailing list<br class=3D"">Users@ovirt.org<br = class=3D"">http://lists.ovirt.org/mailman/listinfo/users</blockquote></blo= ckquote></blockquote></div></blockquote></div><br = class=3D""></body></html>= --Apple-Mail=_64C09473-EDEA-4FA9-8643-DC7E66D3674C--
On 06/06/16 13:26 +0300, Дмитрий Глушенок wrote:
Hello Martin,
Thank you for your time. It is clear how to create partition and assign it to a VM. But libvirt manipulates with contents of devices.list and I didn't find a way to put my devices into devices.list and stop libvirt from removing them.
That is, by the way, pretty interesting. I have tried it right now and when using custom partition, libvirt doesn't touch my devices.list. It might make sense to file a libvirt bug.
Anyway, it looks like that the best solution will be using oVirt 4.0 where scsi pass-through have been fixed.
Hoping so!
-- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su <http://www.jet.msk.su/>+7-495-411-7601 (ext. 1237)
6 июня 2016 г., в 12:33, Martin Polednik <mpolednik@redhat.com> написал(а):
On 03/06/16 14:05 +0300, Дмитрий Глушенок wrote:
Thank you Martin!
Actually I tried the workaround hook, provided in [2], but then VDSM (oVirt 3.6.6) tries to interpret hostdev in XML as PCI device, which leads to:
::The vm start process failed Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 703, in _startUnderlyingVm self._run() File "/usr/share/vdsm/virt/vm.py", line 1949, in _run self._domDependentInit() File "/usr/share/vdsm/virt/vm.py", line 1797, in _domDependentInit self._getUnderlyingVmDevicesInfo() File "/usr/share/vdsm/virt/vm.py", line 1738, in _getUnderlyingVmDevicesInfo self._getUnderlyingHostDeviceInfo() File "/usr/share/vdsm/virt/vm.py", line 4277, in _getUnderlyingHostDeviceInfo **self._getUnderlyingDeviceAddress(source)) TypeError: pci_address_to_name() got an unexpected keyword argument 'target'
XML part was: <hostdev managed="no" mode="subsystem" rawio="yes" type="scsi"> <source> <adapter name="scsi_host2"/> <address bus="0" target="1" unit="0"/> </source> </hostdev> <hostdev managed="no" mode="subsystem" rawio="yes" type="scsi"> <source> <adapter name="scsi_host2"/> <address bus="0" target="2" unit="0"/> </source> </hostdev>
As of creating custom partition - by default machine.slice has "a *:* rwm" in devices.list. But for every new VM libvirt removes *:* mask and fills the list with actually needed devices (as I understand the process). For example:
c 136:* rw c 1:3 rw c 1:7 rw c 1:5 rw c 1:8 rw c 1:9 rw c 5:2 rw c 10:232 rw c 253:0 rw c 10:228 rw c 10:196 rw
What I'm looking for is a way to tell libvirt about my additional devices without breaking oVirt.
The solution would be creating your own partition and somehow (e.g. VDSM hook) appending
<resource> <partition>/machine/custom</partition> </resource>
to the libvirt's <domain> element. I'm not sure how feasible creating your own partition is though. I've tried the process as follows:
$ cat ~/create-partition.sh # sh ~/create-partition.sh for i in blkio cpu,cpuacct cpuset devices freezer memory net_cls perf_event do mkdir /sys/fs/cgroup/$i/machine.slice/custom.partition done
for i in cpuset.cpus cpuset.mems do cat /sys/fs/cgroup/cpuset/machine.slice/$i > /sys/fs/cgroup/cpuset/machine.slice/custom.partition/$i done
(creates /machine/custom partition). Now, we can create vdsm before_vm_start hook that will set given partition for the VM:
$ pwd /usr/libexec/vdsm/hooks/before_vm_start $ cat 10_cgroups
#!/usr/bin/python
import hooking
def custom_partition(domxml): resource = domxml.createElement('resource') partition = domxml.createElement('partition') partition_text = domxml.createTextNode('/machine/custom') partition.appendChild(partition_text) resource.appendChild(partition)
return resource
domxml = hooking.read_domxml() domain = domxml.getElementsByTagName('domain')[0] domain.appendChild(custom_partition(domxml)) hooking.write_domxml(domxml)
-- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su +7-495-411-7601 (ext. 1237)
3 июня 2016 г., в 12:24, Martin Polednik <mpolednik@redhat.com> написал(а):
On 03/06/16 11:48 +0300, Дмитрий Глушенок wrote:
Hello!
Is it possible to tell libvirt to add specific devices to qemu cgroup? By somehow enumerating the devices in XML using a hook for example. I'm passing scsi-generic disks (/dev/sgX) to VM using qemucmdline hook and it doesn't work until I remove "devices" from cgroup_controllers in qemu.conf.
One way to achieve this is creating a hook to generate the scsi device XML instead of modifying qemu cmdline directly. Libvirt assumes ownership of all devices created in the XML and therefore adds them to the machine cgroup.
Example of the XML taken from [1]: <devices> <hostdev mode='subsystem' type='scsi' sgio='filtered' rawio='yes'> <source> <adapter name='scsi_host0'/> <address bus='0' target='0' unit='0'/> </source> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </hostdev> </devices>
There is slight issue with this approach outlined in [2].
If you want to keep the qemu approach, I think creating a custom partition and moving devices there would be the cleanest approach. In this case, [3] could help but I'm not entirely sure if that would solve the issue.
[1] https://libvirt.org/formatdomain.html [2] https://bugzilla.redhat.com/show_bug.cgi?id=1325485 [3] https://libvirt.org/cgroups.html
-- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su +7-495-411-7601 (ext. 1237)
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_F5070364-047E-4F4A-BEE4-93C0242B9F73 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 So, libvirt behaves differently with custom partition? My assumption was = that it will perform the same changes. I will have in mind in case oVirt = 4 will be released too late. May be I wrote incorrectly - in oVirt 4.0 RC1 scsi pass-through already = working. -- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su <http://www.jet.msk.su/>+7-495-411-7601 (ext. = 1237)
6 =D0=B8=D1=8E=D0=BD=D1=8F 2016 =D0=B3., =D0=B2 13:35, Martin Polednik = <mpolednik@redhat.com> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB(=D0=B0):=
6 =D0=B8=D1=8E=D0=BD=D1=8F 2016 =D0=B3., =D0=B2 12:33, Martin = Polednik <mpolednik@redhat.com> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB= (=D0=B0): =20 On 03/06/16 14:05 +0300, =D0=94=D0=BC=D0=B8=D1=82=D1=80=D0=B8=D0=B9 = =D0=93=D0=BB=D1=83=D1=88=D0=B5=D0=BD=D0=BE=D0=BA wrote:
Thank you Martin! =20 Actually I tried the workaround hook, provided in [2], but then = VDSM (oVirt 3.6.6) tries to interpret hostdev in XML as PCI device, = which leads to: =20 ::The vm start process failed Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 703, in _startUnderlyingVm self._run() File "/usr/share/vdsm/virt/vm.py", line 1949, in _run self._domDependentInit() File "/usr/share/vdsm/virt/vm.py", line 1797, in _domDependentInit self._getUnderlyingVmDevicesInfo() File "/usr/share/vdsm/virt/vm.py", line 1738, in = _getUnderlyingVmDevicesInfo self._getUnderlyingHostDeviceInfo() File "/usr/share/vdsm/virt/vm.py", line 4277, in = _getUnderlyingHostDeviceInfo **self._getUnderlyingDeviceAddress(source)) TypeError: pci_address_to_name() got an unexpected keyword argument = 'target' =20 XML part was: <hostdev managed=3D"no" mode=3D"subsystem" rawio=3D"yes" = type=3D"scsi"> <source> <adapter name=3D"scsi_host2"/> <address bus=3D"0" target=3D"1" unit=3D"0"/> </source> </hostdev> <hostdev managed=3D"no" mode=3D"subsystem" rawio=3D"yes" = type=3D"scsi"> <source> <adapter name=3D"scsi_host2"/> <address bus=3D"0" target=3D"2" unit=3D"0"/> </source> </hostdev> =20 =20 As of creating custom partition - by default machine.slice has "a = *:* rwm" in devices.list. But for every new VM libvirt removes *:* mask = and fills the list with actually needed devices (as I understand the =
=20 c 136:* rw c 1:3 rw c 1:7 rw c 1:5 rw c 1:8 rw c 1:9 rw c 5:2 rw c 10:232 rw c 253:0 rw c 10:228 rw c 10:196 rw =20 What I'm looking for is a way to tell libvirt about my additional = devices without breaking oVirt. =20 The solution would be creating your own partition and somehow (e.g. VDSM hook) appending =20 <resource> <partition>/machine/custom</partition> </resource> =20 to the libvirt's <domain> element. I'm not sure how feasible = creating your own partition is though. I've tried the process as follows: =20 $ cat ~/create-partition.sh # sh ~/create-partition.sh for i in blkio cpu,cpuacct cpuset devices freezer memory net_cls =
do mkdir /sys/fs/cgroup/$i/machine.slice/custom.partition done =20 for i in cpuset.cpus cpuset.mems do cat /sys/fs/cgroup/cpuset/machine.slice/$i > = /sys/fs/cgroup/cpuset/machine.slice/custom.partition/$i done =20 (creates /machine/custom partition). Now, we can create vdsm before_vm_start hook that will set given =
for the VM: =20 $ pwd /usr/libexec/vdsm/hooks/before_vm_start $ cat 10_cgroups =20 #!/usr/bin/python =20 import hooking =20 =20 def custom_partition(domxml): resource =3D domxml.createElement('resource') partition =3D domxml.createElement('partition') partition_text =3D domxml.createTextNode('/machine/custom') partition.appendChild(partition_text) resource.appendChild(partition) =20 return resource =20 domxml =3D hooking.read_domxml() domain =3D domxml.getElementsByTagName('domain')[0] domain.appendChild(custom_partition(domxml)) hooking.write_domxml(domxml) =20
-- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su +7-495-411-7601 (ext. 1237) =20
3 =D0=B8=D1=8E=D0=BD=D1=8F 2016 =D0=B3., =D0=B2 12:24, Martin = Polednik <mpolednik@redhat.com> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB= (=D0=B0): =20 On 03/06/16 11:48 +0300, =D0=94=D0=BC=D0=B8=D1=82=D1=80=D0=B8=D0=B9 = =D0=93=D0=BB=D1=83=D1=88=D0=B5=D0=BD=D0=BE=D0=BA wrote:
Hello! =20 Is it possible to tell libvirt to add specific devices to qemu = cgroup? By somehow enumerating the devices in XML using a hook for = example. I'm passing scsi-generic disks (/dev/sgX) to VM using qemucmdline = hook and it doesn't work until I remove "devices" from = cgroup_controllers in qemu.conf. =20 One way to achieve this is creating a hook to generate the scsi = device XML instead of modifying qemu cmdline directly. Libvirt assumes ownership of all devices created in the XML and therefore adds =
=20 process). For example: perf_event partition them to
the machine cgroup. =20 Example of the XML taken from [1]: <devices> <hostdev mode=3D'subsystem' type=3D'scsi' sgio=3D'filtered' = rawio=3D'yes'> <source> <adapter name=3D'scsi_host0'/> <address bus=3D'0' target=3D'0' unit=3D'0'/> </source> <readonly/> <address type=3D'drive' controller=3D'0' bus=3D'0' target=3D'0' = unit=3D'0'/> </hostdev> </devices> =20 There is slight issue with this approach outlined in [2]. =20 If you want to keep the qemu approach, I think creating a custom partition and moving devices there would be the cleanest approach. = In this case, [3] could help but I'm not entirely sure if that would solve the issue. =20 [1] https://libvirt.org/formatdomain.html [2] https://bugzilla.redhat.com/show_bug.cgi?id=3D1325485 [3] https://libvirt.org/cgroups.html
-- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su +7-495-411-7601 (ext. 1237) =20 _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users =20
Hello Martin, =20 Thank you for your time. It is clear how to create partition and = assign it to a VM. But libvirt manipulates with contents of devices.list = and I didn't find a way to put my devices into devices.list and stop =
=20 On 06/06/16 13:26 +0300, =D0=94=D0=BC=D0=B8=D1=82=D1=80=D0=B8=D0=B9 = =D0=93=D0=BB=D1=83=D1=88=D0=B5=D0=BD=D0=BE=D0=BA wrote: libvirt from removing them. =20 That is, by the way, pretty interesting. I have tried it right now and when using custom partition, libvirt doesn't touch my devices.list. =20 It might make sense to file a libvirt bug. =20
Anyway, it looks like that the best solution will be using oVirt 4.0 = where scsi pass-through have been fixed. =20 Hoping so! =20 -- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su <http://www.jet.msk.su/>+7-495-411-7601 (ext. =
--Apple-Mail=_F5070364-047E-4F4A-BEE4-93C0242B9F73 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div class=3D"">So, libvirt behaves differently with custom = partition? My assumption was that it will perform the same changes. I = will have in mind in case oVirt 4 will be released too late.</div><div = class=3D"">May be I wrote incorrectly - in oVirt 4.0 RC1 scsi = pass-through already working.</div><div class=3D""><br = class=3D""></div><div class=3D""> <div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; orphans: = auto; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div class=3D"">--</div><div class=3D""><div class=3D"" = style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;"><div class=3D"" = style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;"><div class=3D"" = style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;"><div class=3D"">Dmitry = Glushenok</div><div class=3D"">Jet Infosystems</div><div class=3D""><a = href=3D"http://www.jet.msk.su" = class=3D"">http://www.jet.msk.su</a></div></div></div></div>+7-495-411-760= 1 (ext. 1237)</div></div></div></div></div></div> </div> <br class=3D""><div><blockquote type=3D"cite" class=3D""><div class=3D"">6= =D0=B8=D1=8E=D0=BD=D1=8F 2016 =D0=B3., =D0=B2 13:35, Martin Polednik = <<a href=3D"mailto:mpolednik@redhat.com" = class=3D"">mpolednik@redhat.com</a>> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0= =D0=BB(=D0=B0):</div><br class=3D"Apple-interchange-newline"><div = class=3D"">On 06/06/16 13:26 +0300, =D0=94=D0=BC=D0=B8=D1=82=D1=80=D0=B8=D0= =B9 =D0=93=D0=BB=D1=83=D1=88=D0=B5=D0=BD=D0=BE=D0=BA wrote:<br = class=3D""><blockquote type=3D"cite" class=3D"">Hello Martin,<br = class=3D""><br class=3D"">Thank you for your time. It is clear how to = create partition and assign it to a VM. But libvirt manipulates with = contents of devices.list and I didn't find a way to put my devices into = devices.list and stop libvirt from removing them.<br = class=3D""></blockquote><br class=3D"">That is, by the way, pretty = interesting. I have tried it right now and<br class=3D"">when using = custom partition, libvirt doesn't touch my devices.list.<br class=3D""><br= class=3D"">It might make sense to file a libvirt bug.<br class=3D""><br = class=3D""><blockquote type=3D"cite" class=3D"">Anyway, it looks like = that the best solution will be using oVirt 4.0 where scsi pass-through = have been fixed.<br class=3D""></blockquote><br class=3D"">Hoping so!<br = class=3D""><br class=3D""><blockquote type=3D"cite" class=3D"">--<br = class=3D"">Dmitry Glushenok<br class=3D"">Jet Infosystems<br class=3D""><a= href=3D"http://www.jet.msk.su" class=3D"">http://www.jet.msk.su</a> = <<a href=3D"http://www.jet.msk.su/" = class=3D"">http://www.jet.msk.su/</a>>+7-495-411-7601 (ext. 1237)<br = class=3D""><br class=3D""><blockquote type=3D"cite" class=3D"">6 = =D0=B8=D1=8E=D0=BD=D1=8F 2016 =D0=B3., =D0=B2 12:33, Martin Polednik = <<a href=3D"mailto:mpolednik@redhat.com" = class=3D"">mpolednik@redhat.com</a>> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0= =D0=BB(=D0=B0):<br class=3D""><br class=3D"">On 03/06/16 14:05 +0300, = =D0=94=D0=BC=D0=B8=D1=82=D1=80=D0=B8=D0=B9 =D0=93=D0=BB=D1=83=D1=88=D0=B5=D0= =BD=D0=BE=D0=BA wrote:<br class=3D""><blockquote type=3D"cite" = class=3D"">Thank you Martin!<br class=3D""><br class=3D"">Actually I = tried the workaround hook, provided in [2], but then VDSM (oVirt 3.6.6) = tries to interpret hostdev in XML as PCI device, which leads to:<br = class=3D""><br class=3D"">::The vm start process failed<br = class=3D"">Traceback (most recent call last):<br class=3D"">File = "/usr/share/vdsm/virt/vm.py", line 703, in _startUnderlyingVm<br = class=3D""> self._run()<br class=3D"">File = "/usr/share/vdsm/virt/vm.py", line 1949, in _run<br class=3D""> = self._domDependentInit()<br class=3D"">File = "/usr/share/vdsm/virt/vm.py", line 1797, in _domDependentInit<br = class=3D""> self._getUnderlyingVmDevicesInfo()<br class=3D"">File = "/usr/share/vdsm/virt/vm.py", line 1738, in = _getUnderlyingVmDevicesInfo<br class=3D""> = self._getUnderlyingHostDeviceInfo()<br class=3D"">File = "/usr/share/vdsm/virt/vm.py", line 4277, in = _getUnderlyingHostDeviceInfo<br class=3D""> = **self._getUnderlyingDeviceAddress(source))<br class=3D"">TypeError:= pci_address_to_name() got an unexpected keyword argument 'target'<br = class=3D""><br class=3D"">XML part was:<br class=3D""><hostdev = managed=3D"no" mode=3D"subsystem" rawio=3D"yes" type=3D"scsi"><br = class=3D""><span class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span><source><br class=3D""><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span><adapter = name=3D"scsi_host2"/><br class=3D""><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span><address bus=3D"0" target=3D"1" = unit=3D"0"/><br class=3D""><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span></source><br = class=3D""></hostdev><br class=3D""><hostdev managed=3D"no" = mode=3D"subsystem" rawio=3D"yes" type=3D"scsi"><br class=3D""><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> = </span><source><br class=3D""><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span><adapter = name=3D"scsi_host2"/><br class=3D""><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span><address bus=3D"0" target=3D"2" = unit=3D"0"/><br class=3D""><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span></source><br = class=3D""></hostdev><br class=3D""><br class=3D""><br class=3D"">As= of creating custom partition - by default machine.slice has "a *:* rwm" = in devices.list. But for every new VM libvirt removes *:* mask and fills = the list with actually needed devices (as I understand the process). For = example:<br class=3D""><br class=3D"">c 136:* rw<br class=3D"">c 1:3 = rw<br class=3D"">c 1:7 rw<br class=3D"">c 1:5 rw<br class=3D"">c 1:8 = rw<br class=3D"">c 1:9 rw<br class=3D"">c 5:2 rw<br class=3D"">c 10:232 = rw<br class=3D"">c 253:0 rw<br class=3D"">c 10:228 rw<br class=3D"">c = 10:196 rw<br class=3D""><br class=3D"">What I'm looking for is a way to = tell libvirt about my additional devices without breaking oVirt.<br = class=3D""></blockquote><br class=3D"">The solution would be creating = your own partition and somehow (e.g.<br class=3D"">VDSM hook) = appending<br class=3D""><br class=3D""><resource><br class=3D""> = <partition>/machine/custom</partition><br = class=3D""></resource><br class=3D""><br class=3D"">to the = libvirt's <domain> element. I'm not sure how feasible creating<br = class=3D"">your own partition is though. I've tried the process as = follows:<br class=3D""><br class=3D"">$ cat ~/create-partition.sh<br = class=3D""># sh ~/create-partition.sh<br class=3D"">for i in blkio = cpu,cpuacct cpuset devices freezer memory net_cls perf_event<br = class=3D"">do<br class=3D""> mkdir = /sys/fs/cgroup/$i/machine.slice/custom.partition<br class=3D"">done<br = class=3D""><br class=3D"">for i in cpuset.cpus cpuset.mems<br = class=3D"">do<br class=3D""> cat = /sys/fs/cgroup/cpuset/machine.slice/$i > = /sys/fs/cgroup/cpuset/machine.slice/custom.partition/$i<br = class=3D"">done<br class=3D""><br class=3D"">(creates /machine/custom = partition).<br class=3D"">Now, we can create vdsm before_vm_start hook = that will set given partition<br class=3D"">for the VM:<br class=3D""><br = class=3D"">$ pwd /usr/libexec/vdsm/hooks/before_vm_start<br class=3D"">$ = cat 10_cgroups<br class=3D""><br class=3D"">#!/usr/bin/python<br = class=3D""><br class=3D"">import hooking<br class=3D""><br class=3D""><br = class=3D"">def custom_partition(domxml):<br class=3D""> resource =3D= domxml.createElement('resource')<br class=3D""> partition =3D = domxml.createElement('partition')<br class=3D""> partition_text =3D = domxml.createTextNode('/machine/custom')<br class=3D""> = partition.appendChild(partition_text)<br class=3D""> = resource.appendChild(partition)<br class=3D""><br class=3D""> = return resource<br class=3D""><br class=3D"">domxml =3D = hooking.read_domxml()<br class=3D"">domain =3D = domxml.getElementsByTagName('domain')[0]<br = class=3D"">domain.appendChild(custom_partition(domxml))<br = class=3D"">hooking.write_domxml(domxml)<br class=3D""><br = class=3D""><blockquote type=3D"cite" class=3D"">--<br class=3D"">Dmitry = Glushenok<br class=3D"">Jet Infosystems<br class=3D""><a = href=3D"http://www.jet.msk.su" class=3D"">http://www.jet.msk.su</a><br = class=3D"">+7-495-411-7601 (ext. 1237)<br class=3D""><br = class=3D""><blockquote type=3D"cite" class=3D"">3 =D0=B8=D1=8E=D0=BD=D1=8F= 2016 =D0=B3., =D0=B2 12:24, Martin Polednik = <mpolednik@redhat.com> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB(=D0= =B0):<br class=3D""><br class=3D"">On 03/06/16 11:48 +0300, = =D0=94=D0=BC=D0=B8=D1=82=D1=80=D0=B8=D0=B9 =D0=93=D0=BB=D1=83=D1=88=D0=B5=D0= =BD=D0=BE=D0=BA wrote:<br class=3D""><blockquote type=3D"cite" = class=3D"">Hello!<br class=3D""><br class=3D"">Is it possible to tell = libvirt to add specific devices to qemu cgroup? By somehow enumerating = the devices in XML using a hook for example.<br class=3D"">I'm passing = scsi-generic disks (/dev/sgX) to VM using qemucmdline hook and it = doesn't work until I remove "devices" from cgroup_controllers in = qemu.conf.<br class=3D""></blockquote><br class=3D"">One way to achieve = this is creating a hook to generate the scsi device<br class=3D"">XML = instead of modifying qemu cmdline directly. Libvirt assumes<br = class=3D"">ownership of all devices created in the XML and therefore = adds them to<br class=3D"">the machine cgroup.<br class=3D""><br = class=3D"">Example of the XML taken from [1]:<br = class=3D""><devices><br class=3D""> <hostdev mode=3D'subsystem' = type=3D'scsi' sgio=3D'filtered' rawio=3D'yes'><br class=3D""> = <source><br class=3D""> = <adapter = name=3D'scsi_host0'/><br class=3D""> = <address bus=3D'0' = target=3D'0' unit=3D'0'/><br class=3D""> = </source><br class=3D""> = <readonly/><br class=3D""> = <address type=3D'drive' controller=3D'0' = bus=3D'0' target=3D'0' unit=3D'0'/><br class=3D""> = </hostdev><br class=3D""></devices><br class=3D""><br = class=3D"">There is slight issue with this approach outlined in [2].<br = class=3D""><br class=3D"">If you want to keep the qemu approach, I think = creating a custom<br class=3D"">partition and moving devices there would = be the cleanest approach. In<br class=3D"">this case, [3] could help but = I'm not entirely sure if that would<br class=3D"">solve the issue.<br = class=3D""><br class=3D"">[1] https://libvirt.org/formatdomain.html<br = class=3D"">[2] https://bugzilla.redhat.com/show_bug.cgi?id=3D1325485<br = class=3D"">[3] https://libvirt.org/cgroups.html<br class=3D""><blockquote = type=3D"cite" class=3D"">--<br class=3D"">Dmitry Glushenok<br = class=3D"">Jet Infosystems<br class=3D"">http://www.jet.msk.su<br = class=3D"">+7-495-411-7601 (ext. 1237)<br class=3D""><br = class=3D"">_______________________________________________<br = class=3D"">Users mailing list<br class=3D"">Users@ovirt.org<br = class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br = class=3D""></blockquote></blockquote></blockquote></blockquote><br = class=3D""></blockquote></div></blockquote></div><br = class=3D""></body></html>= --Apple-Mail=_F5070364-047E-4F4A-BEE4-93C0242B9F73--
On 06/06/16 16:15 +0300, Дмитрий Глушенок wrote:
So, libvirt behaves differently with custom partition? My assumption was that it will perform the same changes. I will have in mind in case oVirt 4 will be released too late. May be I wrote incorrectly - in oVirt 4.0 RC1 scsi pass-through already working.
It should be working.
-- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su <http://www.jet.msk.su/>+7-495-411-7601 (ext. 1237)
6 июня 2016 г., в 13:35, Martin Polednik <mpolednik@redhat.com> написал(а):
On 06/06/16 13:26 +0300, Дмитрий Глушенок wrote:
Hello Martin,
Thank you for your time. It is clear how to create partition and assign it to a VM. But libvirt manipulates with contents of devices.list and I didn't find a way to put my devices into devices.list and stop libvirt from removing them.
That is, by the way, pretty interesting. I have tried it right now and when using custom partition, libvirt doesn't touch my devices.list.
It might make sense to file a libvirt bug.
Anyway, it looks like that the best solution will be using oVirt 4.0 where scsi pass-through have been fixed.
Hoping so!
-- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su <http://www.jet.msk.su/>+7-495-411-7601 (ext. 1237)
6 июня 2016 г., в 12:33, Martin Polednik <mpolednik@redhat.com> написал(а):
On 03/06/16 14:05 +0300, Дмитрий Глушенок wrote:
Thank you Martin!
Actually I tried the workaround hook, provided in [2], but then VDSM (oVirt 3.6.6) tries to interpret hostdev in XML as PCI device, which leads to:
::The vm start process failed Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 703, in _startUnderlyingVm self._run() File "/usr/share/vdsm/virt/vm.py", line 1949, in _run self._domDependentInit() File "/usr/share/vdsm/virt/vm.py", line 1797, in _domDependentInit self._getUnderlyingVmDevicesInfo() File "/usr/share/vdsm/virt/vm.py", line 1738, in _getUnderlyingVmDevicesInfo self._getUnderlyingHostDeviceInfo() File "/usr/share/vdsm/virt/vm.py", line 4277, in _getUnderlyingHostDeviceInfo **self._getUnderlyingDeviceAddress(source)) TypeError: pci_address_to_name() got an unexpected keyword argument 'target'
XML part was: <hostdev managed="no" mode="subsystem" rawio="yes" type="scsi"> <source> <adapter name="scsi_host2"/> <address bus="0" target="1" unit="0"/> </source> </hostdev> <hostdev managed="no" mode="subsystem" rawio="yes" type="scsi"> <source> <adapter name="scsi_host2"/> <address bus="0" target="2" unit="0"/> </source> </hostdev>
As of creating custom partition - by default machine.slice has "a *:* rwm" in devices.list. But for every new VM libvirt removes *:* mask and fills the list with actually needed devices (as I understand the process). For example:
c 136:* rw c 1:3 rw c 1:7 rw c 1:5 rw c 1:8 rw c 1:9 rw c 5:2 rw c 10:232 rw c 253:0 rw c 10:228 rw c 10:196 rw
What I'm looking for is a way to tell libvirt about my additional devices without breaking oVirt.
The solution would be creating your own partition and somehow (e.g. VDSM hook) appending
<resource> <partition>/machine/custom</partition> </resource>
to the libvirt's <domain> element. I'm not sure how feasible creating your own partition is though. I've tried the process as follows:
$ cat ~/create-partition.sh # sh ~/create-partition.sh for i in blkio cpu,cpuacct cpuset devices freezer memory net_cls perf_event do mkdir /sys/fs/cgroup/$i/machine.slice/custom.partition done
for i in cpuset.cpus cpuset.mems do cat /sys/fs/cgroup/cpuset/machine.slice/$i > /sys/fs/cgroup/cpuset/machine.slice/custom.partition/$i done
(creates /machine/custom partition). Now, we can create vdsm before_vm_start hook that will set given partition for the VM:
$ pwd /usr/libexec/vdsm/hooks/before_vm_start $ cat 10_cgroups
#!/usr/bin/python
import hooking
def custom_partition(domxml): resource = domxml.createElement('resource') partition = domxml.createElement('partition') partition_text = domxml.createTextNode('/machine/custom') partition.appendChild(partition_text) resource.appendChild(partition)
return resource
domxml = hooking.read_domxml() domain = domxml.getElementsByTagName('domain')[0] domain.appendChild(custom_partition(domxml)) hooking.write_domxml(domxml)
-- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su +7-495-411-7601 (ext. 1237)
3 июня 2016 г., в 12:24, Martin Polednik <mpolednik@redhat.com> написал(а):
On 03/06/16 11:48 +0300, Дмитрий Глушенок wrote: > Hello! > > Is it possible to tell libvirt to add specific devices to qemu cgroup? By somehow enumerating the devices in XML using a hook for example. > I'm passing scsi-generic disks (/dev/sgX) to VM using qemucmdline hook and it doesn't work until I remove "devices" from cgroup_controllers in qemu.conf.
One way to achieve this is creating a hook to generate the scsi device XML instead of modifying qemu cmdline directly. Libvirt assumes ownership of all devices created in the XML and therefore adds them to the machine cgroup.
Example of the XML taken from [1]: <devices> <hostdev mode='subsystem' type='scsi' sgio='filtered' rawio='yes'> <source> <adapter name='scsi_host0'/> <address bus='0' target='0' unit='0'/> </source> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </hostdev> </devices>
There is slight issue with this approach outlined in [2].
If you want to keep the qemu approach, I think creating a custom partition and moving devices there would be the cleanest approach. In this case, [3] could help but I'm not entirely sure if that would solve the issue.
[1] https://libvirt.org/formatdomain.html [2] https://bugzilla.redhat.com/show_bug.cgi?id=1325485 [3] https://libvirt.org/cgroups.html > -- > Dmitry Glushenok > Jet Infosystems > http://www.jet.msk.su > +7-495-411-7601 (ext. 1237) > > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users
participants (2)
-
Martin Polednik -
Дмитрий Глушенок