Re: [ovirt-users] qemu cgroup_controllers

6 Jun 2016

      --Apple-Mail=_F5070364-047E-4F4A-BEE4-93C0242B9F73
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

So, libvirt behaves differently with custom partition? My assumption was =
that it will perform the same changes. I will have in mind in case oVirt =
4 will be released too late.
May be I wrote incorrectly - in oVirt 4.0 RC1 scsi pass-through already =
working.

--
Dmitry Glushenok
Jet Infosystems
http://www.jet.msk.su <http://www.jet.msk.su/>+7-495-411-7601 (ext. =
1237)
...
6 =D0=B8=D1=8E=D0=BD=D1=8F 2016 =D0=B3., =D0=B2 13:35, Martin Polednik =
<mpolednik@redhat.com> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB(=D0=B0):=

...
...
...
6 =D0=B8=D1=8E=D0=BD=D1=8F 2016 =D0=B3., =D0=B2 12:33, Martin =
Polednik <mpolednik@redhat.com> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB=
(=D0=B0):
=20
On 03/06/16 14:05 +0300, =D0=94=D0=BC=D0=B8=D1=82=D1=80=D0=B8=D0=B9 =
=D0=93=D0=BB=D1=83=D1=88=D0=B5=D0=BD=D0=BE=D0=BA wrote:
...
Thank you Martin!
=20
Actually I tried the workaround hook, provided in [2], but then =
VDSM (oVirt 3.6.6) tries to interpret hostdev in XML as PCI device, =
which leads to:
=20
::The vm start process failed
Traceback (most recent call last):
File "/usr/share/vdsm/virt/vm.py", line 703, in _startUnderlyingVm
 self._run()
File "/usr/share/vdsm/virt/vm.py", line 1949, in _run
 self._domDependentInit()
File "/usr/share/vdsm/virt/vm.py", line 1797, in _domDependentInit
 self._getUnderlyingVmDevicesInfo()
File "/usr/share/vdsm/virt/vm.py", line 1738, in =
_getUnderlyingVmDevicesInfo
 self._getUnderlyingHostDeviceInfo()
File "/usr/share/vdsm/virt/vm.py", line 4277, in =
_getUnderlyingHostDeviceInfo
 **self._getUnderlyingDeviceAddress(source))
TypeError: pci_address_to_name() got an unexpected keyword argument =
'target'
=20
XML part was:
<hostdev managed=3D"no" mode=3D"subsystem" rawio=3D"yes" =
type=3D"scsi">
  <source>
      <adapter name=3D"scsi_host2"/>
      <address bus=3D"0" target=3D"1" unit=3D"0"/>
  </source>
</hostdev>
<hostdev managed=3D"no" mode=3D"subsystem" rawio=3D"yes" =
type=3D"scsi">
  <source>
      <adapter name=3D"scsi_host2"/>
      <address bus=3D"0" target=3D"2" unit=3D"0"/>
  </source>
</hostdev>
=20
=20
As of creating custom partition - by default machine.slice has "a =
*:* rwm" in devices.list. But for every new VM libvirt removes *:* mask =
and fills the list with actually needed devices (as I understand the =
...
...
=20
c 136:* rw
c 1:3 rw
c 1:7 rw
c 1:5 rw
c 1:8 rw
c 1:9 rw
c 5:2 rw
c 10:232 rw
c 253:0 rw
c 10:228 rw
c 10:196 rw
=20
What I'm looking for is a way to tell libvirt about my additional =
devices without breaking oVirt.
=20
The solution would be creating your own partition and somehow (e.g.
VDSM hook) appending
=20
<resource>
 <partition>/machine/custom</partition>
</resource>
=20
to the libvirt's <domain> element. I'm not sure how feasible =
creating
your own partition is though. I've tried the process as follows:
=20
$ cat ~/create-partition.sh
# sh ~/create-partition.sh
for i in blkio cpu,cpuacct cpuset devices freezer memory net_cls =
...
do
 mkdir /sys/fs/cgroup/$i/machine.slice/custom.partition
done
=20
for i in cpuset.cpus cpuset.mems
do
 cat /sys/fs/cgroup/cpuset/machine.slice/$i > =
/sys/fs/cgroup/cpuset/machine.slice/custom.partition/$i
done
=20
(creates /machine/custom partition).
Now, we can create vdsm before_vm_start hook that will set given =
...
for the VM:
=20
$ pwd /usr/libexec/vdsm/hooks/before_vm_start
$ cat 10_cgroups
=20
#!/usr/bin/python
=20
import hooking
=20
=20
def custom_partition(domxml):
 resource =3D domxml.createElement('resource')
 partition =3D domxml.createElement('partition')
 partition_text =3D domxml.createTextNode('/machine/custom')
 partition.appendChild(partition_text)
 resource.appendChild(partition)
=20
 return resource
=20
domxml =3D hooking.read_domxml()
domain =3D domxml.getElementsByTagName('domain')[0]
domain.appendChild(custom_partition(domxml))
hooking.write_domxml(domxml)
=20
...
--
Dmitry Glushenok
Jet Infosystems
http://www.jet.msk.su
+7-495-411-7601 (ext. 1237)
=20
...
3 =D0=B8=D1=8E=D0=BD=D1=8F 2016 =D0=B3., =D0=B2 12:24, Martin =
Polednik <mpolednik@redhat.com> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB=
(=D0=B0):
=20
On 03/06/16 11:48 +0300, =D0=94=D0=BC=D0=B8=D1=82=D1=80=D0=B8=D0=B9 =
=D0=93=D0=BB=D1=83=D1=88=D0=B5=D0=BD=D0=BE=D0=BA wrote:
...
Hello!
=20
Is it possible to tell libvirt to add specific devices to qemu =
cgroup? By somehow enumerating the devices in XML using a hook for =
example.
I'm passing scsi-generic disks (/dev/sgX) to VM using qemucmdline =
hook and it doesn't work until I remove "devices" from =
cgroup_controllers in qemu.conf.
=20
One way to achieve this is creating a hook to generate the scsi =
device
XML instead of modifying qemu cmdline directly. Libvirt assumes
ownership of all devices created in the XML and therefore adds =
=20
process). For example:
perf_event
partition
them to
...
...
...
the machine cgroup.
=20
Example of the XML taken from [1]:
<devices>
<hostdev mode=3D'subsystem' type=3D'scsi' sgio=3D'filtered' =
rawio=3D'yes'>
    <source>
        <adapter name=3D'scsi_host0'/>
        <address bus=3D'0' target=3D'0' unit=3D'0'/>
    </source>
    <readonly/>
    <address type=3D'drive' controller=3D'0' bus=3D'0' target=3D'0' =
unit=3D'0'/>
</hostdev>
</devices>
=20
There is slight issue with this approach outlined in [2].
=20
If you want to keep the qemu approach, I think creating a custom
partition and moving devices there would be the cleanest approach. =
In
this case, [3] could help but I'm not entirely sure if that would
solve the issue.
=20
[1] https://libvirt.org/formatdomain.html
[2] https://bugzilla.redhat.com/show_bug.cgi?id=3D1325485
[3] https://libvirt.org/cgroups.html
...
--
Dmitry Glushenok
Jet Infosystems
http://www.jet.msk.su
+7-495-411-7601 (ext. 1237)
=20
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
=20
...
...
Hello Martin,
=20
Thank you for your time. It is clear how to create partition and =
assign it to a VM. But libvirt manipulates with contents of devices.list =
and I didn't find a way to put my devices into devices.list and stop =
=20
On 06/06/16 13:26 +0300, =D0=94=D0=BC=D0=B8=D1=82=D1=80=D0=B8=D0=B9 =
=D0=93=D0=BB=D1=83=D1=88=D0=B5=D0=BD=D0=BE=D0=BA wrote:
libvirt from removing them.
=20
That is, by the way, pretty interesting. I have tried it right now and
when using custom partition, libvirt doesn't touch my devices.list.
=20
It might make sense to file a libvirt bug.
=20
...
Anyway, it looks like that the best solution will be using oVirt 4.0 =
where scsi pass-through have been fixed.
=20
Hoping so!
=20
--
Dmitry Glushenok
Jet Infosystems
http://www.jet.msk.su <http://www.jet.msk.su/>+7-495-411-7601 (ext. =
--Apple-Mail=_F5070364-047E-4F4A-BEE4-93C0242B9F73
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><div class=3D"">So, libvirt behaves differently with custom =
partition? My assumption was that it will perform the same changes. I =
will have in mind in case oVirt 4 will be released too late.</div><div =
class=3D"">May be I wrote incorrectly - in oVirt 4.0 RC1 scsi =
pass-through already working.</div><div class=3D""><br =
class=3D""></div><div class=3D"">
<div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; orphans: =
auto; text-align: start; text-indent: 0px; text-transform: none; =
white-space: normal; widows: auto; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; =
orphans: auto; text-align: start; text-indent: 0px; text-transform: =
none; white-space: normal; widows: auto; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; =
orphans: auto; text-align: start; text-indent: 0px; text-transform: =
none; white-space: normal; widows: auto; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; =
orphans: auto; text-align: start; text-indent: 0px; text-transform: =
none; white-space: normal; widows: auto; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; =
orphans: auto; text-align: start; text-indent: 0px; text-transform: =
none; white-space: normal; widows: auto; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><div class=3D"">--</div><div class=3D""><div class=3D"" =
style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; =
-webkit-line-break: after-white-space;"><div class=3D"" =
style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; =
-webkit-line-break: after-white-space;"><div class=3D"" =
style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; =
-webkit-line-break: after-white-space;"><div class=3D"">Dmitry =
Glushenok</div><div class=3D"">Jet Infosystems</div><div class=3D""><a =
href=3D"http://www.jet.msk.su" =
class=3D"">http://www.jet.msk.su</a></div></div></div></div>+7-495-411-760=
1 (ext. 1237)</div></div></div></div></div></div>
</div>

<div><blockquote type=3D"cite" class=3D""><div class=3D"">6=
 =D0=B8=D1=8E=D0=BD=D1=8F 2016 =D0=B3., =D0=B2 13:35, Martin Polednik =
<<a href=3D"mailto:mpolednik@redhat.com" =
class=3D"">mpolednik@redhat.com</a>> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=
=D0=BB(=D0=B0):</div> <div =
class=3D"">On 06/06/16 13:26 +0300, =D0=94=D0=BC=D0=B8=D1=82=D1=80=D0=B8=D0=
=B9 =D0=93=D0=BB=D1=83=D1=88=D0=B5=D0=BD=D0=BE=D0=BA wrote: <blockquote type=3D"cite" class=3D"">Hello Martin, Thank you for your time. It is clear how to =
create partition and assign it to a VM. But libvirt manipulates with =
contents of devices.list and I didn't find a way to put my devices into =
devices.list and stop libvirt from removing them. </blockquote> That is, by the way, pretty =
interesting. I have tried it right now and when using =
custom partition, libvirt doesn't touch my devices.list. <br=
 class=3D"">It might make sense to file a libvirt bug. <blockquote type=3D"cite" class=3D"">Anyway, it looks like =
that the best solution will be using oVirt 4.0 where scsi pass-through =
have been fixed. </blockquote> Hoping so! <blockquote type=3D"cite" class=3D"">-- Dmitry Glushenok Jet Infosystems <a=
 href=3D"http://www.jet.msk.su" class=3D"">http://www.jet.msk.su</a> =
<<a href=3D"http://www.jet.msk.su/" =
class=3D"">http://www.jet.msk.su/</a>>+7-495-411-7601 (ext. 1237) <blockquote type=3D"cite" class=3D"">6 =
=D0=B8=D1=8E=D0=BD=D1=8F 2016 =D0=B3., =D0=B2 12:33, Martin Polednik =
<<a href=3D"mailto:mpolednik@redhat.com" =
class=3D"">mpolednik@redhat.com</a>> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=
=D0=BB(=D0=B0): On 03/06/16 14:05 +0300, =
=D0=94=D0=BC=D0=B8=D1=82=D1=80=D0=B8=D0=B9 =D0=93=D0=BB=D1=83=D1=88=D0=B5=D0=
=BD=D0=BE=D0=BA wrote: <blockquote type=3D"cite" =
class=3D"">Thank you Martin! Actually I =
tried the workaround hook, provided in [2], but then VDSM (oVirt 3.6.6) =
tries to interpret hostdev in XML as PCI device, which leads to: ::The vm start process failed Traceback (most recent call last): File =
"/usr/share/vdsm/virt/vm.py", line 703, in _startUnderlyingVm  self._run() File =
"/usr/share/vdsm/virt/vm.py", line 1949, in _run =
 self._domDependentInit() File =
"/usr/share/vdsm/virt/vm.py", line 1797, in _domDependentInit  self._getUnderlyingVmDevicesInfo() File =
"/usr/share/vdsm/virt/vm.py", line 1738, in =
_getUnderlyingVmDevicesInfo =
 self._getUnderlyingHostDeviceInfo() File =
"/usr/share/vdsm/virt/vm.py", line 4277, in =
_getUnderlyingHostDeviceInfo =
 **self._getUnderlyingDeviceAddress(source)) TypeError:=
 pci_address_to_name() got an unexpected keyword argument 'target' XML part was: <hostdev =
managed=3D"no" mode=3D"subsystem" rawio=3D"yes" type=3D"scsi"> 	=
<source> 		<adapter =
name=3D"scsi_host2"/> 		<address bus=3D"0" target=3D"1" =
unit=3D"0"/> 	</source> </hostdev> <hostdev managed=3D"no" =
mode=3D"subsystem" rawio=3D"yes" type=3D"scsi"> 	=
<source> 		<adapter =
name=3D"scsi_host2"/> 		<address bus=3D"0" target=3D"2" =
unit=3D"0"/> 	</source> </hostdev> As=
 of creating custom partition - by default machine.slice has "a *:* rwm" =
in devices.list. But for every new VM libvirt removes *:* mask and fills =
the list with actually needed devices (as I understand the process). For =
example: c 136:* rw c 1:3 =
rw c 1:7 rw c 1:5 rw c 1:8 =
rw c 1:9 rw c 5:2 rw c 10:232 =
rw c 253:0 rw c 10:228 rw c =
10:196 rw What I'm looking for is a way to =
tell libvirt about my additional devices without breaking oVirt. </blockquote> The solution would be creating =
your own partition and somehow (e.g. VDSM hook) =
appending <resource> =
 <partition>/machine/custom</partition> </resource> to the =
libvirt's <domain> element. I'm not sure how feasible creating your own partition is though. I've tried the process as =
follows: $ cat ~/create-partition.sh # sh ~/create-partition.sh for i in blkio =
cpu,cpuacct cpuset devices freezer memory net_cls perf_event do  mkdir =
/sys/fs/cgroup/$i/machine.slice/custom.partition done for i in cpuset.cpus cpuset.mems do  cat =
/sys/fs/cgroup/cpuset/machine.slice/$i > =
/sys/fs/cgroup/cpuset/machine.slice/custom.partition/$i done (creates /machine/custom =
partition). Now, we can create vdsm before_vm_start hook =
that will set given partition for the VM: $ pwd /usr/libexec/vdsm/hooks/before_vm_start $ =
cat 10_cgroups #!/usr/bin/python import hooking def custom_partition(domxml):  resource =3D=
 domxml.createElement('resource')  partition =3D =
domxml.createElement('partition')  partition_text =3D =
domxml.createTextNode('/machine/custom') =
 partition.appendChild(partition_text) =
 resource.appendChild(partition) =
 return resource domxml =3D =
hooking.read_domxml() domain =3D =
domxml.getElementsByTagName('domain')[0] domain.appendChild(custom_partition(domxml)) hooking.write_domxml(domxml) <blockquote type=3D"cite" class=3D"">-- Dmitry =
Glushenok Jet Infosystems <a =
href=3D"http://www.jet.msk.su" class=3D"">http://www.jet.msk.su</a> +7-495-411-7601 (ext. 1237) <blockquote type=3D"cite" class=3D"">3 =D0=B8=D1=8E=D0=BD=D1=8F=
 2016 =D0=B3., =D0=B2 12:24, Martin Polednik =
<mpolednik@redhat.com> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB(=D0=
=B0): On 03/06/16 11:48 +0300, =
=D0=94=D0=BC=D0=B8=D1=82=D1=80=D0=B8=D0=B9 =D0=93=D0=BB=D1=83=D1=88=D0=B5=D0=
=BD=D0=BE=D0=BA wrote: <blockquote type=3D"cite" =
class=3D"">Hello! Is it possible to tell =
libvirt to add specific devices to qemu cgroup? By somehow enumerating =
the devices in XML using a hook for example. I'm passing =
scsi-generic disks (/dev/sgX) to VM using qemucmdline hook and it =
doesn't work until I remove "devices" from cgroup_controllers in =
qemu.conf. </blockquote> One way to achieve =
this is creating a hook to generate the scsi device XML =
instead of modifying qemu cmdline directly. Libvirt assumes ownership of all devices created in the XML and therefore =
adds them to the machine cgroup. Example of the XML taken from [1]: <devices> <hostdev mode=3D'subsystem' =
type=3D'scsi' sgio=3D'filtered' rawio=3D'yes'> =
    <source> =
        <adapter =
name=3D'scsi_host0'/> =
        <address bus=3D'0' =
target=3D'0' unit=3D'0'/> =
    </source> =
    <readonly/> =
    <address type=3D'drive' controller=3D'0' =
bus=3D'0' target=3D'0' unit=3D'0'/> =
</hostdev> </devices> There is slight issue with this approach outlined in [2]. If you want to keep the qemu approach, I think =
creating a custom partition and moving devices there would =
be the cleanest approach. In this case, [3] could help but =
I'm not entirely sure if that would solve the issue. [1] https://libvirt.org/formatdomain.html [2] https://bugzilla.redhat.com/show_bug.cgi?id=3D1325485 [3] https://libvirt.org/cgroups.html <blockquote =
type=3D"cite" class=3D"">-- Dmitry Glushenok Jet Infosystems http://www.jet.msk.su +7-495-411-7601 (ext. 1237) _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users </blockquote></blockquote></blockquote></blockquote> </blockquote></div></blockquote></div> </body></html>=

--Apple-Mail=_F5070364-047E-4F4A-BEE4-93C0242B9F73--

Re: [ovirt-users] qemu cgroup_controllers

Дмитрий Глушенок