[ovirt-devel] [VDSM] health: Introduce Vdsm health monitoring
Nir Soffer
nsoffer at redhat.com
Fri Jan 15 09:22:41 UTC 2016
This test remove pthreading, using Python built-in threading, with
both https://gerrit.ovirt.org/51630 and https://gerrit.ovirt.org/51868.
pthreading is mostly useless on python 2.7, except when waiting
for threading.Condition.wait, however when using cpu affinity, the difference
is small, as can be seen here:
https://github.com/nirs/cthreading/wiki/performance#sleeplesspy
In that link we run 400 threads waiting on a single condition variable,
with pthreading we spend about 3% cpu, and with built-in threading about 7%.
Vdsm use now only 30-40 threads, so the cost of waiting on condition
variables should be insignificant.
Removing pthreading eliminates the pthreading leaked object as
expected, but we still leak constant number of objects on idle,
and about 6000 object per vm (start and stop).
# tail -f /var/log/vdsm/vdsm.log | grep health
Thread-15::DEBUG::2016-01-15
11:00:26,869::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
11:00:26,889::health::86::health::(_check) Collected 41 objects
Thread-15::WARNING::2016-01-15
11:00:26,889::health::100::health::(_check) Found 404 uncollectible
objects, 0 suspects implementing __del__: []
activating host...
Thread-15::DEBUG::2016-01-15
11:01:26,891::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
11:01:26,915::health::86::health::(_check) Collected 302 objects
Thread-15::WARNING::2016-01-15
11:01:26,919::health::100::health::(_check) Found 5790 uncollectible
objects, 0 suspects implementing __del__: []
starting vm...
Thread-15::DEBUG::2016-01-15
11:02:26,920::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
11:02:26,951::health::86::health::(_check) Collected 1686 objects
Thread-15::WARNING::2016-01-15
11:02:26,965::health::100::health::(_check) Found 10755 uncollectible
objects, 0 suspects implementing __del__: []
stopping vm...
Thread-15::DEBUG::2016-01-15
11:03:26,966::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
11:03:26,995::health::86::health::(_check) Collected 66 objects
Thread-15::WARNING::2016-01-15
11:03:27,003::health::100::health::(_check) Found 10818 uncollectible
objects, 0 suspects implementing __del__: []
vm shutting down...
Thread-15::DEBUG::2016-01-15
11:04:27,004::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
11:04:27,033::health::86::health::(_check) Collected 1702 objects
Thread-15::WARNING::2016-01-15
11:04:27,042::health::100::health::(_check) Found 12517 uncollectible
objects, 0 suspects implementing __del__: []
vm is stopped here...
Thread-15::DEBUG::2016-01-15
11:05:27,044::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
11:05:27,072::health::86::health::(_check) Collected 27 objects
Thread-15::WARNING::2016-01-15
11:05:27,090::health::100::health::(_check) Found 12541 uncollectible
objects, 0 suspects implementing __del__: []
no activitiy in engine...
Thread-15::DEBUG::2016-01-15
11:06:27,092::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
11:06:27,121::health::86::health::(_check) Collected 27 objects
Thread-15::WARNING::2016-01-15
11:06:27,131::health::100::health::(_check) Found 12589 uncollectible
objects, 0 suspects implementing __del__: []
Thread-15::DEBUG::2016-01-15
11:07:27,131::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
11:07:27,159::health::86::health::(_check) Collected 27 objects
Thread-15::WARNING::2016-01-15
11:07:27,169::health::100::health::(_check) Found 12613 uncollectible
objects, 0 suspects implementing __del__: []
starting vm...
Thread-15::DEBUG::2016-01-15
11:08:27,169::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
11:08:27,204::health::86::health::(_check) Collected 1695 objects
Thread-15::WARNING::2016-01-15
11:08:27,219::health::100::health::(_check) Found 17599 uncollectible
objects, 0 suspects implementing __del__: []
Thread-15::DEBUG::2016-01-15
11:09:27,221::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
11:09:27,255::health::86::health::(_check) Collected 63 objects
Thread-15::WARNING::2016-01-15
11:09:27,269::health::100::health::(_check) Found 17659 uncollectible
objects, 0 suspects implementing __del__: []
stopping vm...
Thread-15::DEBUG::2016-01-15
11:10:27,269::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
11:10:27,303::health::86::health::(_check) Collected 78 objects
Thread-15::WARNING::2016-01-15
11:10:27,317::health::100::health::(_check) Found 17734 uncollectible
objects, 0 suspects implementing __del__: []
vm is stopped here...
Thread-15::DEBUG::2016-01-15
11:11:27,318::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
11:11:27,352::health::86::health::(_check) Collected 1687 objects
Thread-15::WARNING::2016-01-15
11:11:27,368::health::100::health::(_check) Found 19451 uncollectible
objects, 0 suspects implementing __del__: []
On Fri, Jan 15, 2016 at 10:52 AM, Nir Soffer <nsoffer at redhat.com> wrote:
> I tested with https://gerrit.ovirt.org/#/c/51630
>
> libvirt.virDomain object does not leak any more, but pthreading object
> are still leaking,
> and the number of uncollectible objects is still very high (12,915).
>
> Each time we start and stop a vm, we leak about 6000 objects which are part
> of a reference cycle, and each time we create and leak 27 pthreading
> locks/conditions
> which are probably the reason for this leak (because they implement
> __del__ or reference object
> implementing __del__.
>
> When vdsm is idle, we leak 18 objects per minute, we don't know yet why.
>
> # tail -f /var/log/vdsm/vdsm.log | grep health
>
> Vdsm was connected to storage pool here, running as the spm...
>
> Thread-15::DEBUG::2016-01-15
> 10:00:00,159::health::84::health::(_check) Checking health
> Thread-15::DEBUG::2016-01-15
> 10:00:00,178::health::86::health::(_check) Collected 468 objects
> Thread-15::WARNING::2016-01-15
> 10:00:00,182::health::100::health::(_check) Found 5953 uncollectible
> objects, 0 suspects implementing __del__: []
> Thread-15::DEBUG::2016-01-15
> 10:01:00,182::health::84::health::(_check) Checking health
> Thread-15::DEBUG::2016-01-15
> 10:01:00,207::health::86::health::(_check) Collected 202 objects
> Thread-15::WARNING::2016-01-15
> 10:01:00,211::health::100::health::(_check) Found 6169 uncollectible
> objects, 23 suspects implementing __del__: ['<pthreading.Lock object
> at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>',
> '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object
> at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>',
> '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object
> at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>',
> '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at
> 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>',
> '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at
> 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>',
> '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at
> 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>',
> '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at
> 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>',
> '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at
> 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>']
>
> vm was started here...
>
> Thread-15::DEBUG::2016-01-15
> 10:02:00,212::health::84::health::(_check) Checking health
> Thread-15::DEBUG::2016-01-15
> 10:02:00,243::health::86::health::(_check) Collected 1689 objects
> Thread-15::WARNING::2016-01-15
> 10:02:00,252::health::100::health::(_check) Found 11107 uncollectible
> objects, 23 suspects implementing __del__: ['<pthreading.Lock object
> at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>',
> '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object
> at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>',
> '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object
> at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>',
> '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at
> 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>',
> '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at
> 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>',
> '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at
> 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>',
> '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at
> 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>',
> '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at
> 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>']
> Thread-15::DEBUG::2016-01-15
> 10:03:00,254::health::84::health::(_check) Checking health
> Thread-15::DEBUG::2016-01-15
> 10:03:00,285::health::86::health::(_check) Collected 57 objects
> Thread-15::WARNING::2016-01-15
> 10:03:00,294::health::100::health::(_check) Found 11162 uncollectible
> objects, 23 suspects implementing __del__: ['<pthreading.Lock object
> at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>',
> '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object
> at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>',
> '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object
> at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>',
> '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at
> 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>',
> '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at
> 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>',
> '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at
> 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>',
> '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at
> 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>',
> '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at
> 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>']
>
> vm was stopped here...
>
> Thread-15::DEBUG::2016-01-15
> 10:04:00,294::health::84::health::(_check) Checking health
> Thread-15::DEBUG::2016-01-15
> 10:04:00,324::health::86::health::(_check) Collected 1755 objects
> Thread-15::WARNING::2016-01-15
> 10:04:00,335::health::100::health::(_check) Found 12915 uncollectible
> objects, 50 suspects implementing __del__: ['<pthreading.Lock object
> at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>',
> '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object
> at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>',
> '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object
> at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>',
> '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at
> 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>',
> '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at
> 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>',
> '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at
> 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>',
> '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at
> 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>',
> '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at
> 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>',
> '<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object
> at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>',
> '<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object
> at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>',
> '<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object
> at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>',
> '<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at
> 0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>',
> '<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at
> 0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>',
> '<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at
> 0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>',
> '<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at
> 0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>',
> '<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at
> 0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>',
> '<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object
> at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>']
>
> Skipping some logs, no activity in engine...
>
> Thread-15::DEBUG::2016-01-15
> 10:23:01,082::health::84::health::(_check) Checking health
> Thread-15::DEBUG::2016-01-15
> 10:23:01,113::health::86::health::(_check) Collected 18 objects
> Thread-15::WARNING::2016-01-15
> 10:23:01,124::health::100::health::(_check) Found 13267 uncollectible
> objects, 50 suspects implementing __del__: ['<pthreading.Lock object
> at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>',
> '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object
> at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>',
> '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object
> at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>',
> '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at
> 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>',
> '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at
> 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>',
> '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at
> 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>',
> '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at
> 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>',
> '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at
> 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>',
> '<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object
> at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>',
> '<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object
> at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>',
> '<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object
> at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>',
> '<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at
> 0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>',
> '<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at
> 0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>',
> '<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at
> 0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>',
> '<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at
> 0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>',
> '<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at
> 0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>',
> '<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object
> at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>']
> Thread-15::DEBUG::2016-01-15
> 10:24:01,125::health::84::health::(_check) Checking health
> Thread-15::DEBUG::2016-01-15
> 10:24:01,156::health::86::health::(_check) Collected 18 objects
> Thread-15::WARNING::2016-01-15
> 10:24:01,167::health::100::health::(_check) Found 13283 uncollectible
> objects, 50 suspects implementing __del__: ['<pthreading.Lock object
> at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>',
> '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object
> at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>',
> '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object
> at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>',
> '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at
> 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>',
> '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at
> 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>',
> '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at
> 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>',
> '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at
> 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>',
> '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at
> 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>',
> '<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object
> at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>',
> '<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object
> at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>',
> '<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object
> at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>',
> '<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at
> 0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>',
> '<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at
> 0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>',
> '<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at
> 0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>',
> '<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at
> 0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>',
> '<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at
> 0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>',
> '<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object
> at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>']
>
> vm was started here...
>
> Thread-15::DEBUG::2016-01-15
> 10:25:01,167::health::84::health::(_check) Checking health
> Thread-15::DEBUG::2016-01-15
> 10:25:01,204::health::86::health::(_check) Collected 1537 objects
> Thread-15::WARNING::2016-01-15
> 10:25:01,219::health::100::health::(_check) Found 18264 uncollectible
> objects, 50 suspects implementing __del__: ['<pthreading.Lock object
> at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>',
> '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object
> at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>',
> '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object
> at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>',
> '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at
> 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>',
> '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at
> 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>',
> '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at
> 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>',
> '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at
> 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>',
> '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at
> 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>',
> '<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object
> at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>',
> '<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object
> at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>',
> '<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object
> at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>',
> '<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at
> 0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>',
> '<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at
> 0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>',
> '<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at
> 0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>',
> '<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at
> 0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>',
> '<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at
> 0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>',
> '<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object
> at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>']
>
> vm was ask to stop here...
>
> Thread-15::DEBUG::2016-01-15
> 10:26:01,219::health::84::health::(_check) Checking health
> Thread-15::DEBUG::2016-01-15
> 10:26:01,255::health::86::health::(_check) Collected 57 objects
> Thread-15::WARNING::2016-01-15
> 10:26:01,270::health::100::health::(_check) Found 18319 uncollectible
> objects, 50 suspects implementing __del__: ['<pthreading.Lock object
> at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>',
> '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object
> at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>',
> '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object
> at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>',
> '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at
> 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>',
> '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at
> 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>',
> '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at
> 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>',
> '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at
> 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>',
> '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at
> 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>',
> '<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object
> at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>',
> '<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object
> at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>',
> '<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object
> at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>',
> '<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at
> 0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>',
> '<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at
> 0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>',
> '<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at
> 0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>',
> '<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at
> 0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>',
> '<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at
> 0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>',
> '<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object
> at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>']
>
> vm was shutdown here...
>
> Thread-15::DEBUG::2016-01-15
> 10:27:01,270::health::84::health::(_check) Checking health
> Thread-15::DEBUG::2016-01-15
> 10:27:01,306::health::86::health::(_check) Collected 1752 objects
> Thread-15::WARNING::2016-01-15
> 10:27:01,322::health::100::health::(_check) Found 20069 uncollectible
> objects, 77 suspects implementing __del__: ['<pthreading.Lock object
> at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>',
> '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object
> at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>',
> '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object
> at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>',
> '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at
> 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>',
> '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at
> 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>',
> '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at
> 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>',
> '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at
> 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>',
> '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at
> 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>',
> '<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object
> at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>',
> '<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object
> at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>',
> '<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object
> at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>',
> '<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at
> 0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>',
> '<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at
> 0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>',
> '<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at
> 0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>',
> '<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at
> 0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>',
> '<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at
> 0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>',
> '<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object
> at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>',
> '<pthreading.Lock object at 0x7fa904078710>', '<pthreading.Lock object
> at 0x7fa9b40af440>', '<pthreading.Lock object at 0x7fa90405b200>',
> '<pthreading.Lock object at 0x7fa904078998>', '<pthreading.Lock object
> at 0x7fa9a407cfc8>', '<pthreading.Lock object at 0x7fa904078638>',
> '<pthreading.Lock object at 0x7fa904078560>', '<pthreading.Lock object
> at 0x7fa904078200>', '<pthread.Cond object at 0x7fa9a4142d10>',
> '<pthreading.Lock object at 0x7fa9040785f0>', '<pthread.Cond object at
> 0x7fa9b40d1a50>', '<pthreading.Lock object at 0x7fa90405b9e0>',
> '<pthread.Cond object at 0x7fa9a4142610>', '<pthreading.Lock object at
> 0x7fa9040787a0>', '<pthread.Cond object at 0x7fa9b40d1f10>',
> '<pthreading.Lock object at 0x7fa9040786c8>', '<pthread.Cond object at
> 0x7fa9a4142b10>', '<pthreading.Lock object at 0x7fa904078368>',
> '<pthread.Cond object at 0x7fa9b40d1790>', '<pthreading.Lock object at
> 0x7fa904078290>', '<pthread.Cond object at 0x7fa9b40d14d0>',
> '<pthreading.Lock object at 0x7fa904078248>', '<pthread.Cond object at
> 0x7fa9b40d1150>', '<pthread.Cond object at 0x7fa9a41360d0>',
> '<pthreading.Lock object at 0x7fa9441a8d40>', '<pthreading.Lock object
> at 0x7fa944049b00>', '<pthread.Cond object at 0x7fa9a4136410>']
>
> On Thu, Jan 14, 2016 at 8:00 PM, Nir Soffer <nsoffer at redhat.com> wrote:
>> Here are example logs with running and stopping one vm:
>>
>> # tail -f /var/log/vdsm/vdsm.log | grep health
>> Thread-13::DEBUG::2016-01-14
>> 19:42:04,625::health::84::health::(_check) Checking health
>> Thread-13::DEBUG::2016-01-14
>> 19:42:04,651::health::86::health::(_check) Collected 25 objects
>> Thread-13::WARNING::2016-01-14
>> 19:42:04,656::health::100::health::(_check) Found 6085 uncollectible
>> objects, 0 suspects implementing __del__: []
>>
>> Vm was started here...
>>
>> Thread-13::DEBUG::2016-01-14
>> 19:43:04,657::health::84::health::(_check) Checking health
>> Thread-13::DEBUG::2016-01-14
>> 19:43:04,688::health::86::health::(_check) Collected 1680 objects
>> Thread-13::WARNING::2016-01-14
>> 19:43:04,697::health::100::health::(_check) Found 11041 uncollectible
>> objects, 0 suspects implementing __del__: []
>> Thread-13::DEBUG::2016-01-14
>> 19:44:04,698::health::84::health::(_check) Checking health
>> Thread-13::DEBUG::2016-01-14
>> 19:44:04,729::health::86::health::(_check) Collected 57 objects
>> Thread-13::WARNING::2016-01-14
>> 19:44:04,737::health::100::health::(_check) Found 11096 uncollectible
>> objects, 0 suspects implementing __del__: []
>> Thread-13::DEBUG::2016-01-14
>> 19:45:04,738::health::84::health::(_check) Checking health
>> Thread-13::DEBUG::2016-01-14
>> 19:45:04,769::health::86::health::(_check) Collected 69 objects
>> Thread-13::WARNING::2016-01-14
>> 19:45:04,778::health::100::health::(_check) Found 11163 uncollectible
>> objects, 0 suspects implementing __del__: []
>>
>> Vm was shutdown here...
>>
>> Thread-13::DEBUG::2016-01-14
>> 19:46:04,779::health::84::health::(_check) Checking health
>> Thread-13::DEBUG::2016-01-14
>> 19:46:04,809::health::86::health::(_check) Collected 2272 objects
>> Thread-13::WARNING::2016-01-14
>> 19:46:04,819::health::100::health::(_check) Found 13464 uncollectible
>> objects, 28 suspects implementing __del__: ['<pthreading.Lock object
>> at 0x7f1d8016f908>', '<pthreading.Lock object at 0x7f1d801804d0>',
>> '<pthreading.Lock object at 0x7f1cf8010f38>', '<pthreading.Lock object
>> at 0x7f1d8016f7a0>', '<pthreading.Lock object at 0x7f1d20003680>',
>> '<pthreading.Lock object at 0x7f1da8112998>', '<pthreading.Lock object
>> at 0x7f1d8016fa28>', '<pthreading.Lock object at 0x7f1da8112830>',
>> '<pthread.Cond object at 0x7f1da811cf50>', '<pthreading.Lock object at
>> 0x7f1d8016f200>', '<pthread.Cond object at 0x7f1da811ca50>',
>> '<pthreading.Lock object at 0x7f1cf8010fc8>', '<pthread.Cond object at
>> 0x7f1da8115c50>', '<pthreading.Lock object at 0x7f1da8112878>',
>> '<pthread.Cond object at 0x7f1da811c550>', '<pthreading.Lock object at
>> 0x7f1d8016f9e0>', '<pthread.Cond object at 0x7f1da811ca90>',
>> '<pthreading.Lock object at 0x7f1d8016f680>', '<pthread.Cond object at
>> 0x7f1da811cc50>', '<libvirt.virDomain object at 0x1ba8210>',
>> '<pthreading.Lock object at 0x7f1d8016f758>', '<pthread.Cond object at
>> 0x7f1da811cb50>', '<pthreading.Lock object at 0x7f1d8016fb90>',
>> '<pthread.Cond object at 0x7f1da811cc10>', '<pthread.Cond object at
>> 0x233ed50>', '<pthreading.Lock object at 0x7f1ce0227c20>',
>> '<pthreading.Lock object at 0x7f1ce0227ab8>', '<pthread.Cond object at
>> 0x233ed10>']
>>
>> Need to test with should be fixed by https://gerrit.ovirt.org/51630,
>> which should
>> eliminate '<libvirt.virDomain object at 0x1ba8210>', and maybe also
>> the pthreading
>> objects.
>>
>> Nir
>>
>> On Thu, Jan 14, 2016 at 7:38 PM, Nir Soffer <nsoffer at redhat.com> wrote:
>>> Hi all,
>>>
>>> Continuing the leak investigation Francesco and Milan are working on,
>>> I posted this patch, adding leak health monitoring to vdsm [1]. This patch
>>> currently monitor collectible objects, and we may add other interesting
>>> stuff later if needed.
>>>
>>> To enable the monitor, you must set:
>>>
>>> [devel]
>>> health_monitor_enable = true
>>>
>>> In vdsm.conf, and restart vdsm.
>>>
>>> Here is an example logs - this was logged 60 seconds after starting vdsm,
>>> when running as spm:
>>>
>>> Thread-13::DEBUG::2016-01-14
>>> 18:53:43,239::health::84::health::(_check) Checking health
>>> Thread-13::DEBUG::2016-01-14
>>> 18:53:43,272::health::86::health::(_check) Collected 460 objects
>>> Thread-13::WARNING::2016-01-14
>>> 18:53:43,277::health::100::health::(_check) Found 5873 uncollectible
>>> objects, 10 suspects implementing __del__: ['<pthreading.Lock object
>>> at 0x7f012c115320>', '<pthreading.Lock object at 0x7f012c115440>',
>>> '<pthreading.Lock object at 0x7f012c1153b0>', '<pthreading.Lock object
>>> at 0x7f012c1154d0>', '<pthread.Cond object at 0x7f012c111350>',
>>> '<pthread.Cond object at 0x7f012c1112d0>', '<pthread.Cond object at
>>> 0x7f012c111250>', '<pthread.Cond object at 0x7f012c111850>',
>>> '<pthread.Cond object at 0x7f012c1117d0>', '<pthread.Cond object at
>>> 0x7f012c111750>']
>>>
>>> We found 460 uncollectible objects, 10 of them are objects implementing __del__.
>>> Having such object in a reference cycle make the entire cycle uncollectible.
>>>
>>> The pthreading objects in this log grow as vdsm continue to run, even without
>>> any interaction with engine. The pthreading objects leak is caused by a cycle
>>> in storage.misc.RWLock, fixed in patch [2].
>>>
>>> After solving the pthreading leak, we still have many uncollectible
>>> objects in the logs,
>>> listing them, it seems that the suspect is hiding inside a collection
>>> (dict, list, tuple)
>>> and does not appear in the gc.garbage list. I will address this in later patch,
>>> searching in the entire object graph instead of the top level of gc.garbage.
>>>
>>> See end of this mail for list of these objects. I printed them like this:
>>>
>>> Enable manhole debugging shell:
>>>
>>> [devel]
>>> manhole_enable = true
>>>
>>> Connect to vdsm using manhole:
>>>
>>> nc -U /run/vdsm/vdsmd.manhole
>>>
>>> Print garbage:
>>>
>>>>>> import pprint
>>>>>> import gc
>>>>>> with open('/tmp/garbage.txt', 'w') as f:
>>> pprint.pprint(gc.garbage, stream=f, indent=4)
>>>
>>> These objects looks related to libvirt - Francesco, can you take a look?
>>>
>>> [1] https://gerrit.ovirt.org/51708
>>> [2] https://gerrit.ovirt.org/51868
>>>
>>> Nir
>>>
>>> --- garbage after patch [2] ---
>>>
>>> [ (<type '_ctypes.Array'>,),
>>> { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>,
>>> '__doc__': None,
>>> '__module__': 'vdsm.netlink',
>>> '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>,
>>> '_length_': 60,
>>> '_type_': <class 'ctypes.c_char'>,
>>> 'raw': <attribute 'raw' of 'c_char_Array_60' objects>,
>>> 'value': <attribute 'value' of 'c_char_Array_60' objects>},
>>> <class 'vdsm.netlink.c_char_Array_60'>,
>>> <attribute '__dict__' of 'c_char_Array_60' objects>,
>>> <attribute '__weakref__' of 'c_char_Array_60' objects>,
>>> ( <class 'vdsm.netlink.c_char_Array_60'>,
>>> <type '_ctypes.Array'>,
>>> <type '_ctypes._CData'>,
>>> <type 'object'>),
>>> <attribute 'raw' of 'c_char_Array_60' objects>,
>>> <attribute 'value' of 'c_char_Array_60' objects>,
>>> <cell at 0x23662f0: bool object at 0x7f1e086c8c40>,
>>> <cell at 0x2366328: NoneType object at 0x7f1e086dcf00>,
>>> <cell at 0x2366360: function object at 0x233f488>,
>>> ( <cell at 0x23662f0: bool object at 0x7f1e086c8c40>,
>>> <cell at 0x2366328: NoneType object at 0x7f1e086dcf00>,
>>> <cell at 0x2366360: function object at 0x233f488>),
>>> <function wrapMethod at 0x233f488>,
>>> [ [ <Recursion on list with id=34033464>,
>>> <Recursion on list with id=34033464>,
>>> 'class'],
>>> [ <Recursion on list with id=34033464>,
>>> <Recursion on list with id=34033464>,
>>> 'class'],
>>> None],
>>> [ [ <Recursion on list with id=33979064>,
>>> <Recursion on list with id=33979064>,
>>> None],
>>> [ <Recursion on list with id=33979064>,
>>> <Recursion on list with id=33979064>,
>>> None],
>>> 'class'],
>>> [ [ <Recursion on list with id=35535256>,
>>> <Recursion on list with id=35535256>,
>>> 'class'],
>>> [ <Recursion on list with id=35535256>,
>>> <Recursion on list with id=35535256>,
>>> 'class'],
>>> None],
>>> [ [ <Recursion on list with id=35535832>,
>>> <Recursion on list with id=35535832>,
>>> None],
>>> [ <Recursion on list with id=35535832>,
>>> <Recursion on list with id=35535832>,
>>> None],
>>> 'class'],
>>> [ [ <Recursion on list with id=35556312>,
>>> <Recursion on list with id=35556312>,
>>> 'class'],
>>> [ <Recursion on list with id=35556312>,
>>> <Recursion on list with id=35556312>,
>>> 'class'],
>>> None],
>>> [ [ <Recursion on list with id=35556600>,
>>> <Recursion on list with id=35556600>,
>>> None],
>>> [ <Recursion on list with id=35556600>,
>>> <Recursion on list with id=35556600>,
>>> None],
>>> 'class'],
>>> [ [ <Recursion on list with id=35556960>,
>>> <Recursion on list with id=35556960>,
>>> 'class'],
>>> [ <Recursion on list with id=35556960>,
>>> <Recursion on list with id=35556960>,
>>> 'class'],
>>> None],
>>> [ [ <Recursion on list with id=35561912>,
>>> <Recursion on list with id=35561912>,
>>> None],
>>> [ <Recursion on list with id=35561912>,
>>> <Recursion on list with id=35561912>,
>>> None],
>>> 'class'],
>>> [ [ <Recursion on list with id=35583904>,
>>> <Recursion on list with id=35583904>,
>>> 'class'],
>>> [ <Recursion on list with id=35583904>,
>>> <Recursion on list with id=35583904>,
>>> 'class'],
>>> None],
>>> [ [ <Recursion on list with id=35584912>,
>>> <Recursion on list with id=35584912>,
>>> None],
>>> [ <Recursion on list with id=35584912>,
>>> <Recursion on list with id=35584912>,
>>> None],
>>> 'class'],
>>> [ [ <Recursion on list with id=36340368>,
>>> <Recursion on list with id=36340368>,
>>> 'class'],
>>> [ <Recursion on list with id=36340368>,
>>> <Recursion on list with id=36340368>,
>>> 'class'],
>>> None],
>>> [ [ <Recursion on list with id=36340872>,
>>> <Recursion on list with id=36340872>,
>>> None],
>>> [ <Recursion on list with id=36340872>,
>>> <Recursion on list with id=36340872>,
>>> None],
>>> 'class'],
>>> [ [ <Recursion on list with id=36364872>,
>>> <Recursion on list with id=36364872>,
>>> 'class'],
>>> [ <Recursion on list with id=36364872>,
>>> <Recursion on list with id=36364872>,
>>> 'class'],
>>> None],
>>> [ [ <Recursion on list with id=36365520>,
>>> <Recursion on list with id=36365520>,
>>> None],
>>> [ <Recursion on list with id=36365520>,
>>> <Recursion on list with id=36365520>,
>>> None],
>>> 'class'],
>>> [ [ <Recursion on list with id=36457216>,
>>> <Recursion on list with id=36457216>,
>>> 'class'],
>>> [ <Recursion on list with id=36457216>,
>>> <Recursion on list with id=36457216>,
>>> 'class'],
>>> None],
>>> [ [ <Recursion on list with id=36458008>,
>>> <Recursion on list with id=36458008>,
>>> None],
>>> [ <Recursion on list with id=36458008>,
>>> <Recursion on list with id=36458008>,
>>> None],
>>> 'class'],
>>> [ [ <Recursion on list with id=36460520>,
>>> <Recursion on list with id=36460520>,
>>> 'class'],
>>> [ <Recursion on list with id=36460520>,
>>> <Recursion on list with id=36460520>,
>>> 'class'],
>>> None],
>>> [ [ <Recursion on list with id=36461168>,
>>> <Recursion on list with id=36461168>,
>>> None],
>>> [ <Recursion on list with id=36461168>,
>>> <Recursion on list with id=36461168>,
>>> None],
>>> 'class'],
>>> [ [ <Recursion on list with id=36562344>,
>>> <Recursion on list with id=36562344>,
>>> 'class'],
>>> [ <Recursion on list with id=36562344>,
>>> <Recursion on list with id=36562344>,
>>> 'class'],
>>> None],
>>> [ [ <Recursion on list with id=36563280>,
>>> <Recursion on list with id=36563280>,
>>> None],
>>> [ <Recursion on list with id=36563280>,
>>> <Recursion on list with id=36563280>,
>>> None],
>>> 'class'],
>>> <Element 'memory' at 0x2364710>,
>>> { '_children': [],
>>> 'attrib': { 'unit': 'KiB'},
>>> 'tag': 'memory',
>>> 'tail': '\n ',
>>> 'text': '4095476'},
>>> [],
>>> <Element 'pages' at 0x2364750>,
>>> { '_children': [],
>>> 'attrib': { 'size': '4', 'unit': 'KiB'},
>>> 'tag': 'pages',
>>> 'tail': '\n ',
>>> 'text': '1023869'},
>>> [],
>>> <Element 'pages' at 0x2364790>,
>>> { '_children': [],
>>> 'attrib': { 'size': '2048', 'unit': 'KiB'},
>>> 'tag': 'pages',
>>> 'tail': '\n ',
>>> 'text': '0'},
>>> [],
>>> <Element 'distances' at 0x23647d0>,
>>> { '_children': [<Element 'sibling' at 0x2364810>],
>>> 'attrib': { },
>>> 'tag': 'distances',
>>> 'tail': '\n ',
>>> 'text': '\n '},
>>> [<Element 'sibling' at 0x2364810>],
>>> <Element 'sibling' at 0x2364810>,
>>> { '_children': [],
>>> 'attrib': { 'id': '0', 'value': '10'},
>>> 'tag': 'sibling',
>>> 'tail': '\n '},
>>> [],
>>> <Element 'cpus' at 0x2364850>,
>>> { '_children': [ <Element 'cpu' at 0x2364890>,
>>> <Element 'cpu' at 0x23648d0>,
>>> <Element 'cpu' at 0x2364910>,
>>> <Element 'cpu' at 0x2364950>,
>>> <Element 'cpu' at 0x2364990>,
>>> <Element 'cpu' at 0x23649d0>,
>>> <Element 'cpu' at 0x2364a10>,
>>> <Element 'cpu' at 0x2364a50>],
>>> 'attrib': { 'num': '8'},
>>> 'tag': 'cpus',
>>> 'tail': '\n ',
>>> 'text': '\n '},
>>> [ <Element 'cpu' at 0x2364890>,
>>> <Element 'cpu' at 0x23648d0>,
>>> <Element 'cpu' at 0x2364910>,
>>> <Element 'cpu' at 0x2364950>,
>>> <Element 'cpu' at 0x2364990>,
>>> <Element 'cpu' at 0x23649d0>,
>>> <Element 'cpu' at 0x2364a10>,
>>> <Element 'cpu' at 0x2364a50>],
>>> <Element 'cpu' at 0x2364890>,
>>> { '_children': [],
>>> 'attrib': { 'core_id': '0',
>>> 'id': '0',
>>> 'siblings': '0',
>>> 'socket_id': '0'},
>>> 'tag': 'cpu',
>>> 'tail': '\n '},
>>> [],
>>> <Element 'cpu' at 0x23648d0>,
>>> { '_children': [],
>>> 'attrib': { 'core_id': '0',
>>> 'id': '1',
>>> 'siblings': '1',
>>> 'socket_id': '1'},
>>> 'tag': 'cpu',
>>> 'tail': '\n '},
>>> [],
>>> <Element 'cpu' at 0x2364910>,
>>> { '_children': [],
>>> 'attrib': { 'core_id': '0',
>>> 'id': '2',
>>> 'siblings': '2',
>>> 'socket_id': '2'},
>>> 'tag': 'cpu',
>>> 'tail': '\n '},
>>> [],
>>> <Element 'cpu' at 0x2364950>,
>>> { '_children': [],
>>> 'attrib': { 'core_id': '0',
>>> 'id': '3',
>>> 'siblings': '3',
>>> 'socket_id': '3'},
>>> 'tag': 'cpu',
>>> 'tail': '\n '},
>>> [],
>>> <Element 'cpu' at 0x2364990>,
>>> { '_children': [],
>>> 'attrib': { 'core_id': '0',
>>> 'id': '4',
>>> 'siblings': '4',
>>> 'socket_id': '4'},
>>> 'tag': 'cpu',
>>> 'tail': '\n '},
>>> [],
>>> <Element 'cpu' at 0x23649d0>,
>>> { '_children': [],
>>> 'attrib': { 'core_id': '0',
>>> 'id': '5',
>>> 'siblings': '5',
>>> 'socket_id': '5'},
>>> 'tag': 'cpu',
>>> 'tail': '\n '},
>>> [],
>>> <Element 'cpu' at 0x2364a10>,
>>> { '_children': [],
>>> 'attrib': { 'core_id': '0',
>>> 'id': '6',
>>> 'siblings': '6',
>>> 'socket_id': '6'},
>>> 'tag': 'cpu',
>>> 'tail': '\n '},
>>> [],
>>> <Element 'cpu' at 0x2364a50>,
>>> { '_children': [],
>>> 'attrib': { 'core_id': '0',
>>> 'id': '7',
>>> 'siblings': '7',
>>> 'socket_id': '7'},
>>> 'tag': 'cpu',
>>> 'tail': '\n '},
>>> [],
>>> <Element 'secmodel' at 0x2364a90>,
>>> { '_children': [ <Element 'model' at 0x2364ad0>,
>>> <Element 'doi' at 0x2364b10>,
>>> <Element 'baselabel' at 0x2364b50>,
>>> <Element 'baselabel' at 0x2364b90>],
>>> 'attrib': { },
>>> 'tag': 'secmodel',
>>> 'tail': '\n ',
>>> 'text': '\n '},
>>> [ <Element 'model' at 0x2364ad0>,
>>> <Element 'doi' at 0x2364b10>,
>>> <Element 'baselabel' at 0x2364b50>,
>>> <Element 'baselabel' at 0x2364b90>],
>>> <Element 'model' at 0x2364ad0>,
>>> { '_children': [],
>>> 'attrib': { },
>>> 'tag': 'model',
>>> 'tail': '\n ',
>>> 'text': 'selinux'},
>>> [],
>>> <Element 'doi' at 0x2364b10>,
>>> { '_children': [],
>>> 'attrib': { },
>>> 'tag': 'doi',
>>> 'tail': '\n ',
>>> 'text': '0'},
>>> [],
>>> <Element 'baselabel' at 0x2364b50>,
>>> { '_children': [],
>>> 'attrib': { 'type': 'kvm'},
>>> 'tag': 'baselabel',
>>> 'tail': '\n ',
>>> 'text': 'system_u:system_r:svirt_t:s0'},
>>> [],
>>> <Element 'baselabel' at 0x2364b90>,
>>> { '_children': [],
>>> 'attrib': { 'type': 'qemu'},
>>> 'tag': 'baselabel',
>>> 'tail': '\n ',
>>> 'text': 'system_u:system_r:svirt_tcg_t:s0'},
>>> [],
>>> <Element 'secmodel' at 0x2364bd0>,
>>> { '_children': [ <Element 'model' at 0x2364c10>,
>>> <Element 'doi' at 0x2364c50>,
>>> <Element 'baselabel' at 0x2364c90>,
>>> <Element 'baselabel' at 0x2364cd0>],
>>> 'attrib': { },
>>> 'tag': 'secmodel',
>>> 'tail': '\n ',
>>> 'text': '\n '},
>>> [ <Element 'model' at 0x2364c10>,
>>> <Element 'doi' at 0x2364c50>,
>>> <Element 'baselabel' at 0x2364c90>,
>>> <Element 'baselabel' at 0x2364cd0>],
>>> <Element 'model' at 0x2364c10>,
>>> { '_children': [],
>>> 'attrib': { },
>>> 'tag': 'model',
>>> 'tail': '\n ',
>>> 'text': 'dac'},
>>> [],
>>> <Element 'doi' at 0x2364c50>,
>>> { '_children': [],
>>> 'attrib': { },
>>> 'tag': 'doi',
>>> 'tail': '\n ',
>>> 'text': '0'},
>>> [],
>>> <Element 'baselabel' at 0x2364c90>,
>>> { '_children': [],
>>> 'attrib': { 'type': 'kvm'},
>>> 'tag': 'baselabel',
>>> 'tail': '\n ',
>>> 'text': '+107:+107'},
>>> [],
>>> <Element 'baselabel' at 0x2364cd0>,
>>> { '_children': [],
>>> 'attrib': { 'type': 'qemu'},
>>> 'tag': 'baselabel',
>>> 'tail': '\n ',
>>> 'text': '+107:+107'},
>>> [],
>>> <Element 'guest' at 0x2364d10>,
>>> { '_children': [ <Element 'os_type' at 0x2364d50>,
>>> <Element 'arch' at 0x2364d90>,
>>> <Element 'features' at 0x2369390>],
>>> 'attrib': { },
>>> 'tag': 'guest',
>>> 'tail': '\n\n ',
>>> 'text': '\n '},
>>> [ <Element 'os_type' at 0x2364d50>,
>>> <Element 'arch' at 0x2364d90>,
>>> <Element 'features' at 0x2369390>],
>>> <Element 'os_type' at 0x2364d50>,
>>> { '_children': [],
>>> 'attrib': { },
>>> 'tag': 'os_type',
>>> 'tail': '\n ',
>>> 'text': 'hvm'},
>>> [],
>>> <Element 'arch' at 0x2364d90>,
>>> { '_children': [ <Element 'wordsize' at 0x2364dd0>,
>>> <Element 'emulator' at 0x2364e10>,
>>> <Element 'machine' at 0x2364e90>,
>>> <Element 'machine' at 0x2364ed0>,
>>> <Element 'machine' at 0x2364f10>,
>>> <Element 'machine' at 0x2364f50>,
>>> <Element 'machine' at 0x2364f90>,
>>> <Element 'machine' at 0x2364fd0>,
>>> <Element 'machine' at 0x2369050>,
>>> <Element 'machine' at 0x2369090>,
>>> <Element 'machine' at 0x23690d0>,
>>> <Element 'machine' at 0x2369110>,
>>> <Element 'machine' at 0x2369150>,
>>> <Element 'machine' at 0x2369190>,
>>> <Element 'machine' at 0x23691d0>,
>>> <Element 'machine' at 0x2369210>,
>>> <Element 'machine' at 0x2369250>,
>>> <Element 'domain' at 0x2369290>,
>>> <Element 'domain' at 0x23692d0>],
>>> 'attrib': { 'name': 'i686'},
>>> 'tag': 'arch',
>>> 'tail': '\n ',
>>> 'text': '\n '},
>>> [ <Element 'wordsize' at 0x2364dd0>,
>>> <Element 'emulator' at 0x2364e10>,
>>> <Element 'machine' at 0x2364e90>,
>>> <Element 'machine' at 0x2364ed0>,
>>> <Element 'machine' at 0x2364f10>,
>>> <Element 'machine' at 0x2364f50>,
>>> <Element 'machine' at 0x2364f90>,
>>> <Element 'machine' at 0x2364fd0>,
>>> <Element 'machine' at 0x2369050>,
>>> <Element 'machine' at 0x2369090>,
>>> <Element 'machine' at 0x23690d0>,
>>> <Element 'machine' at 0x2369110>,
>>> <Element 'machine' at 0x2369150>,
>>> <Element 'machine' at 0x2369190>,
>>> <Element 'machine' at 0x23691d0>,
>>> <Element 'machine' at 0x2369210>,
>>> <Element 'machine' at 0x2369250>,
>>> <Element 'domain' at 0x2369290>,
>>> <Element 'domain' at 0x23692d0>],
>>> <Element 'wordsize' at 0x2364dd0>,
>>> { '_children': [],
>>> 'attrib': { },
>>> 'tag': 'wordsize',
>>> 'tail': '\n ',
>>> 'text': '32'},
>>> [],
>>> <Element 'emulator' at 0x2364e10>,
>>> { '_children': [],
>>> 'attrib': { },
>>> 'tag': 'emulator',
>>> 'tail': '\n ',
>>> 'text': '/usr/libexec/qemu-kvm'},
>>> [],
>>> <Element 'machine' at 0x2364e90>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'pc-i440fx-rhel7.2.0'},
>>> [],
>>> <Element 'machine' at 0x2364ed0>,
>>> { '_children': [],
>>> 'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'pc'},
>>> [],
>>> <Element 'machine' at 0x2364f10>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'pc-i440fx-rhel7.0.0'},
>>> [],
>>> <Element 'machine' at 0x2364f50>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'pc-q35-rhel7.1.0'},
>>> [],
>>> <Element 'machine' at 0x2364f90>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'rhel6.3.0'},
>>> [],
>>> <Element 'machine' at 0x2364fd0>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'pc-q35-rhel7.2.0'},
>>> [],
>>> <Element 'machine' at 0x2369050>,
>>> { '_children': [],
>>> 'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'q35'},
>>> [],
>>> <Element 'machine' at 0x2369090>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'rhel6.4.0'},
>>> [],
>>> <Element 'machine' at 0x23690d0>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'rhel6.0.0'},
>>> [],
>>> <Element 'machine' at 0x2369110>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'pc-i440fx-rhel7.1.0'},
>>> [],
>>> <Element 'machine' at 0x2369150>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'rhel6.5.0'},
>>> [],
>>> <Element 'machine' at 0x2369190>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'rhel6.6.0'},
>>> [],
>>> <Element 'machine' at 0x23691d0>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'rhel6.1.0'},
>>> [],
>>> <Element 'machine' at 0x2369210>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'pc-q35-rhel7.0.0'},
>>> [],
>>> <Element 'machine' at 0x2369250>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'rhel6.2.0'},
>>> [],
>>> <Element 'domain' at 0x2369290>,
>>> { '_children': [],
>>> 'attrib': { 'type': 'qemu'},
>>> 'tag': 'domain',
>>> 'tail': '\n '},
>>> [],
>>> <Element 'domain' at 0x23692d0>,
>>> { '_children': [<Element 'emulator' at 0x2369310>],
>>> 'attrib': { 'type': 'kvm'},
>>> 'tag': 'domain',
>>> 'tail': '\n ',
>>> 'text': '\n '},
>>> [<Element 'emulator' at 0x2369310>],
>>> <Element 'emulator' at 0x2369310>,
>>> { '_children': [],
>>> 'attrib': { },
>>> 'tag': 'emulator',
>>> 'tail': '\n ',
>>> 'text': '/usr/libexec/qemu-kvm'},
>>> [],
>>> <Element 'features' at 0x2369390>,
>>> { '_children': [ <Element 'cpuselection' at 0x23693d0>,
>>> <Element 'deviceboot' at 0x2369410>,
>>> <Element 'disksnapshot' at 0x2369450>,
>>> <Element 'acpi' at 0x2369490>,
>>> <Element 'apic' at 0x23694d0>,
>>> <Element 'pae' at 0x2369510>,
>>> <Element 'nonpae' at 0x2369550>],
>>> 'attrib': { },
>>> 'tag': 'features',
>>> 'tail': '\n ',
>>> 'text': '\n '},
>>> [ <Element 'cpuselection' at 0x23693d0>,
>>> <Element 'deviceboot' at 0x2369410>,
>>> <Element 'disksnapshot' at 0x2369450>,
>>> <Element 'acpi' at 0x2369490>,
>>> <Element 'apic' at 0x23694d0>,
>>> <Element 'pae' at 0x2369510>,
>>> <Element 'nonpae' at 0x2369550>],
>>> <Element 'cpuselection' at 0x23693d0>,
>>> { '_children': [], 'attrib': { }, 'tag': 'cpuselection',
>>> 'tail': '\n '},
>>> [],
>>> <Element 'deviceboot' at 0x2369410>,
>>> { '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail':
>>> '\n '},
>>> [],
>>> <Element 'disksnapshot' at 0x2369450>,
>>> { '_children': [],
>>> 'attrib': { 'default': 'on', 'toggle': 'no'},
>>> 'tag': 'disksnapshot',
>>> 'tail': '\n '},
>>> [],
>>> <Element 'acpi' at 0x2369490>,
>>> { '_children': [],
>>> 'attrib': { 'default': 'on', 'toggle': 'yes'},
>>> 'tag': 'acpi',
>>> 'tail': '\n '},
>>> [],
>>> <Element 'apic' at 0x23694d0>,
>>> { '_children': [],
>>> 'attrib': { 'default': 'on', 'toggle': 'no'},
>>> 'tag': 'apic',
>>> 'tail': '\n '},
>>> [],
>>> <Element 'pae' at 0x2369510>,
>>> { '_children': [], 'attrib': { }, 'tag': 'pae', 'tail': '\n '},
>>> [],
>>> <Element 'nonpae' at 0x2369550>,
>>> { '_children': [], 'attrib': { }, 'tag': 'nonpae', 'tail': '\n '},
>>> [],
>>> <Element 'guest' at 0x2369590>,
>>> { '_children': [ <Element 'os_type' at 0x23695d0>,
>>> <Element 'arch' at 0x2369610>,
>>> <Element 'features' at 0x2369bd0>],
>>> 'attrib': { },
>>> 'tag': 'guest',
>>> 'tail': '\n\n',
>>> 'text': '\n '},
>>> [ <Element 'os_type' at 0x23695d0>,
>>> <Element 'arch' at 0x2369610>,
>>> <Element 'features' at 0x2369bd0>],
>>> <Element 'os_type' at 0x23695d0>,
>>> { '_children': [],
>>> 'attrib': { },
>>> 'tag': 'os_type',
>>> 'tail': '\n ',
>>> 'text': 'hvm'},
>>> [],
>>> <Element 'arch' at 0x2369610>,
>>> { '_children': [ <Element 'wordsize' at 0x2369650>,
>>> <Element 'emulator' at 0x2369690>,
>>> <Element 'machine' at 0x2369710>,
>>> <Element 'machine' at 0x2369750>,
>>> <Element 'machine' at 0x2369790>,
>>> <Element 'machine' at 0x23697d0>,
>>> <Element 'machine' at 0x2369810>,
>>> <Element 'machine' at 0x2369850>,
>>> <Element 'machine' at 0x2369890>,
>>> <Element 'machine' at 0x23698d0>,
>>> <Element 'machine' at 0x2369910>,
>>> <Element 'machine' at 0x2369950>,
>>> <Element 'machine' at 0x2369990>,
>>> <Element 'machine' at 0x23699d0>,
>>> <Element 'machine' at 0x2369a10>,
>>> <Element 'machine' at 0x2369a50>,
>>> <Element 'machine' at 0x2369a90>,
>>> <Element 'domain' at 0x2369ad0>,
>>> <Element 'domain' at 0x2369b10>],
>>> 'attrib': { 'name': 'x86_64'},
>>> 'tag': 'arch',
>>> 'tail': '\n ',
>>> 'text': '\n '},
>>> [ <Element 'wordsize' at 0x2369650>,
>>> <Element 'emulator' at 0x2369690>,
>>> <Element 'machine' at 0x2369710>,
>>> <Element 'machine' at 0x2369750>,
>>> <Element 'machine' at 0x2369790>,
>>> <Element 'machine' at 0x23697d0>,
>>> <Element 'machine' at 0x2369810>,
>>> <Element 'machine' at 0x2369850>,
>>> <Element 'machine' at 0x2369890>,
>>> <Element 'machine' at 0x23698d0>,
>>> <Element 'machine' at 0x2369910>,
>>> <Element 'machine' at 0x2369950>,
>>> <Element 'machine' at 0x2369990>,
>>> <Element 'machine' at 0x23699d0>,
>>> <Element 'machine' at 0x2369a10>,
>>> <Element 'machine' at 0x2369a50>,
>>> <Element 'machine' at 0x2369a90>,
>>> <Element 'domain' at 0x2369ad0>,
>>> <Element 'domain' at 0x2369b10>],
>>> <Element 'wordsize' at 0x2369650>,
>>> { '_children': [],
>>> 'attrib': { },
>>> 'tag': 'wordsize',
>>> 'tail': '\n ',
>>> 'text': '64'},
>>> [],
>>> <Element 'emulator' at 0x2369690>,
>>> { '_children': [],
>>> 'attrib': { },
>>> 'tag': 'emulator',
>>> 'tail': '\n ',
>>> 'text': '/usr/libexec/qemu-kvm'},
>>> [],
>>> <Element 'machine' at 0x2369710>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'pc-i440fx-rhel7.2.0'},
>>> [],
>>> <Element 'machine' at 0x2369750>,
>>> { '_children': [],
>>> 'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'pc'},
>>> [],
>>> <Element 'machine' at 0x2369790>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'pc-i440fx-rhel7.0.0'},
>>> [],
>>> <Element 'machine' at 0x23697d0>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'pc-q35-rhel7.1.0'},
>>> [],
>>> <Element 'machine' at 0x2369810>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'rhel6.3.0'},
>>> [],
>>> <Element 'machine' at 0x2369850>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'pc-q35-rhel7.2.0'},
>>> [],
>>> <Element 'machine' at 0x2369890>,
>>> { '_children': [],
>>> 'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'q35'},
>>> [],
>>> <Element 'machine' at 0x23698d0>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'rhel6.4.0'},
>>> [],
>>> <Element 'machine' at 0x2369910>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'rhel6.0.0'},
>>> [],
>>> <Element 'machine' at 0x2369950>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'pc-i440fx-rhel7.1.0'},
>>> [],
>>> <Element 'machine' at 0x2369990>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'rhel6.5.0'},
>>> [],
>>> <Element 'machine' at 0x23699d0>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'rhel6.6.0'},
>>> [],
>>> <Element 'machine' at 0x2369a10>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'rhel6.1.0'},
>>> [],
>>> <Element 'machine' at 0x2369a50>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'pc-q35-rhel7.0.0'},
>>> [],
>>> <Element 'machine' at 0x2369a90>,
>>> { '_children': [],
>>> 'attrib': { 'maxCpus': '240'},
>>> 'tag': 'machine',
>>> 'tail': '\n ',
>>> 'text': 'rhel6.2.0'},
>>> [],
>>> <Element 'domain' at 0x2369ad0>,
>>> { '_children': [],
>>> 'attrib': { 'type': 'qemu'},
>>> 'tag': 'domain',
>>> 'tail': '\n '},
>>> [],
>>> <Element 'domain' at 0x2369b10>,
>>> { '_children': [<Element 'emulator' at 0x2369b50>],
>>> 'attrib': { 'type': 'kvm'},
>>> 'tag': 'domain',
>>> 'tail': '\n ',
>>> 'text': '\n '},
>>> [<Element 'emulator' at 0x2369b50>],
>>> <Element 'emulator' at 0x2369b50>,
>>> { '_children': [],
>>> 'attrib': { },
>>> 'tag': 'emulator',
>>> 'tail': '\n ',
>>> 'text': '/usr/libexec/qemu-kvm'},
>>> [],
>>> <Element 'features' at 0x2369bd0>,
>>> { '_children': [ <Element 'cpuselection' at 0x2369c10>,
>>> <Element 'deviceboot' at 0x2369c50>,
>>> <Element 'disksnapshot' at 0x2369c90>,
>>> <Element 'acpi' at 0x2369cd0>,
>>> <Element 'apic' at 0x2369d10>],
>>> 'attrib': { },
>>> 'tag': 'features',
>>> 'tail': '\n ',
>>> 'text': '\n '},
>>> [ <Element 'cpuselection' at 0x2369c10>,
>>> <Element 'deviceboot' at 0x2369c50>,
>>> <Element 'disksnapshot' at 0x2369c90>,
>>> <Element 'acpi' at 0x2369cd0>,
>>> <Element 'apic' at 0x2369d10>],
>>> <Element 'cpuselection' at 0x2369c10>,
>>> { '_children': [], 'attrib': { }, 'tag': 'cpuselection',
>>> 'tail': '\n '},
>>> [],
>>> <Element 'deviceboot' at 0x2369c50>,
>>> { '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail':
>>> '\n '},
>>> [],
>>> <Element 'disksnapshot' at 0x2369c90>,
>>> { '_children': [],
>>> 'attrib': { 'default': 'on', 'toggle': 'no'},
>>> 'tag': 'disksnapshot',
>>> 'tail': '\n '},
>>> [],
>>> <Element 'acpi' at 0x2369cd0>,
>>> { '_children': [],
>>> 'attrib': { 'default': 'on', 'toggle': 'yes'},
>>> 'tag': 'acpi',
>>> 'tail': '\n '},
>>> [],
>>> <Element 'apic' at 0x2369d10>,
>>> { '_children': [],
>>> 'attrib': { 'default': 'on', 'toggle': 'no'},
>>> 'tag': 'apic',
>>> 'tail': '\n '},
>>> [],
>>> [],
>>> (<type '_ctypes.Array'>,),
>>> { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>,
>>> '__doc__': None,
>>> '__module__': 'vdsm.netlink',
>>> '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>,
>>> '_length_': 60,
>>> '_type_': <class 'ctypes.c_char'>,
>>> 'raw': <attribute 'raw' of 'c_char_Array_60' objects>,
>>> 'value': <attribute 'value' of 'c_char_Array_60' objects>},
>>> <class 'vdsm.netlink.c_char_Array_60'>,
>>> <attribute '__dict__' of 'c_char_Array_60' objects>,
>>> <attribute '__weakref__' of 'c_char_Array_60' objects>,
>>> ( <class 'vdsm.netlink.c_char_Array_60'>,
>>> <type '_ctypes.Array'>,
>>> <type '_ctypes._CData'>,
>>> <type 'object'>),
>>> <attribute 'raw' of 'c_char_Array_60' objects>,
>>> <attribute 'value' of 'c_char_Array_60' objects>,
>>> <xml.etree.ElementTree.XMLParser object at 0x2357f50>,
>>> { '_doctype': None,
>>> '_error': <class 'xml.parsers.expat.ExpatError'>,
>>> '_names': { u'acpi': 'acpi',
>>> u'apic': 'apic',
>>> u'arch': 'arch',
>>> u'baselabel': 'baselabel',
>>> u'canonical': 'canonical',
>>> u'capabilities': 'capabilities',
>>> u'cell': 'cell',
>>> u'cells': 'cells',
>>> u'core_id': 'core_id',
>>> u'cores': 'cores',
>>> u'cpu': 'cpu',
>>> u'cpus': 'cpus',
>>> u'cpuselection': 'cpuselection',
>>> u'default': 'default',
>>> u'deviceboot': 'deviceboot',
>>> u'disksnapshot': 'disksnapshot',
>>> u'distances': 'distances',
>>> u'doi': 'doi',
>>> u'domain': 'domain',
>>> u'emulator': 'emulator',
>>> u'feature': 'feature',
>>> u'features': 'features',
>>> u'guest': 'guest',
>>> u'host': 'host',
>>> u'id': 'id',
>>> u'live': 'live',
>>> u'machine': 'machine',
>>> u'maxCpus': 'maxCpus',
>>> u'memory': 'memory',
>>> u'migration_features': 'migration_features',
>>> u'model': 'model',
>>> u'name': 'name',
>>> u'nonpae': 'nonpae',
>>> u'num': 'num',
>>> u'os_type': 'os_type',
>>> u'pae': 'pae',
>>> u'pages': 'pages',
>>> u'power_management': 'power_management',
>>> u'secmodel': 'secmodel',
>>> u'sibling': 'sibling',
>>> u'siblings': 'siblings',
>>> u'size': 'size',
>>> u'socket_id': 'socket_id',
>>> u'sockets': 'sockets',
>>> u'suspend_disk': 'suspend_disk',
>>> u'suspend_hybrid': 'suspend_hybrid',
>>> u'suspend_mem': 'suspend_mem',
>>> u'threads': 'threads',
>>> u'toggle': 'toggle',
>>> u'topology': 'topology',
>>> u'type': 'type',
>>> u'unit': 'unit',
>>> u'uri_transport': 'uri_transport',
>>> u'uri_transports': 'uri_transports',
>>> u'uuid': 'uuid',
>>> u'value': 'value',
>>> u'vendor': 'vendor',
>>> u'wordsize': 'wordsize'},
>>> '_target': <xml.etree.ElementTree.TreeBuilder object at 0x2357f10>,
>>> 'entity': { },
>>> 'parser': <pyexpat.xmlparser object at 0x22e5360>,
>>> 'version': 'Expat 2.1.0'},
>>> <pyexpat.xmlparser object at 0x22e5360>,
>>> <xml.etree.ElementTree.TreeBuilder object at 0x2357f10>,
>>> <bound method XMLParser._start_list of
>>> <xml.etree.ElementTree.XMLParser object at 0x2357f50>>,
>>> <bound method XMLParser._end of <xml.etree.ElementTree.XMLParser
>>> object at 0x2357f50>>,
>>> <bound method XMLParser._pi of <xml.etree.ElementTree.XMLParser
>>> object at 0x2357f50>>,
>>> <bound method XMLParser._data of <xml.etree.ElementTree.XMLParser
>>> object at 0x2357f50>>,
>>> <bound method XMLParser._comment of
>>> <xml.etree.ElementTree.XMLParser object at 0x2357f50>>,
>>> <bound method XMLParser._default of
>>> <xml.etree.ElementTree.XMLParser object at 0x2357f50>>,
>>> { '_data': [],
>>> '_elem': [],
>>> '_factory': <class 'xml.etree.ElementTree.Element'>,
>>> '_last': <Element 'capabilities' at 0x2357f90>,
>>> '_tail': 1},
>>> [],
>>> <Element 'cell' at 0x23646d0>,
>>> <Element 'cells' at 0x2364690>,
>>> <Element 'topology' at 0x2364650>,
>>> <Element 'host' at 0x2357fd0>,
>>> <Element 'capabilities' at 0x2357f90>,
>>> { '_children': [ <Element 'memory' at 0x2364710>,
>>> <Element 'pages' at 0x2364750>,
>>> <Element 'pages' at 0x2364790>,
>>> <Element 'distances' at 0x23647d0>,
>>> <Element 'cpus' at 0x2364850>],
>>> 'attrib': { 'id': '0'},
>>> 'tag': 'cell',
>>> 'tail': '\n ',
>>> 'text': '\n '},
>>> { '_children': [<Element 'cell' at 0x23646d0>],
>>> 'attrib': { 'num': '1'},
>>> 'tag': 'cells',
>>> 'tail': '\n ',
>>> 'text': '\n '},
>>> { '_children': [<Element 'cells' at 0x2364690>],
>>> 'attrib': { },
>>> 'tag': 'topology',
>>> 'tail': '\n ',
>>> 'text': '\n '},
>>> { '_children': [ <Element 'uuid' at 0x2364050>,
>>> <Element 'cpu' at 0x2364090>,
>>> <Element 'power_management' at 0x2364390>,
>>> <Element 'migration_features' at 0x23644d0>,
>>> <Element 'topology' at 0x2364650>,
>>> <Element 'secmodel' at 0x2364a90>,
>>> <Element 'secmodel' at 0x2364bd0>],
>>> 'attrib': { },
>>> 'tag': 'host',
>>> 'tail': '\n\n ',
>>> 'text': '\n '},
>>> { '_children': [ <Element 'host' at 0x2357fd0>,
>>> <Element 'guest' at 0x2364d10>,
>>> <Element 'guest' at 0x2369590>],
>>> 'attrib': { },
>>> 'tag': 'capabilities',
>>> 'text': '\n\n '},
>>> [ <Element 'memory' at 0x2364710>,
>>> <Element 'pages' at 0x2364750>,
>>> <Element 'pages' at 0x2364790>,
>>> <Element 'distances' at 0x23647d0>,
>>> <Element 'cpus' at 0x2364850>],
>>> [<Element 'cell' at 0x23646d0>],
>>> [<Element 'cells' at 0x2364690>],
>>> [ <Element 'uuid' at 0x2364050>,
>>> <Element 'cpu' at 0x2364090>,
>>> <Element 'power_management' at 0x2364390>,
>>> <Element 'migration_features' at 0x23644d0>,
>>> <Element 'topology' at 0x2364650>,
>>> <Element 'secmodel' at 0x2364a90>,
>>> <Element 'secmodel' at 0x2364bd0>],
>>> [ <Element 'host' at 0x2357fd0>,
>>> <Element 'guest' at 0x2364d10>,
>>> <Element 'guest' at 0x2369590>],
>>> <Element 'migration_features' at 0x23644d0>,
>>> <Element 'power_management' at 0x2364390>,
>>> <Element 'cpu' at 0x2364090>,
>>> <Element 'uuid' at 0x2364050>,
>>> { '_children': [ <Element 'live' at 0x2364510>,
>>> <Element 'uri_transports' at 0x2364590>],
>>> 'attrib': { },
>>> 'tag': 'migration_features',
>>> 'tail': '\n ',
>>> 'text': '\n '},
>>> { '_children': [ <Element 'suspend_mem' at 0x23643d0>,
>>> <Element 'suspend_disk' at 0x2364410>,
>>> <Element 'suspend_hybrid' at 0x2364490>],
>>> 'attrib': { },
>>> 'tag': 'power_management',
>>> 'tail': '\n ',
>>> 'text': '\n '},
>>> { '_children': [ <Element 'arch' at 0x23640d0>,
>>> <Element 'model' at 0x2364110>,
>>> <Element 'vendor' at 0x2364150>,
>>> <Element 'topology' at 0x2364190>,
>>> <Element 'feature' at 0x23641d0>,
>>> <Element 'feature' at 0x2364210>,
>>> <Element 'feature' at 0x2364250>,
>>> <Element 'feature' at 0x2364290>,
>>> <Element 'feature' at 0x23642d0>,
>>> <Element 'pages' at 0x2364310>,
>>> <Element 'pages' at 0x2364350>],
>>> 'attrib': { },
>>> 'tag': 'cpu',
>>> 'tail': '\n ',
>>> 'text': '\n '},
>>> { '_children': [],
>>> 'attrib': { },
>>> 'tag': 'uuid',
>>> 'tail': '\n ',
>>> 'text': 'a920d963-c18f-4fda-aa5a-deee9bbcde9d'},
>>> [<Element 'live' at 0x2364510>, <Element 'uri_transports' at 0x2364590>],
>>> [ <Element 'suspend_mem' at 0x23643d0>,
>>> <Element 'suspend_disk' at 0x2364410>,
>>> <Element 'suspend_hybrid' at 0x2364490>],
>>> [ <Element 'arch' at 0x23640d0>,
>>> <Element 'model' at 0x2364110>,
>>> <Element 'vendor' at 0x2364150>,
>>> <Element 'topology' at 0x2364190>,
>>> <Element 'feature' at 0x23641d0>,
>>> <Element 'feature' at 0x2364210>,
>>> <Element 'feature' at 0x2364250>,
>>> <Element 'feature' at 0x2364290>,
>>> <Element 'feature' at 0x23642d0>,
>>> <Element 'pages' at 0x2364310>,
>>> <Element 'pages' at 0x2364350>],
>>> [],
>>> <Element 'uri_transports' at 0x2364590>,
>>> <Element 'live' at 0x2364510>,
>>> <Element 'suspend_hybrid' at 0x2364490>,
>>> <Element 'suspend_disk' at 0x2364410>,
>>> <Element 'suspend_mem' at 0x23643d0>,
>>> <Element 'pages' at 0x2364350>,
>>> <Element 'pages' at 0x2364310>,
>>> <Element 'feature' at 0x23642d0>,
>>> <Element 'feature' at 0x2364290>,
>>> <Element 'feature' at 0x2364250>,
>>> <Element 'feature' at 0x2364210>,
>>> <Element 'feature' at 0x23641d0>,
>>> <Element 'topology' at 0x2364190>,
>>> <Element 'vendor' at 0x2364150>,
>>> <Element 'model' at 0x2364110>,
>>> <Element 'arch' at 0x23640d0>,
>>> { '_children': [ <Element 'uri_transport' at 0x23645d0>,
>>> <Element 'uri_transport' at 0x2364610>],
>>> 'attrib': { },
>>> 'tag': 'uri_transports',
>>> 'tail': '\n ',
>>> 'text': '\n '},
>>> { '_children': [], 'attrib': { }, 'tag': 'live', 'tail': '\n '},
>>> { '_children': [], 'attrib': { }, 'tag': 'suspend_hybrid',
>>> 'tail': '\n '},
>>> { '_children': [], 'attrib': { }, 'tag': 'suspend_disk',
>>> 'tail': '\n '},
>>> { '_children': [], 'attrib': { }, 'tag': 'suspend_mem',
>>> 'tail': '\n '},
>>> { '_children': [],
>>> 'attrib': { 'size': '2048', 'unit': 'KiB'},
>>> 'tag': 'pages',
>>> 'tail': '\n '},
>>> { '_children': [],
>>> 'attrib': { 'size': '4', 'unit': 'KiB'},
>>> 'tag': 'pages',
>>> 'tail': '\n '},
>>> { '_children': [],
>>> 'attrib': { 'name': 'ss'},
>>> 'tag': 'feature',
>>> 'tail': '\n '},
>>> { '_children': [],
>>> 'attrib': { 'name': 'vmx'},
>>> 'tag': 'feature',
>>> 'tail': '\n '},
>>> { '_children': [],
>>> 'attrib': { 'name': 'pcid'},
>>> 'tag': 'feature',
>>> 'tail': '\n '},
>>> { '_children': [],
>>> 'attrib': { 'name': 'osxsave'},
>>> 'tag': 'feature',
>>> 'tail': '\n '},
>>> { '_children': [],
>>> 'attrib': { 'name': 'hypervisor'},
>>> 'tag': 'feature',
>>> 'tail': '\n '},
>>> { '_children': [],
>>> 'attrib': { 'cores': '1', 'sockets': '8', 'threads': '1'},
>>> 'tag': 'topology',
>>> 'tail': '\n '},
>>> { '_children': [],
>>> 'attrib': { },
>>> 'tag': 'vendor',
>>> 'tail': '\n ',
>>> 'text': 'Intel'},
>>> { '_children': [],
>>> 'attrib': { },
>>> 'tag': 'model',
>>> 'tail': '\n ',
>>> 'text': 'IvyBridge'},
>>> { '_children': [],
>>> 'attrib': { },
>>> 'tag': 'arch',
>>> 'tail': '\n ',
>>> 'text': 'x86_64'},
>>> [ <Element 'uri_transport' at 0x23645d0>,
>>> <Element 'uri_transport' at 0x2364610>],
>>> [],
>>> [],
>>> [],
>>> [],
>>> [],
>>> [],
>>> [],
>>> [],
>>> [],
>>> [],
>>> [],
>>> [],
>>> [],
>>> [],
>>> [],
>>> <Element 'uri_transport' at 0x2364610>,
>>> <Element 'uri_transport' at 0x23645d0>,
>>> { '_children': [],
>>> 'attrib': { },
>>> 'tag': 'uri_transport',
>>> 'tail': '\n ',
>>> 'text': 'rdma'},
>>> { '_children': [],
>>> 'attrib': { },
>>> 'tag': 'uri_transport',
>>> 'tail': '\n ',
>>> 'text': 'tcp'},
>>> [],
>>> [],
>>> (<type '_ctypes.Array'>,),
>>> { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>,
>>> '__doc__': None,
>>> '__module__': 'vdsm.netlink',
>>> '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>,
>>> '_length_': 60,
>>> '_type_': <class 'ctypes.c_char'>,
>>> 'raw': <attribute 'raw' of 'c_char_Array_60' objects>,
>>> 'value': <attribute 'value' of 'c_char_Array_60' objects>},
>>> <class 'vdsm.netlink.c_char_Array_60'>,
>>> <attribute '__dict__' of 'c_char_Array_60' objects>,
>>> <attribute '__weakref__' of 'c_char_Array_60' objects>,
>>> ( <class 'vdsm.netlink.c_char_Array_60'>,
>>> <type '_ctypes.Array'>,
>>> <type '_ctypes._CData'>,
>>> <type 'object'>),
>>> <attribute 'raw' of 'c_char_Array_60' objects>,
>>> <attribute 'value' of 'c_char_Array_60' objects>]
More information about the Devel
mailing list