[ovirt-devel] [VDSM] health: Introduce Vdsm health monitoring
Nir Soffer
nsoffer at redhat.com
Fri Jan 15 08:52:13 UTC 2016
I tested with https://gerrit.ovirt.org/#/c/51630
libvirt.virDomain object does not leak any more, but pthreading object
are still leaking,
and the number of uncollectible objects is still very high (12,915).
Each time we start and stop a vm, we leak about 6000 objects which are part
of a reference cycle, and each time we create and leak 27 pthreading
locks/conditions
which are probably the reason for this leak (because they implement
__del__ or reference object
implementing __del__.
When vdsm is idle, we leak 18 objects per minute, we don't know yet why.
# tail -f /var/log/vdsm/vdsm.log | grep health
Vdsm was connected to storage pool here, running as the spm...
Thread-15::DEBUG::2016-01-15
10:00:00,159::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
10:00:00,178::health::86::health::(_check) Collected 468 objects
Thread-15::WARNING::2016-01-15
10:00:00,182::health::100::health::(_check) Found 5953 uncollectible
objects, 0 suspects implementing __del__: []
Thread-15::DEBUG::2016-01-15
10:01:00,182::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
10:01:00,207::health::86::health::(_check) Collected 202 objects
Thread-15::WARNING::2016-01-15
10:01:00,211::health::100::health::(_check) Found 6169 uncollectible
objects, 23 suspects implementing __del__: ['<pthreading.Lock object
at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>',
'<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object
at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>',
'<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object
at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>',
'<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at
0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>',
'<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at
0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>',
'<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at
0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>',
'<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at
0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>',
'<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at
0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>']
vm was started here...
Thread-15::DEBUG::2016-01-15
10:02:00,212::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
10:02:00,243::health::86::health::(_check) Collected 1689 objects
Thread-15::WARNING::2016-01-15
10:02:00,252::health::100::health::(_check) Found 11107 uncollectible
objects, 23 suspects implementing __del__: ['<pthreading.Lock object
at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>',
'<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object
at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>',
'<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object
at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>',
'<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at
0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>',
'<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at
0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>',
'<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at
0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>',
'<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at
0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>',
'<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at
0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>']
Thread-15::DEBUG::2016-01-15
10:03:00,254::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
10:03:00,285::health::86::health::(_check) Collected 57 objects
Thread-15::WARNING::2016-01-15
10:03:00,294::health::100::health::(_check) Found 11162 uncollectible
objects, 23 suspects implementing __del__: ['<pthreading.Lock object
at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>',
'<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object
at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>',
'<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object
at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>',
'<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at
0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>',
'<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at
0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>',
'<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at
0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>',
'<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at
0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>',
'<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at
0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>']
vm was stopped here...
Thread-15::DEBUG::2016-01-15
10:04:00,294::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
10:04:00,324::health::86::health::(_check) Collected 1755 objects
Thread-15::WARNING::2016-01-15
10:04:00,335::health::100::health::(_check) Found 12915 uncollectible
objects, 50 suspects implementing __del__: ['<pthreading.Lock object
at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>',
'<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object
at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>',
'<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object
at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>',
'<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at
0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>',
'<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at
0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>',
'<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at
0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>',
'<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at
0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>',
'<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at
0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>',
'<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object
at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>',
'<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object
at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>',
'<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object
at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>',
'<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at
0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>',
'<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at
0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>',
'<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at
0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>',
'<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at
0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>',
'<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at
0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>',
'<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object
at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>']
Skipping some logs, no activity in engine...
Thread-15::DEBUG::2016-01-15
10:23:01,082::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
10:23:01,113::health::86::health::(_check) Collected 18 objects
Thread-15::WARNING::2016-01-15
10:23:01,124::health::100::health::(_check) Found 13267 uncollectible
objects, 50 suspects implementing __del__: ['<pthreading.Lock object
at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>',
'<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object
at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>',
'<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object
at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>',
'<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at
0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>',
'<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at
0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>',
'<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at
0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>',
'<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at
0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>',
'<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at
0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>',
'<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object
at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>',
'<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object
at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>',
'<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object
at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>',
'<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at
0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>',
'<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at
0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>',
'<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at
0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>',
'<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at
0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>',
'<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at
0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>',
'<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object
at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>']
Thread-15::DEBUG::2016-01-15
10:24:01,125::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
10:24:01,156::health::86::health::(_check) Collected 18 objects
Thread-15::WARNING::2016-01-15
10:24:01,167::health::100::health::(_check) Found 13283 uncollectible
objects, 50 suspects implementing __del__: ['<pthreading.Lock object
at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>',
'<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object
at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>',
'<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object
at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>',
'<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at
0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>',
'<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at
0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>',
'<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at
0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>',
'<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at
0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>',
'<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at
0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>',
'<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object
at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>',
'<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object
at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>',
'<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object
at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>',
'<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at
0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>',
'<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at
0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>',
'<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at
0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>',
'<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at
0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>',
'<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at
0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>',
'<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object
at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>']
vm was started here...
Thread-15::DEBUG::2016-01-15
10:25:01,167::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
10:25:01,204::health::86::health::(_check) Collected 1537 objects
Thread-15::WARNING::2016-01-15
10:25:01,219::health::100::health::(_check) Found 18264 uncollectible
objects, 50 suspects implementing __del__: ['<pthreading.Lock object
at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>',
'<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object
at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>',
'<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object
at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>',
'<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at
0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>',
'<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at
0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>',
'<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at
0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>',
'<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at
0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>',
'<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at
0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>',
'<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object
at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>',
'<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object
at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>',
'<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object
at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>',
'<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at
0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>',
'<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at
0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>',
'<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at
0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>',
'<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at
0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>',
'<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at
0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>',
'<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object
at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>']
vm was ask to stop here...
Thread-15::DEBUG::2016-01-15
10:26:01,219::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
10:26:01,255::health::86::health::(_check) Collected 57 objects
Thread-15::WARNING::2016-01-15
10:26:01,270::health::100::health::(_check) Found 18319 uncollectible
objects, 50 suspects implementing __del__: ['<pthreading.Lock object
at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>',
'<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object
at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>',
'<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object
at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>',
'<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at
0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>',
'<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at
0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>',
'<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at
0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>',
'<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at
0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>',
'<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at
0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>',
'<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object
at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>',
'<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object
at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>',
'<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object
at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>',
'<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at
0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>',
'<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at
0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>',
'<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at
0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>',
'<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at
0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>',
'<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at
0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>',
'<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object
at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>']
vm was shutdown here...
Thread-15::DEBUG::2016-01-15
10:27:01,270::health::84::health::(_check) Checking health
Thread-15::DEBUG::2016-01-15
10:27:01,306::health::86::health::(_check) Collected 1752 objects
Thread-15::WARNING::2016-01-15
10:27:01,322::health::100::health::(_check) Found 20069 uncollectible
objects, 77 suspects implementing __del__: ['<pthreading.Lock object
at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>',
'<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object
at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>',
'<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object
at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>',
'<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at
0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>',
'<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at
0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>',
'<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at
0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>',
'<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at
0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>',
'<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at
0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>',
'<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object
at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>',
'<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object
at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>',
'<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object
at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>',
'<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at
0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>',
'<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at
0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>',
'<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at
0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>',
'<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at
0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>',
'<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at
0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>',
'<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object
at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>',
'<pthreading.Lock object at 0x7fa904078710>', '<pthreading.Lock object
at 0x7fa9b40af440>', '<pthreading.Lock object at 0x7fa90405b200>',
'<pthreading.Lock object at 0x7fa904078998>', '<pthreading.Lock object
at 0x7fa9a407cfc8>', '<pthreading.Lock object at 0x7fa904078638>',
'<pthreading.Lock object at 0x7fa904078560>', '<pthreading.Lock object
at 0x7fa904078200>', '<pthread.Cond object at 0x7fa9a4142d10>',
'<pthreading.Lock object at 0x7fa9040785f0>', '<pthread.Cond object at
0x7fa9b40d1a50>', '<pthreading.Lock object at 0x7fa90405b9e0>',
'<pthread.Cond object at 0x7fa9a4142610>', '<pthreading.Lock object at
0x7fa9040787a0>', '<pthread.Cond object at 0x7fa9b40d1f10>',
'<pthreading.Lock object at 0x7fa9040786c8>', '<pthread.Cond object at
0x7fa9a4142b10>', '<pthreading.Lock object at 0x7fa904078368>',
'<pthread.Cond object at 0x7fa9b40d1790>', '<pthreading.Lock object at
0x7fa904078290>', '<pthread.Cond object at 0x7fa9b40d14d0>',
'<pthreading.Lock object at 0x7fa904078248>', '<pthread.Cond object at
0x7fa9b40d1150>', '<pthread.Cond object at 0x7fa9a41360d0>',
'<pthreading.Lock object at 0x7fa9441a8d40>', '<pthreading.Lock object
at 0x7fa944049b00>', '<pthread.Cond object at 0x7fa9a4136410>']
On Thu, Jan 14, 2016 at 8:00 PM, Nir Soffer <nsoffer at redhat.com> wrote:
> Here are example logs with running and stopping one vm:
>
> # tail -f /var/log/vdsm/vdsm.log | grep health
> Thread-13::DEBUG::2016-01-14
> 19:42:04,625::health::84::health::(_check) Checking health
> Thread-13::DEBUG::2016-01-14
> 19:42:04,651::health::86::health::(_check) Collected 25 objects
> Thread-13::WARNING::2016-01-14
> 19:42:04,656::health::100::health::(_check) Found 6085 uncollectible
> objects, 0 suspects implementing __del__: []
>
> Vm was started here...
>
> Thread-13::DEBUG::2016-01-14
> 19:43:04,657::health::84::health::(_check) Checking health
> Thread-13::DEBUG::2016-01-14
> 19:43:04,688::health::86::health::(_check) Collected 1680 objects
> Thread-13::WARNING::2016-01-14
> 19:43:04,697::health::100::health::(_check) Found 11041 uncollectible
> objects, 0 suspects implementing __del__: []
> Thread-13::DEBUG::2016-01-14
> 19:44:04,698::health::84::health::(_check) Checking health
> Thread-13::DEBUG::2016-01-14
> 19:44:04,729::health::86::health::(_check) Collected 57 objects
> Thread-13::WARNING::2016-01-14
> 19:44:04,737::health::100::health::(_check) Found 11096 uncollectible
> objects, 0 suspects implementing __del__: []
> Thread-13::DEBUG::2016-01-14
> 19:45:04,738::health::84::health::(_check) Checking health
> Thread-13::DEBUG::2016-01-14
> 19:45:04,769::health::86::health::(_check) Collected 69 objects
> Thread-13::WARNING::2016-01-14
> 19:45:04,778::health::100::health::(_check) Found 11163 uncollectible
> objects, 0 suspects implementing __del__: []
>
> Vm was shutdown here...
>
> Thread-13::DEBUG::2016-01-14
> 19:46:04,779::health::84::health::(_check) Checking health
> Thread-13::DEBUG::2016-01-14
> 19:46:04,809::health::86::health::(_check) Collected 2272 objects
> Thread-13::WARNING::2016-01-14
> 19:46:04,819::health::100::health::(_check) Found 13464 uncollectible
> objects, 28 suspects implementing __del__: ['<pthreading.Lock object
> at 0x7f1d8016f908>', '<pthreading.Lock object at 0x7f1d801804d0>',
> '<pthreading.Lock object at 0x7f1cf8010f38>', '<pthreading.Lock object
> at 0x7f1d8016f7a0>', '<pthreading.Lock object at 0x7f1d20003680>',
> '<pthreading.Lock object at 0x7f1da8112998>', '<pthreading.Lock object
> at 0x7f1d8016fa28>', '<pthreading.Lock object at 0x7f1da8112830>',
> '<pthread.Cond object at 0x7f1da811cf50>', '<pthreading.Lock object at
> 0x7f1d8016f200>', '<pthread.Cond object at 0x7f1da811ca50>',
> '<pthreading.Lock object at 0x7f1cf8010fc8>', '<pthread.Cond object at
> 0x7f1da8115c50>', '<pthreading.Lock object at 0x7f1da8112878>',
> '<pthread.Cond object at 0x7f1da811c550>', '<pthreading.Lock object at
> 0x7f1d8016f9e0>', '<pthread.Cond object at 0x7f1da811ca90>',
> '<pthreading.Lock object at 0x7f1d8016f680>', '<pthread.Cond object at
> 0x7f1da811cc50>', '<libvirt.virDomain object at 0x1ba8210>',
> '<pthreading.Lock object at 0x7f1d8016f758>', '<pthread.Cond object at
> 0x7f1da811cb50>', '<pthreading.Lock object at 0x7f1d8016fb90>',
> '<pthread.Cond object at 0x7f1da811cc10>', '<pthread.Cond object at
> 0x233ed50>', '<pthreading.Lock object at 0x7f1ce0227c20>',
> '<pthreading.Lock object at 0x7f1ce0227ab8>', '<pthread.Cond object at
> 0x233ed10>']
>
> Need to test with should be fixed by https://gerrit.ovirt.org/51630,
> which should
> eliminate '<libvirt.virDomain object at 0x1ba8210>', and maybe also
> the pthreading
> objects.
>
> Nir
>
> On Thu, Jan 14, 2016 at 7:38 PM, Nir Soffer <nsoffer at redhat.com> wrote:
>> Hi all,
>>
>> Continuing the leak investigation Francesco and Milan are working on,
>> I posted this patch, adding leak health monitoring to vdsm [1]. This patch
>> currently monitor collectible objects, and we may add other interesting
>> stuff later if needed.
>>
>> To enable the monitor, you must set:
>>
>> [devel]
>> health_monitor_enable = true
>>
>> In vdsm.conf, and restart vdsm.
>>
>> Here is an example logs - this was logged 60 seconds after starting vdsm,
>> when running as spm:
>>
>> Thread-13::DEBUG::2016-01-14
>> 18:53:43,239::health::84::health::(_check) Checking health
>> Thread-13::DEBUG::2016-01-14
>> 18:53:43,272::health::86::health::(_check) Collected 460 objects
>> Thread-13::WARNING::2016-01-14
>> 18:53:43,277::health::100::health::(_check) Found 5873 uncollectible
>> objects, 10 suspects implementing __del__: ['<pthreading.Lock object
>> at 0x7f012c115320>', '<pthreading.Lock object at 0x7f012c115440>',
>> '<pthreading.Lock object at 0x7f012c1153b0>', '<pthreading.Lock object
>> at 0x7f012c1154d0>', '<pthread.Cond object at 0x7f012c111350>',
>> '<pthread.Cond object at 0x7f012c1112d0>', '<pthread.Cond object at
>> 0x7f012c111250>', '<pthread.Cond object at 0x7f012c111850>',
>> '<pthread.Cond object at 0x7f012c1117d0>', '<pthread.Cond object at
>> 0x7f012c111750>']
>>
>> We found 460 uncollectible objects, 10 of them are objects implementing __del__.
>> Having such object in a reference cycle make the entire cycle uncollectible.
>>
>> The pthreading objects in this log grow as vdsm continue to run, even without
>> any interaction with engine. The pthreading objects leak is caused by a cycle
>> in storage.misc.RWLock, fixed in patch [2].
>>
>> After solving the pthreading leak, we still have many uncollectible
>> objects in the logs,
>> listing them, it seems that the suspect is hiding inside a collection
>> (dict, list, tuple)
>> and does not appear in the gc.garbage list. I will address this in later patch,
>> searching in the entire object graph instead of the top level of gc.garbage.
>>
>> See end of this mail for list of these objects. I printed them like this:
>>
>> Enable manhole debugging shell:
>>
>> [devel]
>> manhole_enable = true
>>
>> Connect to vdsm using manhole:
>>
>> nc -U /run/vdsm/vdsmd.manhole
>>
>> Print garbage:
>>
>>>>> import pprint
>>>>> import gc
>>>>> with open('/tmp/garbage.txt', 'w') as f:
>> pprint.pprint(gc.garbage, stream=f, indent=4)
>>
>> These objects looks related to libvirt - Francesco, can you take a look?
>>
>> [1] https://gerrit.ovirt.org/51708
>> [2] https://gerrit.ovirt.org/51868
>>
>> Nir
>>
>> --- garbage after patch [2] ---
>>
>> [ (<type '_ctypes.Array'>,),
>> { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>,
>> '__doc__': None,
>> '__module__': 'vdsm.netlink',
>> '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>,
>> '_length_': 60,
>> '_type_': <class 'ctypes.c_char'>,
>> 'raw': <attribute 'raw' of 'c_char_Array_60' objects>,
>> 'value': <attribute 'value' of 'c_char_Array_60' objects>},
>> <class 'vdsm.netlink.c_char_Array_60'>,
>> <attribute '__dict__' of 'c_char_Array_60' objects>,
>> <attribute '__weakref__' of 'c_char_Array_60' objects>,
>> ( <class 'vdsm.netlink.c_char_Array_60'>,
>> <type '_ctypes.Array'>,
>> <type '_ctypes._CData'>,
>> <type 'object'>),
>> <attribute 'raw' of 'c_char_Array_60' objects>,
>> <attribute 'value' of 'c_char_Array_60' objects>,
>> <cell at 0x23662f0: bool object at 0x7f1e086c8c40>,
>> <cell at 0x2366328: NoneType object at 0x7f1e086dcf00>,
>> <cell at 0x2366360: function object at 0x233f488>,
>> ( <cell at 0x23662f0: bool object at 0x7f1e086c8c40>,
>> <cell at 0x2366328: NoneType object at 0x7f1e086dcf00>,
>> <cell at 0x2366360: function object at 0x233f488>),
>> <function wrapMethod at 0x233f488>,
>> [ [ <Recursion on list with id=34033464>,
>> <Recursion on list with id=34033464>,
>> 'class'],
>> [ <Recursion on list with id=34033464>,
>> <Recursion on list with id=34033464>,
>> 'class'],
>> None],
>> [ [ <Recursion on list with id=33979064>,
>> <Recursion on list with id=33979064>,
>> None],
>> [ <Recursion on list with id=33979064>,
>> <Recursion on list with id=33979064>,
>> None],
>> 'class'],
>> [ [ <Recursion on list with id=35535256>,
>> <Recursion on list with id=35535256>,
>> 'class'],
>> [ <Recursion on list with id=35535256>,
>> <Recursion on list with id=35535256>,
>> 'class'],
>> None],
>> [ [ <Recursion on list with id=35535832>,
>> <Recursion on list with id=35535832>,
>> None],
>> [ <Recursion on list with id=35535832>,
>> <Recursion on list with id=35535832>,
>> None],
>> 'class'],
>> [ [ <Recursion on list with id=35556312>,
>> <Recursion on list with id=35556312>,
>> 'class'],
>> [ <Recursion on list with id=35556312>,
>> <Recursion on list with id=35556312>,
>> 'class'],
>> None],
>> [ [ <Recursion on list with id=35556600>,
>> <Recursion on list with id=35556600>,
>> None],
>> [ <Recursion on list with id=35556600>,
>> <Recursion on list with id=35556600>,
>> None],
>> 'class'],
>> [ [ <Recursion on list with id=35556960>,
>> <Recursion on list with id=35556960>,
>> 'class'],
>> [ <Recursion on list with id=35556960>,
>> <Recursion on list with id=35556960>,
>> 'class'],
>> None],
>> [ [ <Recursion on list with id=35561912>,
>> <Recursion on list with id=35561912>,
>> None],
>> [ <Recursion on list with id=35561912>,
>> <Recursion on list with id=35561912>,
>> None],
>> 'class'],
>> [ [ <Recursion on list with id=35583904>,
>> <Recursion on list with id=35583904>,
>> 'class'],
>> [ <Recursion on list with id=35583904>,
>> <Recursion on list with id=35583904>,
>> 'class'],
>> None],
>> [ [ <Recursion on list with id=35584912>,
>> <Recursion on list with id=35584912>,
>> None],
>> [ <Recursion on list with id=35584912>,
>> <Recursion on list with id=35584912>,
>> None],
>> 'class'],
>> [ [ <Recursion on list with id=36340368>,
>> <Recursion on list with id=36340368>,
>> 'class'],
>> [ <Recursion on list with id=36340368>,
>> <Recursion on list with id=36340368>,
>> 'class'],
>> None],
>> [ [ <Recursion on list with id=36340872>,
>> <Recursion on list with id=36340872>,
>> None],
>> [ <Recursion on list with id=36340872>,
>> <Recursion on list with id=36340872>,
>> None],
>> 'class'],
>> [ [ <Recursion on list with id=36364872>,
>> <Recursion on list with id=36364872>,
>> 'class'],
>> [ <Recursion on list with id=36364872>,
>> <Recursion on list with id=36364872>,
>> 'class'],
>> None],
>> [ [ <Recursion on list with id=36365520>,
>> <Recursion on list with id=36365520>,
>> None],
>> [ <Recursion on list with id=36365520>,
>> <Recursion on list with id=36365520>,
>> None],
>> 'class'],
>> [ [ <Recursion on list with id=36457216>,
>> <Recursion on list with id=36457216>,
>> 'class'],
>> [ <Recursion on list with id=36457216>,
>> <Recursion on list with id=36457216>,
>> 'class'],
>> None],
>> [ [ <Recursion on list with id=36458008>,
>> <Recursion on list with id=36458008>,
>> None],
>> [ <Recursion on list with id=36458008>,
>> <Recursion on list with id=36458008>,
>> None],
>> 'class'],
>> [ [ <Recursion on list with id=36460520>,
>> <Recursion on list with id=36460520>,
>> 'class'],
>> [ <Recursion on list with id=36460520>,
>> <Recursion on list with id=36460520>,
>> 'class'],
>> None],
>> [ [ <Recursion on list with id=36461168>,
>> <Recursion on list with id=36461168>,
>> None],
>> [ <Recursion on list with id=36461168>,
>> <Recursion on list with id=36461168>,
>> None],
>> 'class'],
>> [ [ <Recursion on list with id=36562344>,
>> <Recursion on list with id=36562344>,
>> 'class'],
>> [ <Recursion on list with id=36562344>,
>> <Recursion on list with id=36562344>,
>> 'class'],
>> None],
>> [ [ <Recursion on list with id=36563280>,
>> <Recursion on list with id=36563280>,
>> None],
>> [ <Recursion on list with id=36563280>,
>> <Recursion on list with id=36563280>,
>> None],
>> 'class'],
>> <Element 'memory' at 0x2364710>,
>> { '_children': [],
>> 'attrib': { 'unit': 'KiB'},
>> 'tag': 'memory',
>> 'tail': '\n ',
>> 'text': '4095476'},
>> [],
>> <Element 'pages' at 0x2364750>,
>> { '_children': [],
>> 'attrib': { 'size': '4', 'unit': 'KiB'},
>> 'tag': 'pages',
>> 'tail': '\n ',
>> 'text': '1023869'},
>> [],
>> <Element 'pages' at 0x2364790>,
>> { '_children': [],
>> 'attrib': { 'size': '2048', 'unit': 'KiB'},
>> 'tag': 'pages',
>> 'tail': '\n ',
>> 'text': '0'},
>> [],
>> <Element 'distances' at 0x23647d0>,
>> { '_children': [<Element 'sibling' at 0x2364810>],
>> 'attrib': { },
>> 'tag': 'distances',
>> 'tail': '\n ',
>> 'text': '\n '},
>> [<Element 'sibling' at 0x2364810>],
>> <Element 'sibling' at 0x2364810>,
>> { '_children': [],
>> 'attrib': { 'id': '0', 'value': '10'},
>> 'tag': 'sibling',
>> 'tail': '\n '},
>> [],
>> <Element 'cpus' at 0x2364850>,
>> { '_children': [ <Element 'cpu' at 0x2364890>,
>> <Element 'cpu' at 0x23648d0>,
>> <Element 'cpu' at 0x2364910>,
>> <Element 'cpu' at 0x2364950>,
>> <Element 'cpu' at 0x2364990>,
>> <Element 'cpu' at 0x23649d0>,
>> <Element 'cpu' at 0x2364a10>,
>> <Element 'cpu' at 0x2364a50>],
>> 'attrib': { 'num': '8'},
>> 'tag': 'cpus',
>> 'tail': '\n ',
>> 'text': '\n '},
>> [ <Element 'cpu' at 0x2364890>,
>> <Element 'cpu' at 0x23648d0>,
>> <Element 'cpu' at 0x2364910>,
>> <Element 'cpu' at 0x2364950>,
>> <Element 'cpu' at 0x2364990>,
>> <Element 'cpu' at 0x23649d0>,
>> <Element 'cpu' at 0x2364a10>,
>> <Element 'cpu' at 0x2364a50>],
>> <Element 'cpu' at 0x2364890>,
>> { '_children': [],
>> 'attrib': { 'core_id': '0',
>> 'id': '0',
>> 'siblings': '0',
>> 'socket_id': '0'},
>> 'tag': 'cpu',
>> 'tail': '\n '},
>> [],
>> <Element 'cpu' at 0x23648d0>,
>> { '_children': [],
>> 'attrib': { 'core_id': '0',
>> 'id': '1',
>> 'siblings': '1',
>> 'socket_id': '1'},
>> 'tag': 'cpu',
>> 'tail': '\n '},
>> [],
>> <Element 'cpu' at 0x2364910>,
>> { '_children': [],
>> 'attrib': { 'core_id': '0',
>> 'id': '2',
>> 'siblings': '2',
>> 'socket_id': '2'},
>> 'tag': 'cpu',
>> 'tail': '\n '},
>> [],
>> <Element 'cpu' at 0x2364950>,
>> { '_children': [],
>> 'attrib': { 'core_id': '0',
>> 'id': '3',
>> 'siblings': '3',
>> 'socket_id': '3'},
>> 'tag': 'cpu',
>> 'tail': '\n '},
>> [],
>> <Element 'cpu' at 0x2364990>,
>> { '_children': [],
>> 'attrib': { 'core_id': '0',
>> 'id': '4',
>> 'siblings': '4',
>> 'socket_id': '4'},
>> 'tag': 'cpu',
>> 'tail': '\n '},
>> [],
>> <Element 'cpu' at 0x23649d0>,
>> { '_children': [],
>> 'attrib': { 'core_id': '0',
>> 'id': '5',
>> 'siblings': '5',
>> 'socket_id': '5'},
>> 'tag': 'cpu',
>> 'tail': '\n '},
>> [],
>> <Element 'cpu' at 0x2364a10>,
>> { '_children': [],
>> 'attrib': { 'core_id': '0',
>> 'id': '6',
>> 'siblings': '6',
>> 'socket_id': '6'},
>> 'tag': 'cpu',
>> 'tail': '\n '},
>> [],
>> <Element 'cpu' at 0x2364a50>,
>> { '_children': [],
>> 'attrib': { 'core_id': '0',
>> 'id': '7',
>> 'siblings': '7',
>> 'socket_id': '7'},
>> 'tag': 'cpu',
>> 'tail': '\n '},
>> [],
>> <Element 'secmodel' at 0x2364a90>,
>> { '_children': [ <Element 'model' at 0x2364ad0>,
>> <Element 'doi' at 0x2364b10>,
>> <Element 'baselabel' at 0x2364b50>,
>> <Element 'baselabel' at 0x2364b90>],
>> 'attrib': { },
>> 'tag': 'secmodel',
>> 'tail': '\n ',
>> 'text': '\n '},
>> [ <Element 'model' at 0x2364ad0>,
>> <Element 'doi' at 0x2364b10>,
>> <Element 'baselabel' at 0x2364b50>,
>> <Element 'baselabel' at 0x2364b90>],
>> <Element 'model' at 0x2364ad0>,
>> { '_children': [],
>> 'attrib': { },
>> 'tag': 'model',
>> 'tail': '\n ',
>> 'text': 'selinux'},
>> [],
>> <Element 'doi' at 0x2364b10>,
>> { '_children': [],
>> 'attrib': { },
>> 'tag': 'doi',
>> 'tail': '\n ',
>> 'text': '0'},
>> [],
>> <Element 'baselabel' at 0x2364b50>,
>> { '_children': [],
>> 'attrib': { 'type': 'kvm'},
>> 'tag': 'baselabel',
>> 'tail': '\n ',
>> 'text': 'system_u:system_r:svirt_t:s0'},
>> [],
>> <Element 'baselabel' at 0x2364b90>,
>> { '_children': [],
>> 'attrib': { 'type': 'qemu'},
>> 'tag': 'baselabel',
>> 'tail': '\n ',
>> 'text': 'system_u:system_r:svirt_tcg_t:s0'},
>> [],
>> <Element 'secmodel' at 0x2364bd0>,
>> { '_children': [ <Element 'model' at 0x2364c10>,
>> <Element 'doi' at 0x2364c50>,
>> <Element 'baselabel' at 0x2364c90>,
>> <Element 'baselabel' at 0x2364cd0>],
>> 'attrib': { },
>> 'tag': 'secmodel',
>> 'tail': '\n ',
>> 'text': '\n '},
>> [ <Element 'model' at 0x2364c10>,
>> <Element 'doi' at 0x2364c50>,
>> <Element 'baselabel' at 0x2364c90>,
>> <Element 'baselabel' at 0x2364cd0>],
>> <Element 'model' at 0x2364c10>,
>> { '_children': [],
>> 'attrib': { },
>> 'tag': 'model',
>> 'tail': '\n ',
>> 'text': 'dac'},
>> [],
>> <Element 'doi' at 0x2364c50>,
>> { '_children': [],
>> 'attrib': { },
>> 'tag': 'doi',
>> 'tail': '\n ',
>> 'text': '0'},
>> [],
>> <Element 'baselabel' at 0x2364c90>,
>> { '_children': [],
>> 'attrib': { 'type': 'kvm'},
>> 'tag': 'baselabel',
>> 'tail': '\n ',
>> 'text': '+107:+107'},
>> [],
>> <Element 'baselabel' at 0x2364cd0>,
>> { '_children': [],
>> 'attrib': { 'type': 'qemu'},
>> 'tag': 'baselabel',
>> 'tail': '\n ',
>> 'text': '+107:+107'},
>> [],
>> <Element 'guest' at 0x2364d10>,
>> { '_children': [ <Element 'os_type' at 0x2364d50>,
>> <Element 'arch' at 0x2364d90>,
>> <Element 'features' at 0x2369390>],
>> 'attrib': { },
>> 'tag': 'guest',
>> 'tail': '\n\n ',
>> 'text': '\n '},
>> [ <Element 'os_type' at 0x2364d50>,
>> <Element 'arch' at 0x2364d90>,
>> <Element 'features' at 0x2369390>],
>> <Element 'os_type' at 0x2364d50>,
>> { '_children': [],
>> 'attrib': { },
>> 'tag': 'os_type',
>> 'tail': '\n ',
>> 'text': 'hvm'},
>> [],
>> <Element 'arch' at 0x2364d90>,
>> { '_children': [ <Element 'wordsize' at 0x2364dd0>,
>> <Element 'emulator' at 0x2364e10>,
>> <Element 'machine' at 0x2364e90>,
>> <Element 'machine' at 0x2364ed0>,
>> <Element 'machine' at 0x2364f10>,
>> <Element 'machine' at 0x2364f50>,
>> <Element 'machine' at 0x2364f90>,
>> <Element 'machine' at 0x2364fd0>,
>> <Element 'machine' at 0x2369050>,
>> <Element 'machine' at 0x2369090>,
>> <Element 'machine' at 0x23690d0>,
>> <Element 'machine' at 0x2369110>,
>> <Element 'machine' at 0x2369150>,
>> <Element 'machine' at 0x2369190>,
>> <Element 'machine' at 0x23691d0>,
>> <Element 'machine' at 0x2369210>,
>> <Element 'machine' at 0x2369250>,
>> <Element 'domain' at 0x2369290>,
>> <Element 'domain' at 0x23692d0>],
>> 'attrib': { 'name': 'i686'},
>> 'tag': 'arch',
>> 'tail': '\n ',
>> 'text': '\n '},
>> [ <Element 'wordsize' at 0x2364dd0>,
>> <Element 'emulator' at 0x2364e10>,
>> <Element 'machine' at 0x2364e90>,
>> <Element 'machine' at 0x2364ed0>,
>> <Element 'machine' at 0x2364f10>,
>> <Element 'machine' at 0x2364f50>,
>> <Element 'machine' at 0x2364f90>,
>> <Element 'machine' at 0x2364fd0>,
>> <Element 'machine' at 0x2369050>,
>> <Element 'machine' at 0x2369090>,
>> <Element 'machine' at 0x23690d0>,
>> <Element 'machine' at 0x2369110>,
>> <Element 'machine' at 0x2369150>,
>> <Element 'machine' at 0x2369190>,
>> <Element 'machine' at 0x23691d0>,
>> <Element 'machine' at 0x2369210>,
>> <Element 'machine' at 0x2369250>,
>> <Element 'domain' at 0x2369290>,
>> <Element 'domain' at 0x23692d0>],
>> <Element 'wordsize' at 0x2364dd0>,
>> { '_children': [],
>> 'attrib': { },
>> 'tag': 'wordsize',
>> 'tail': '\n ',
>> 'text': '32'},
>> [],
>> <Element 'emulator' at 0x2364e10>,
>> { '_children': [],
>> 'attrib': { },
>> 'tag': 'emulator',
>> 'tail': '\n ',
>> 'text': '/usr/libexec/qemu-kvm'},
>> [],
>> <Element 'machine' at 0x2364e90>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'pc-i440fx-rhel7.2.0'},
>> [],
>> <Element 'machine' at 0x2364ed0>,
>> { '_children': [],
>> 'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'pc'},
>> [],
>> <Element 'machine' at 0x2364f10>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'pc-i440fx-rhel7.0.0'},
>> [],
>> <Element 'machine' at 0x2364f50>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'pc-q35-rhel7.1.0'},
>> [],
>> <Element 'machine' at 0x2364f90>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'rhel6.3.0'},
>> [],
>> <Element 'machine' at 0x2364fd0>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'pc-q35-rhel7.2.0'},
>> [],
>> <Element 'machine' at 0x2369050>,
>> { '_children': [],
>> 'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'q35'},
>> [],
>> <Element 'machine' at 0x2369090>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'rhel6.4.0'},
>> [],
>> <Element 'machine' at 0x23690d0>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'rhel6.0.0'},
>> [],
>> <Element 'machine' at 0x2369110>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'pc-i440fx-rhel7.1.0'},
>> [],
>> <Element 'machine' at 0x2369150>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'rhel6.5.0'},
>> [],
>> <Element 'machine' at 0x2369190>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'rhel6.6.0'},
>> [],
>> <Element 'machine' at 0x23691d0>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'rhel6.1.0'},
>> [],
>> <Element 'machine' at 0x2369210>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'pc-q35-rhel7.0.0'},
>> [],
>> <Element 'machine' at 0x2369250>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'rhel6.2.0'},
>> [],
>> <Element 'domain' at 0x2369290>,
>> { '_children': [],
>> 'attrib': { 'type': 'qemu'},
>> 'tag': 'domain',
>> 'tail': '\n '},
>> [],
>> <Element 'domain' at 0x23692d0>,
>> { '_children': [<Element 'emulator' at 0x2369310>],
>> 'attrib': { 'type': 'kvm'},
>> 'tag': 'domain',
>> 'tail': '\n ',
>> 'text': '\n '},
>> [<Element 'emulator' at 0x2369310>],
>> <Element 'emulator' at 0x2369310>,
>> { '_children': [],
>> 'attrib': { },
>> 'tag': 'emulator',
>> 'tail': '\n ',
>> 'text': '/usr/libexec/qemu-kvm'},
>> [],
>> <Element 'features' at 0x2369390>,
>> { '_children': [ <Element 'cpuselection' at 0x23693d0>,
>> <Element 'deviceboot' at 0x2369410>,
>> <Element 'disksnapshot' at 0x2369450>,
>> <Element 'acpi' at 0x2369490>,
>> <Element 'apic' at 0x23694d0>,
>> <Element 'pae' at 0x2369510>,
>> <Element 'nonpae' at 0x2369550>],
>> 'attrib': { },
>> 'tag': 'features',
>> 'tail': '\n ',
>> 'text': '\n '},
>> [ <Element 'cpuselection' at 0x23693d0>,
>> <Element 'deviceboot' at 0x2369410>,
>> <Element 'disksnapshot' at 0x2369450>,
>> <Element 'acpi' at 0x2369490>,
>> <Element 'apic' at 0x23694d0>,
>> <Element 'pae' at 0x2369510>,
>> <Element 'nonpae' at 0x2369550>],
>> <Element 'cpuselection' at 0x23693d0>,
>> { '_children': [], 'attrib': { }, 'tag': 'cpuselection',
>> 'tail': '\n '},
>> [],
>> <Element 'deviceboot' at 0x2369410>,
>> { '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail':
>> '\n '},
>> [],
>> <Element 'disksnapshot' at 0x2369450>,
>> { '_children': [],
>> 'attrib': { 'default': 'on', 'toggle': 'no'},
>> 'tag': 'disksnapshot',
>> 'tail': '\n '},
>> [],
>> <Element 'acpi' at 0x2369490>,
>> { '_children': [],
>> 'attrib': { 'default': 'on', 'toggle': 'yes'},
>> 'tag': 'acpi',
>> 'tail': '\n '},
>> [],
>> <Element 'apic' at 0x23694d0>,
>> { '_children': [],
>> 'attrib': { 'default': 'on', 'toggle': 'no'},
>> 'tag': 'apic',
>> 'tail': '\n '},
>> [],
>> <Element 'pae' at 0x2369510>,
>> { '_children': [], 'attrib': { }, 'tag': 'pae', 'tail': '\n '},
>> [],
>> <Element 'nonpae' at 0x2369550>,
>> { '_children': [], 'attrib': { }, 'tag': 'nonpae', 'tail': '\n '},
>> [],
>> <Element 'guest' at 0x2369590>,
>> { '_children': [ <Element 'os_type' at 0x23695d0>,
>> <Element 'arch' at 0x2369610>,
>> <Element 'features' at 0x2369bd0>],
>> 'attrib': { },
>> 'tag': 'guest',
>> 'tail': '\n\n',
>> 'text': '\n '},
>> [ <Element 'os_type' at 0x23695d0>,
>> <Element 'arch' at 0x2369610>,
>> <Element 'features' at 0x2369bd0>],
>> <Element 'os_type' at 0x23695d0>,
>> { '_children': [],
>> 'attrib': { },
>> 'tag': 'os_type',
>> 'tail': '\n ',
>> 'text': 'hvm'},
>> [],
>> <Element 'arch' at 0x2369610>,
>> { '_children': [ <Element 'wordsize' at 0x2369650>,
>> <Element 'emulator' at 0x2369690>,
>> <Element 'machine' at 0x2369710>,
>> <Element 'machine' at 0x2369750>,
>> <Element 'machine' at 0x2369790>,
>> <Element 'machine' at 0x23697d0>,
>> <Element 'machine' at 0x2369810>,
>> <Element 'machine' at 0x2369850>,
>> <Element 'machine' at 0x2369890>,
>> <Element 'machine' at 0x23698d0>,
>> <Element 'machine' at 0x2369910>,
>> <Element 'machine' at 0x2369950>,
>> <Element 'machine' at 0x2369990>,
>> <Element 'machine' at 0x23699d0>,
>> <Element 'machine' at 0x2369a10>,
>> <Element 'machine' at 0x2369a50>,
>> <Element 'machine' at 0x2369a90>,
>> <Element 'domain' at 0x2369ad0>,
>> <Element 'domain' at 0x2369b10>],
>> 'attrib': { 'name': 'x86_64'},
>> 'tag': 'arch',
>> 'tail': '\n ',
>> 'text': '\n '},
>> [ <Element 'wordsize' at 0x2369650>,
>> <Element 'emulator' at 0x2369690>,
>> <Element 'machine' at 0x2369710>,
>> <Element 'machine' at 0x2369750>,
>> <Element 'machine' at 0x2369790>,
>> <Element 'machine' at 0x23697d0>,
>> <Element 'machine' at 0x2369810>,
>> <Element 'machine' at 0x2369850>,
>> <Element 'machine' at 0x2369890>,
>> <Element 'machine' at 0x23698d0>,
>> <Element 'machine' at 0x2369910>,
>> <Element 'machine' at 0x2369950>,
>> <Element 'machine' at 0x2369990>,
>> <Element 'machine' at 0x23699d0>,
>> <Element 'machine' at 0x2369a10>,
>> <Element 'machine' at 0x2369a50>,
>> <Element 'machine' at 0x2369a90>,
>> <Element 'domain' at 0x2369ad0>,
>> <Element 'domain' at 0x2369b10>],
>> <Element 'wordsize' at 0x2369650>,
>> { '_children': [],
>> 'attrib': { },
>> 'tag': 'wordsize',
>> 'tail': '\n ',
>> 'text': '64'},
>> [],
>> <Element 'emulator' at 0x2369690>,
>> { '_children': [],
>> 'attrib': { },
>> 'tag': 'emulator',
>> 'tail': '\n ',
>> 'text': '/usr/libexec/qemu-kvm'},
>> [],
>> <Element 'machine' at 0x2369710>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'pc-i440fx-rhel7.2.0'},
>> [],
>> <Element 'machine' at 0x2369750>,
>> { '_children': [],
>> 'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'pc'},
>> [],
>> <Element 'machine' at 0x2369790>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'pc-i440fx-rhel7.0.0'},
>> [],
>> <Element 'machine' at 0x23697d0>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'pc-q35-rhel7.1.0'},
>> [],
>> <Element 'machine' at 0x2369810>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'rhel6.3.0'},
>> [],
>> <Element 'machine' at 0x2369850>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'pc-q35-rhel7.2.0'},
>> [],
>> <Element 'machine' at 0x2369890>,
>> { '_children': [],
>> 'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'q35'},
>> [],
>> <Element 'machine' at 0x23698d0>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'rhel6.4.0'},
>> [],
>> <Element 'machine' at 0x2369910>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'rhel6.0.0'},
>> [],
>> <Element 'machine' at 0x2369950>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'pc-i440fx-rhel7.1.0'},
>> [],
>> <Element 'machine' at 0x2369990>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'rhel6.5.0'},
>> [],
>> <Element 'machine' at 0x23699d0>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'rhel6.6.0'},
>> [],
>> <Element 'machine' at 0x2369a10>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'rhel6.1.0'},
>> [],
>> <Element 'machine' at 0x2369a50>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'pc-q35-rhel7.0.0'},
>> [],
>> <Element 'machine' at 0x2369a90>,
>> { '_children': [],
>> 'attrib': { 'maxCpus': '240'},
>> 'tag': 'machine',
>> 'tail': '\n ',
>> 'text': 'rhel6.2.0'},
>> [],
>> <Element 'domain' at 0x2369ad0>,
>> { '_children': [],
>> 'attrib': { 'type': 'qemu'},
>> 'tag': 'domain',
>> 'tail': '\n '},
>> [],
>> <Element 'domain' at 0x2369b10>,
>> { '_children': [<Element 'emulator' at 0x2369b50>],
>> 'attrib': { 'type': 'kvm'},
>> 'tag': 'domain',
>> 'tail': '\n ',
>> 'text': '\n '},
>> [<Element 'emulator' at 0x2369b50>],
>> <Element 'emulator' at 0x2369b50>,
>> { '_children': [],
>> 'attrib': { },
>> 'tag': 'emulator',
>> 'tail': '\n ',
>> 'text': '/usr/libexec/qemu-kvm'},
>> [],
>> <Element 'features' at 0x2369bd0>,
>> { '_children': [ <Element 'cpuselection' at 0x2369c10>,
>> <Element 'deviceboot' at 0x2369c50>,
>> <Element 'disksnapshot' at 0x2369c90>,
>> <Element 'acpi' at 0x2369cd0>,
>> <Element 'apic' at 0x2369d10>],
>> 'attrib': { },
>> 'tag': 'features',
>> 'tail': '\n ',
>> 'text': '\n '},
>> [ <Element 'cpuselection' at 0x2369c10>,
>> <Element 'deviceboot' at 0x2369c50>,
>> <Element 'disksnapshot' at 0x2369c90>,
>> <Element 'acpi' at 0x2369cd0>,
>> <Element 'apic' at 0x2369d10>],
>> <Element 'cpuselection' at 0x2369c10>,
>> { '_children': [], 'attrib': { }, 'tag': 'cpuselection',
>> 'tail': '\n '},
>> [],
>> <Element 'deviceboot' at 0x2369c50>,
>> { '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail':
>> '\n '},
>> [],
>> <Element 'disksnapshot' at 0x2369c90>,
>> { '_children': [],
>> 'attrib': { 'default': 'on', 'toggle': 'no'},
>> 'tag': 'disksnapshot',
>> 'tail': '\n '},
>> [],
>> <Element 'acpi' at 0x2369cd0>,
>> { '_children': [],
>> 'attrib': { 'default': 'on', 'toggle': 'yes'},
>> 'tag': 'acpi',
>> 'tail': '\n '},
>> [],
>> <Element 'apic' at 0x2369d10>,
>> { '_children': [],
>> 'attrib': { 'default': 'on', 'toggle': 'no'},
>> 'tag': 'apic',
>> 'tail': '\n '},
>> [],
>> [],
>> (<type '_ctypes.Array'>,),
>> { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>,
>> '__doc__': None,
>> '__module__': 'vdsm.netlink',
>> '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>,
>> '_length_': 60,
>> '_type_': <class 'ctypes.c_char'>,
>> 'raw': <attribute 'raw' of 'c_char_Array_60' objects>,
>> 'value': <attribute 'value' of 'c_char_Array_60' objects>},
>> <class 'vdsm.netlink.c_char_Array_60'>,
>> <attribute '__dict__' of 'c_char_Array_60' objects>,
>> <attribute '__weakref__' of 'c_char_Array_60' objects>,
>> ( <class 'vdsm.netlink.c_char_Array_60'>,
>> <type '_ctypes.Array'>,
>> <type '_ctypes._CData'>,
>> <type 'object'>),
>> <attribute 'raw' of 'c_char_Array_60' objects>,
>> <attribute 'value' of 'c_char_Array_60' objects>,
>> <xml.etree.ElementTree.XMLParser object at 0x2357f50>,
>> { '_doctype': None,
>> '_error': <class 'xml.parsers.expat.ExpatError'>,
>> '_names': { u'acpi': 'acpi',
>> u'apic': 'apic',
>> u'arch': 'arch',
>> u'baselabel': 'baselabel',
>> u'canonical': 'canonical',
>> u'capabilities': 'capabilities',
>> u'cell': 'cell',
>> u'cells': 'cells',
>> u'core_id': 'core_id',
>> u'cores': 'cores',
>> u'cpu': 'cpu',
>> u'cpus': 'cpus',
>> u'cpuselection': 'cpuselection',
>> u'default': 'default',
>> u'deviceboot': 'deviceboot',
>> u'disksnapshot': 'disksnapshot',
>> u'distances': 'distances',
>> u'doi': 'doi',
>> u'domain': 'domain',
>> u'emulator': 'emulator',
>> u'feature': 'feature',
>> u'features': 'features',
>> u'guest': 'guest',
>> u'host': 'host',
>> u'id': 'id',
>> u'live': 'live',
>> u'machine': 'machine',
>> u'maxCpus': 'maxCpus',
>> u'memory': 'memory',
>> u'migration_features': 'migration_features',
>> u'model': 'model',
>> u'name': 'name',
>> u'nonpae': 'nonpae',
>> u'num': 'num',
>> u'os_type': 'os_type',
>> u'pae': 'pae',
>> u'pages': 'pages',
>> u'power_management': 'power_management',
>> u'secmodel': 'secmodel',
>> u'sibling': 'sibling',
>> u'siblings': 'siblings',
>> u'size': 'size',
>> u'socket_id': 'socket_id',
>> u'sockets': 'sockets',
>> u'suspend_disk': 'suspend_disk',
>> u'suspend_hybrid': 'suspend_hybrid',
>> u'suspend_mem': 'suspend_mem',
>> u'threads': 'threads',
>> u'toggle': 'toggle',
>> u'topology': 'topology',
>> u'type': 'type',
>> u'unit': 'unit',
>> u'uri_transport': 'uri_transport',
>> u'uri_transports': 'uri_transports',
>> u'uuid': 'uuid',
>> u'value': 'value',
>> u'vendor': 'vendor',
>> u'wordsize': 'wordsize'},
>> '_target': <xml.etree.ElementTree.TreeBuilder object at 0x2357f10>,
>> 'entity': { },
>> 'parser': <pyexpat.xmlparser object at 0x22e5360>,
>> 'version': 'Expat 2.1.0'},
>> <pyexpat.xmlparser object at 0x22e5360>,
>> <xml.etree.ElementTree.TreeBuilder object at 0x2357f10>,
>> <bound method XMLParser._start_list of
>> <xml.etree.ElementTree.XMLParser object at 0x2357f50>>,
>> <bound method XMLParser._end of <xml.etree.ElementTree.XMLParser
>> object at 0x2357f50>>,
>> <bound method XMLParser._pi of <xml.etree.ElementTree.XMLParser
>> object at 0x2357f50>>,
>> <bound method XMLParser._data of <xml.etree.ElementTree.XMLParser
>> object at 0x2357f50>>,
>> <bound method XMLParser._comment of
>> <xml.etree.ElementTree.XMLParser object at 0x2357f50>>,
>> <bound method XMLParser._default of
>> <xml.etree.ElementTree.XMLParser object at 0x2357f50>>,
>> { '_data': [],
>> '_elem': [],
>> '_factory': <class 'xml.etree.ElementTree.Element'>,
>> '_last': <Element 'capabilities' at 0x2357f90>,
>> '_tail': 1},
>> [],
>> <Element 'cell' at 0x23646d0>,
>> <Element 'cells' at 0x2364690>,
>> <Element 'topology' at 0x2364650>,
>> <Element 'host' at 0x2357fd0>,
>> <Element 'capabilities' at 0x2357f90>,
>> { '_children': [ <Element 'memory' at 0x2364710>,
>> <Element 'pages' at 0x2364750>,
>> <Element 'pages' at 0x2364790>,
>> <Element 'distances' at 0x23647d0>,
>> <Element 'cpus' at 0x2364850>],
>> 'attrib': { 'id': '0'},
>> 'tag': 'cell',
>> 'tail': '\n ',
>> 'text': '\n '},
>> { '_children': [<Element 'cell' at 0x23646d0>],
>> 'attrib': { 'num': '1'},
>> 'tag': 'cells',
>> 'tail': '\n ',
>> 'text': '\n '},
>> { '_children': [<Element 'cells' at 0x2364690>],
>> 'attrib': { },
>> 'tag': 'topology',
>> 'tail': '\n ',
>> 'text': '\n '},
>> { '_children': [ <Element 'uuid' at 0x2364050>,
>> <Element 'cpu' at 0x2364090>,
>> <Element 'power_management' at 0x2364390>,
>> <Element 'migration_features' at 0x23644d0>,
>> <Element 'topology' at 0x2364650>,
>> <Element 'secmodel' at 0x2364a90>,
>> <Element 'secmodel' at 0x2364bd0>],
>> 'attrib': { },
>> 'tag': 'host',
>> 'tail': '\n\n ',
>> 'text': '\n '},
>> { '_children': [ <Element 'host' at 0x2357fd0>,
>> <Element 'guest' at 0x2364d10>,
>> <Element 'guest' at 0x2369590>],
>> 'attrib': { },
>> 'tag': 'capabilities',
>> 'text': '\n\n '},
>> [ <Element 'memory' at 0x2364710>,
>> <Element 'pages' at 0x2364750>,
>> <Element 'pages' at 0x2364790>,
>> <Element 'distances' at 0x23647d0>,
>> <Element 'cpus' at 0x2364850>],
>> [<Element 'cell' at 0x23646d0>],
>> [<Element 'cells' at 0x2364690>],
>> [ <Element 'uuid' at 0x2364050>,
>> <Element 'cpu' at 0x2364090>,
>> <Element 'power_management' at 0x2364390>,
>> <Element 'migration_features' at 0x23644d0>,
>> <Element 'topology' at 0x2364650>,
>> <Element 'secmodel' at 0x2364a90>,
>> <Element 'secmodel' at 0x2364bd0>],
>> [ <Element 'host' at 0x2357fd0>,
>> <Element 'guest' at 0x2364d10>,
>> <Element 'guest' at 0x2369590>],
>> <Element 'migration_features' at 0x23644d0>,
>> <Element 'power_management' at 0x2364390>,
>> <Element 'cpu' at 0x2364090>,
>> <Element 'uuid' at 0x2364050>,
>> { '_children': [ <Element 'live' at 0x2364510>,
>> <Element 'uri_transports' at 0x2364590>],
>> 'attrib': { },
>> 'tag': 'migration_features',
>> 'tail': '\n ',
>> 'text': '\n '},
>> { '_children': [ <Element 'suspend_mem' at 0x23643d0>,
>> <Element 'suspend_disk' at 0x2364410>,
>> <Element 'suspend_hybrid' at 0x2364490>],
>> 'attrib': { },
>> 'tag': 'power_management',
>> 'tail': '\n ',
>> 'text': '\n '},
>> { '_children': [ <Element 'arch' at 0x23640d0>,
>> <Element 'model' at 0x2364110>,
>> <Element 'vendor' at 0x2364150>,
>> <Element 'topology' at 0x2364190>,
>> <Element 'feature' at 0x23641d0>,
>> <Element 'feature' at 0x2364210>,
>> <Element 'feature' at 0x2364250>,
>> <Element 'feature' at 0x2364290>,
>> <Element 'feature' at 0x23642d0>,
>> <Element 'pages' at 0x2364310>,
>> <Element 'pages' at 0x2364350>],
>> 'attrib': { },
>> 'tag': 'cpu',
>> 'tail': '\n ',
>> 'text': '\n '},
>> { '_children': [],
>> 'attrib': { },
>> 'tag': 'uuid',
>> 'tail': '\n ',
>> 'text': 'a920d963-c18f-4fda-aa5a-deee9bbcde9d'},
>> [<Element 'live' at 0x2364510>, <Element 'uri_transports' at 0x2364590>],
>> [ <Element 'suspend_mem' at 0x23643d0>,
>> <Element 'suspend_disk' at 0x2364410>,
>> <Element 'suspend_hybrid' at 0x2364490>],
>> [ <Element 'arch' at 0x23640d0>,
>> <Element 'model' at 0x2364110>,
>> <Element 'vendor' at 0x2364150>,
>> <Element 'topology' at 0x2364190>,
>> <Element 'feature' at 0x23641d0>,
>> <Element 'feature' at 0x2364210>,
>> <Element 'feature' at 0x2364250>,
>> <Element 'feature' at 0x2364290>,
>> <Element 'feature' at 0x23642d0>,
>> <Element 'pages' at 0x2364310>,
>> <Element 'pages' at 0x2364350>],
>> [],
>> <Element 'uri_transports' at 0x2364590>,
>> <Element 'live' at 0x2364510>,
>> <Element 'suspend_hybrid' at 0x2364490>,
>> <Element 'suspend_disk' at 0x2364410>,
>> <Element 'suspend_mem' at 0x23643d0>,
>> <Element 'pages' at 0x2364350>,
>> <Element 'pages' at 0x2364310>,
>> <Element 'feature' at 0x23642d0>,
>> <Element 'feature' at 0x2364290>,
>> <Element 'feature' at 0x2364250>,
>> <Element 'feature' at 0x2364210>,
>> <Element 'feature' at 0x23641d0>,
>> <Element 'topology' at 0x2364190>,
>> <Element 'vendor' at 0x2364150>,
>> <Element 'model' at 0x2364110>,
>> <Element 'arch' at 0x23640d0>,
>> { '_children': [ <Element 'uri_transport' at 0x23645d0>,
>> <Element 'uri_transport' at 0x2364610>],
>> 'attrib': { },
>> 'tag': 'uri_transports',
>> 'tail': '\n ',
>> 'text': '\n '},
>> { '_children': [], 'attrib': { }, 'tag': 'live', 'tail': '\n '},
>> { '_children': [], 'attrib': { }, 'tag': 'suspend_hybrid',
>> 'tail': '\n '},
>> { '_children': [], 'attrib': { }, 'tag': 'suspend_disk',
>> 'tail': '\n '},
>> { '_children': [], 'attrib': { }, 'tag': 'suspend_mem',
>> 'tail': '\n '},
>> { '_children': [],
>> 'attrib': { 'size': '2048', 'unit': 'KiB'},
>> 'tag': 'pages',
>> 'tail': '\n '},
>> { '_children': [],
>> 'attrib': { 'size': '4', 'unit': 'KiB'},
>> 'tag': 'pages',
>> 'tail': '\n '},
>> { '_children': [],
>> 'attrib': { 'name': 'ss'},
>> 'tag': 'feature',
>> 'tail': '\n '},
>> { '_children': [],
>> 'attrib': { 'name': 'vmx'},
>> 'tag': 'feature',
>> 'tail': '\n '},
>> { '_children': [],
>> 'attrib': { 'name': 'pcid'},
>> 'tag': 'feature',
>> 'tail': '\n '},
>> { '_children': [],
>> 'attrib': { 'name': 'osxsave'},
>> 'tag': 'feature',
>> 'tail': '\n '},
>> { '_children': [],
>> 'attrib': { 'name': 'hypervisor'},
>> 'tag': 'feature',
>> 'tail': '\n '},
>> { '_children': [],
>> 'attrib': { 'cores': '1', 'sockets': '8', 'threads': '1'},
>> 'tag': 'topology',
>> 'tail': '\n '},
>> { '_children': [],
>> 'attrib': { },
>> 'tag': 'vendor',
>> 'tail': '\n ',
>> 'text': 'Intel'},
>> { '_children': [],
>> 'attrib': { },
>> 'tag': 'model',
>> 'tail': '\n ',
>> 'text': 'IvyBridge'},
>> { '_children': [],
>> 'attrib': { },
>> 'tag': 'arch',
>> 'tail': '\n ',
>> 'text': 'x86_64'},
>> [ <Element 'uri_transport' at 0x23645d0>,
>> <Element 'uri_transport' at 0x2364610>],
>> [],
>> [],
>> [],
>> [],
>> [],
>> [],
>> [],
>> [],
>> [],
>> [],
>> [],
>> [],
>> [],
>> [],
>> [],
>> <Element 'uri_transport' at 0x2364610>,
>> <Element 'uri_transport' at 0x23645d0>,
>> { '_children': [],
>> 'attrib': { },
>> 'tag': 'uri_transport',
>> 'tail': '\n ',
>> 'text': 'rdma'},
>> { '_children': [],
>> 'attrib': { },
>> 'tag': 'uri_transport',
>> 'tail': '\n ',
>> 'text': 'tcp'},
>> [],
>> [],
>> (<type '_ctypes.Array'>,),
>> { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>,
>> '__doc__': None,
>> '__module__': 'vdsm.netlink',
>> '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>,
>> '_length_': 60,
>> '_type_': <class 'ctypes.c_char'>,
>> 'raw': <attribute 'raw' of 'c_char_Array_60' objects>,
>> 'value': <attribute 'value' of 'c_char_Array_60' objects>},
>> <class 'vdsm.netlink.c_char_Array_60'>,
>> <attribute '__dict__' of 'c_char_Array_60' objects>,
>> <attribute '__weakref__' of 'c_char_Array_60' objects>,
>> ( <class 'vdsm.netlink.c_char_Array_60'>,
>> <type '_ctypes.Array'>,
>> <type '_ctypes._CData'>,
>> <type 'object'>),
>> <attribute 'raw' of 'c_char_Array_60' objects>,
>> <attribute 'value' of 'c_char_Array_60' objects>]
More information about the Devel
mailing list