Here are example logs with running and stopping one vm: # tail -f /var/log/vdsm/vdsm.log | grep health Thread-13::DEBUG::2016-01-14 19:42:04,625::health::84::health::(_check) Checking health Thread-13::DEBUG::2016-01-14 19:42:04,651::health::86::health::(_check) Collected 25 objects Thread-13::WARNING::2016-01-14 19:42:04,656::health::100::health::(_check) Found 6085 uncollectible objects, 0 suspects implementing __del__: [] Vm was started here... Thread-13::DEBUG::2016-01-14 19:43:04,657::health::84::health::(_check) Checking health Thread-13::DEBUG::2016-01-14 19:43:04,688::health::86::health::(_check) Collected 1680 objects Thread-13::WARNING::2016-01-14 19:43:04,697::health::100::health::(_check) Found 11041 uncollectible objects, 0 suspects implementing __del__: [] Thread-13::DEBUG::2016-01-14 19:44:04,698::health::84::health::(_check) Checking health Thread-13::DEBUG::2016-01-14 19:44:04,729::health::86::health::(_check) Collected 57 objects Thread-13::WARNING::2016-01-14 19:44:04,737::health::100::health::(_check) Found 11096 uncollectible objects, 0 suspects implementing __del__: [] Thread-13::DEBUG::2016-01-14 19:45:04,738::health::84::health::(_check) Checking health Thread-13::DEBUG::2016-01-14 19:45:04,769::health::86::health::(_check) Collected 69 objects Thread-13::WARNING::2016-01-14 19:45:04,778::health::100::health::(_check) Found 11163 uncollectible objects, 0 suspects implementing __del__: [] Vm was shutdown here... Thread-13::DEBUG::2016-01-14 19:46:04,779::health::84::health::(_check) Checking health Thread-13::DEBUG::2016-01-14 19:46:04,809::health::86::health::(_check) Collected 2272 objects Thread-13::WARNING::2016-01-14 19:46:04,819::health::100::health::(_check) Found 13464 uncollectible objects, 28 suspects implementing __del__: ['<pthreading.Lock object at 0x7f1d8016f908>', '<pthreading.Lock object at 0x7f1d801804d0>', '<pthreading.Lock object at 0x7f1cf8010f38>', '<pthreading.Lock object at 0x7f1d8016f7a0>', '<pthreading.Lock object at 0x7f1d20003680>', '<pthreading.Lock object at 0x7f1da8112998>', '<pthreading.Lock object at 0x7f1d8016fa28>', '<pthreading.Lock object at 0x7f1da8112830>', '<pthread.Cond object at 0x7f1da811cf50>', '<pthreading.Lock object at 0x7f1d8016f200>', '<pthread.Cond object at 0x7f1da811ca50>', '<pthreading.Lock object at 0x7f1cf8010fc8>', '<pthread.Cond object at 0x7f1da8115c50>', '<pthreading.Lock object at 0x7f1da8112878>', '<pthread.Cond object at 0x7f1da811c550>', '<pthreading.Lock object at 0x7f1d8016f9e0>', '<pthread.Cond object at 0x7f1da811ca90>', '<pthreading.Lock object at 0x7f1d8016f680>', '<pthread.Cond object at 0x7f1da811cc50>', '<libvirt.virDomain object at 0x1ba8210>', '<pthreading.Lock object at 0x7f1d8016f758>', '<pthread.Cond object at 0x7f1da811cb50>', '<pthreading.Lock object at 0x7f1d8016fb90>', '<pthread.Cond object at 0x7f1da811cc10>', '<pthread.Cond object at 0x233ed50>', '<pthreading.Lock object at 0x7f1ce0227c20>', '<pthreading.Lock object at 0x7f1ce0227ab8>', '<pthread.Cond object at 0x233ed10>'] Need to test with should be fixed by https://gerrit.ovirt.org/51630, which should eliminate '<libvirt.virDomain object at 0x1ba8210>', and maybe also the pthreading objects. Nir On Thu, Jan 14, 2016 at 7:38 PM, Nir Soffer <nsoffer@redhat.com> wrote:
Hi all,
Continuing the leak investigation Francesco and Milan are working on, I posted this patch, adding leak health monitoring to vdsm [1]. This patch currently monitor collectible objects, and we may add other interesting stuff later if needed.
To enable the monitor, you must set:
[devel] health_monitor_enable = true
In vdsm.conf, and restart vdsm.
Here is an example logs - this was logged 60 seconds after starting vdsm, when running as spm:
Thread-13::DEBUG::2016-01-14 18:53:43,239::health::84::health::(_check) Checking health Thread-13::DEBUG::2016-01-14 18:53:43,272::health::86::health::(_check) Collected 460 objects Thread-13::WARNING::2016-01-14 18:53:43,277::health::100::health::(_check) Found 5873 uncollectible objects, 10 suspects implementing __del__: ['<pthreading.Lock object at 0x7f012c115320>', '<pthreading.Lock object at 0x7f012c115440>', '<pthreading.Lock object at 0x7f012c1153b0>', '<pthreading.Lock object at 0x7f012c1154d0>', '<pthread.Cond object at 0x7f012c111350>', '<pthread.Cond object at 0x7f012c1112d0>', '<pthread.Cond object at 0x7f012c111250>', '<pthread.Cond object at 0x7f012c111850>', '<pthread.Cond object at 0x7f012c1117d0>', '<pthread.Cond object at 0x7f012c111750>']
We found 460 uncollectible objects, 10 of them are objects implementing __del__. Having such object in a reference cycle make the entire cycle uncollectible.
The pthreading objects in this log grow as vdsm continue to run, even without any interaction with engine. The pthreading objects leak is caused by a cycle in storage.misc.RWLock, fixed in patch [2].
After solving the pthreading leak, we still have many uncollectible objects in the logs, listing them, it seems that the suspect is hiding inside a collection (dict, list, tuple) and does not appear in the gc.garbage list. I will address this in later patch, searching in the entire object graph instead of the top level of gc.garbage.
See end of this mail for list of these objects. I printed them like this:
Enable manhole debugging shell:
[devel] manhole_enable = true
Connect to vdsm using manhole:
nc -U /run/vdsm/vdsmd.manhole
Print garbage:
import pprint import gc with open('/tmp/garbage.txt', 'w') as f: pprint.pprint(gc.garbage, stream=f, indent=4)
These objects looks related to libvirt - Francesco, can you take a look?
[1] https://gerrit.ovirt.org/51708 [2] https://gerrit.ovirt.org/51868
Nir
--- garbage after patch [2] ---
[ (<type '_ctypes.Array'>,), { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>, '__doc__': None, '__module__': 'vdsm.netlink', '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>, '_length_': 60, '_type_': <class 'ctypes.c_char'>, 'raw': <attribute 'raw' of 'c_char_Array_60' objects>, 'value': <attribute 'value' of 'c_char_Array_60' objects>}, <class 'vdsm.netlink.c_char_Array_60'>, <attribute '__dict__' of 'c_char_Array_60' objects>, <attribute '__weakref__' of 'c_char_Array_60' objects>, ( <class 'vdsm.netlink.c_char_Array_60'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_60' objects>, <attribute 'value' of 'c_char_Array_60' objects>, <cell at 0x23662f0: bool object at 0x7f1e086c8c40>, <cell at 0x2366328: NoneType object at 0x7f1e086dcf00>, <cell at 0x2366360: function object at 0x233f488>, ( <cell at 0x23662f0: bool object at 0x7f1e086c8c40>, <cell at 0x2366328: NoneType object at 0x7f1e086dcf00>, <cell at 0x2366360: function object at 0x233f488>), <function wrapMethod at 0x233f488>, [ [ <Recursion on list with id=34033464>, <Recursion on list with id=34033464>, 'class'], [ <Recursion on list with id=34033464>, <Recursion on list with id=34033464>, 'class'], None], [ [ <Recursion on list with id=33979064>, <Recursion on list with id=33979064>, None], [ <Recursion on list with id=33979064>, <Recursion on list with id=33979064>, None], 'class'], [ [ <Recursion on list with id=35535256>, <Recursion on list with id=35535256>, 'class'], [ <Recursion on list with id=35535256>, <Recursion on list with id=35535256>, 'class'], None], [ [ <Recursion on list with id=35535832>, <Recursion on list with id=35535832>, None], [ <Recursion on list with id=35535832>, <Recursion on list with id=35535832>, None], 'class'], [ [ <Recursion on list with id=35556312>, <Recursion on list with id=35556312>, 'class'], [ <Recursion on list with id=35556312>, <Recursion on list with id=35556312>, 'class'], None], [ [ <Recursion on list with id=35556600>, <Recursion on list with id=35556600>, None], [ <Recursion on list with id=35556600>, <Recursion on list with id=35556600>, None], 'class'], [ [ <Recursion on list with id=35556960>, <Recursion on list with id=35556960>, 'class'], [ <Recursion on list with id=35556960>, <Recursion on list with id=35556960>, 'class'], None], [ [ <Recursion on list with id=35561912>, <Recursion on list with id=35561912>, None], [ <Recursion on list with id=35561912>, <Recursion on list with id=35561912>, None], 'class'], [ [ <Recursion on list with id=35583904>, <Recursion on list with id=35583904>, 'class'], [ <Recursion on list with id=35583904>, <Recursion on list with id=35583904>, 'class'], None], [ [ <Recursion on list with id=35584912>, <Recursion on list with id=35584912>, None], [ <Recursion on list with id=35584912>, <Recursion on list with id=35584912>, None], 'class'], [ [ <Recursion on list with id=36340368>, <Recursion on list with id=36340368>, 'class'], [ <Recursion on list with id=36340368>, <Recursion on list with id=36340368>, 'class'], None], [ [ <Recursion on list with id=36340872>, <Recursion on list with id=36340872>, None], [ <Recursion on list with id=36340872>, <Recursion on list with id=36340872>, None], 'class'], [ [ <Recursion on list with id=36364872>, <Recursion on list with id=36364872>, 'class'], [ <Recursion on list with id=36364872>, <Recursion on list with id=36364872>, 'class'], None], [ [ <Recursion on list with id=36365520>, <Recursion on list with id=36365520>, None], [ <Recursion on list with id=36365520>, <Recursion on list with id=36365520>, None], 'class'], [ [ <Recursion on list with id=36457216>, <Recursion on list with id=36457216>, 'class'], [ <Recursion on list with id=36457216>, <Recursion on list with id=36457216>, 'class'], None], [ [ <Recursion on list with id=36458008>, <Recursion on list with id=36458008>, None], [ <Recursion on list with id=36458008>, <Recursion on list with id=36458008>, None], 'class'], [ [ <Recursion on list with id=36460520>, <Recursion on list with id=36460520>, 'class'], [ <Recursion on list with id=36460520>, <Recursion on list with id=36460520>, 'class'], None], [ [ <Recursion on list with id=36461168>, <Recursion on list with id=36461168>, None], [ <Recursion on list with id=36461168>, <Recursion on list with id=36461168>, None], 'class'], [ [ <Recursion on list with id=36562344>, <Recursion on list with id=36562344>, 'class'], [ <Recursion on list with id=36562344>, <Recursion on list with id=36562344>, 'class'], None], [ [ <Recursion on list with id=36563280>, <Recursion on list with id=36563280>, None], [ <Recursion on list with id=36563280>, <Recursion on list with id=36563280>, None], 'class'], <Element 'memory' at 0x2364710>, { '_children': [], 'attrib': { 'unit': 'KiB'}, 'tag': 'memory', 'tail': '\n ', 'text': '4095476'}, [], <Element 'pages' at 0x2364750>, { '_children': [], 'attrib': { 'size': '4', 'unit': 'KiB'}, 'tag': 'pages', 'tail': '\n ', 'text': '1023869'}, [], <Element 'pages' at 0x2364790>, { '_children': [], 'attrib': { 'size': '2048', 'unit': 'KiB'}, 'tag': 'pages', 'tail': '\n ', 'text': '0'}, [], <Element 'distances' at 0x23647d0>, { '_children': [<Element 'sibling' at 0x2364810>], 'attrib': { }, 'tag': 'distances', 'tail': '\n ', 'text': '\n '}, [<Element 'sibling' at 0x2364810>], <Element 'sibling' at 0x2364810>, { '_children': [], 'attrib': { 'id': '0', 'value': '10'}, 'tag': 'sibling', 'tail': '\n '}, [], <Element 'cpus' at 0x2364850>, { '_children': [ <Element 'cpu' at 0x2364890>, <Element 'cpu' at 0x23648d0>, <Element 'cpu' at 0x2364910>, <Element 'cpu' at 0x2364950>, <Element 'cpu' at 0x2364990>, <Element 'cpu' at 0x23649d0>, <Element 'cpu' at 0x2364a10>, <Element 'cpu' at 0x2364a50>], 'attrib': { 'num': '8'}, 'tag': 'cpus', 'tail': '\n ', 'text': '\n '}, [ <Element 'cpu' at 0x2364890>, <Element 'cpu' at 0x23648d0>, <Element 'cpu' at 0x2364910>, <Element 'cpu' at 0x2364950>, <Element 'cpu' at 0x2364990>, <Element 'cpu' at 0x23649d0>, <Element 'cpu' at 0x2364a10>, <Element 'cpu' at 0x2364a50>], <Element 'cpu' at 0x2364890>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '0', 'siblings': '0', 'socket_id': '0'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x23648d0>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '1', 'siblings': '1', 'socket_id': '1'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364910>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '2', 'siblings': '2', 'socket_id': '2'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364950>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '3', 'siblings': '3', 'socket_id': '3'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364990>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '4', 'siblings': '4', 'socket_id': '4'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x23649d0>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '5', 'siblings': '5', 'socket_id': '5'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364a10>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '6', 'siblings': '6', 'socket_id': '6'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364a50>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '7', 'siblings': '7', 'socket_id': '7'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'secmodel' at 0x2364a90>, { '_children': [ <Element 'model' at 0x2364ad0>, <Element 'doi' at 0x2364b10>, <Element 'baselabel' at 0x2364b50>, <Element 'baselabel' at 0x2364b90>], 'attrib': { }, 'tag': 'secmodel', 'tail': '\n ', 'text': '\n '}, [ <Element 'model' at 0x2364ad0>, <Element 'doi' at 0x2364b10>, <Element 'baselabel' at 0x2364b50>, <Element 'baselabel' at 0x2364b90>], <Element 'model' at 0x2364ad0>, { '_children': [], 'attrib': { }, 'tag': 'model', 'tail': '\n ', 'text': 'selinux'}, [], <Element 'doi' at 0x2364b10>, { '_children': [], 'attrib': { }, 'tag': 'doi', 'tail': '\n ', 'text': '0'}, [], <Element 'baselabel' at 0x2364b50>, { '_children': [], 'attrib': { 'type': 'kvm'}, 'tag': 'baselabel', 'tail': '\n ', 'text': 'system_u:system_r:svirt_t:s0'}, [], <Element 'baselabel' at 0x2364b90>, { '_children': [], 'attrib': { 'type': 'qemu'}, 'tag': 'baselabel', 'tail': '\n ', 'text': 'system_u:system_r:svirt_tcg_t:s0'}, [], <Element 'secmodel' at 0x2364bd0>, { '_children': [ <Element 'model' at 0x2364c10>, <Element 'doi' at 0x2364c50>, <Element 'baselabel' at 0x2364c90>, <Element 'baselabel' at 0x2364cd0>], 'attrib': { }, 'tag': 'secmodel', 'tail': '\n ', 'text': '\n '}, [ <Element 'model' at 0x2364c10>, <Element 'doi' at 0x2364c50>, <Element 'baselabel' at 0x2364c90>, <Element 'baselabel' at 0x2364cd0>], <Element 'model' at 0x2364c10>, { '_children': [], 'attrib': { }, 'tag': 'model', 'tail': '\n ', 'text': 'dac'}, [], <Element 'doi' at 0x2364c50>, { '_children': [], 'attrib': { }, 'tag': 'doi', 'tail': '\n ', 'text': '0'}, [], <Element 'baselabel' at 0x2364c90>, { '_children': [], 'attrib': { 'type': 'kvm'}, 'tag': 'baselabel', 'tail': '\n ', 'text': '+107:+107'}, [], <Element 'baselabel' at 0x2364cd0>, { '_children': [], 'attrib': { 'type': 'qemu'}, 'tag': 'baselabel', 'tail': '\n ', 'text': '+107:+107'}, [], <Element 'guest' at 0x2364d10>, { '_children': [ <Element 'os_type' at 0x2364d50>, <Element 'arch' at 0x2364d90>, <Element 'features' at 0x2369390>], 'attrib': { }, 'tag': 'guest', 'tail': '\n\n ', 'text': '\n '}, [ <Element 'os_type' at 0x2364d50>, <Element 'arch' at 0x2364d90>, <Element 'features' at 0x2369390>], <Element 'os_type' at 0x2364d50>, { '_children': [], 'attrib': { }, 'tag': 'os_type', 'tail': '\n ', 'text': 'hvm'}, [], <Element 'arch' at 0x2364d90>, { '_children': [ <Element 'wordsize' at 0x2364dd0>, <Element 'emulator' at 0x2364e10>, <Element 'machine' at 0x2364e90>, <Element 'machine' at 0x2364ed0>, <Element 'machine' at 0x2364f10>, <Element 'machine' at 0x2364f50>, <Element 'machine' at 0x2364f90>, <Element 'machine' at 0x2364fd0>, <Element 'machine' at 0x2369050>, <Element 'machine' at 0x2369090>, <Element 'machine' at 0x23690d0>, <Element 'machine' at 0x2369110>, <Element 'machine' at 0x2369150>, <Element 'machine' at 0x2369190>, <Element 'machine' at 0x23691d0>, <Element 'machine' at 0x2369210>, <Element 'machine' at 0x2369250>, <Element 'domain' at 0x2369290>, <Element 'domain' at 0x23692d0>], 'attrib': { 'name': 'i686'}, 'tag': 'arch', 'tail': '\n ', 'text': '\n '}, [ <Element 'wordsize' at 0x2364dd0>, <Element 'emulator' at 0x2364e10>, <Element 'machine' at 0x2364e90>, <Element 'machine' at 0x2364ed0>, <Element 'machine' at 0x2364f10>, <Element 'machine' at 0x2364f50>, <Element 'machine' at 0x2364f90>, <Element 'machine' at 0x2364fd0>, <Element 'machine' at 0x2369050>, <Element 'machine' at 0x2369090>, <Element 'machine' at 0x23690d0>, <Element 'machine' at 0x2369110>, <Element 'machine' at 0x2369150>, <Element 'machine' at 0x2369190>, <Element 'machine' at 0x23691d0>, <Element 'machine' at 0x2369210>, <Element 'machine' at 0x2369250>, <Element 'domain' at 0x2369290>, <Element 'domain' at 0x23692d0>], <Element 'wordsize' at 0x2364dd0>, { '_children': [], 'attrib': { }, 'tag': 'wordsize', 'tail': '\n ', 'text': '32'}, [], <Element 'emulator' at 0x2364e10>, { '_children': [], 'attrib': { }, 'tag': 'emulator', 'tail': '\n ', 'text': '/usr/libexec/qemu-kvm'}, [], <Element 'machine' at 0x2364e90>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.2.0'}, [], <Element 'machine' at 0x2364ed0>, { '_children': [], 'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc'}, [], <Element 'machine' at 0x2364f10>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.0.0'}, [], <Element 'machine' at 0x2364f50>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.1.0'}, [], <Element 'machine' at 0x2364f90>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.3.0'}, [], <Element 'machine' at 0x2364fd0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.2.0'}, [], <Element 'machine' at 0x2369050>, { '_children': [], 'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'q35'}, [], <Element 'machine' at 0x2369090>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.4.0'}, [], <Element 'machine' at 0x23690d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.0.0'}, [], <Element 'machine' at 0x2369110>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.1.0'}, [], <Element 'machine' at 0x2369150>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.5.0'}, [], <Element 'machine' at 0x2369190>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.6.0'}, [], <Element 'machine' at 0x23691d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.1.0'}, [], <Element 'machine' at 0x2369210>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.0.0'}, [], <Element 'machine' at 0x2369250>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.2.0'}, [], <Element 'domain' at 0x2369290>, { '_children': [], 'attrib': { 'type': 'qemu'}, 'tag': 'domain', 'tail': '\n '}, [], <Element 'domain' at 0x23692d0>, { '_children': [<Element 'emulator' at 0x2369310>], 'attrib': { 'type': 'kvm'}, 'tag': 'domain', 'tail': '\n ', 'text': '\n '}, [<Element 'emulator' at 0x2369310>], <Element 'emulator' at 0x2369310>, { '_children': [], 'attrib': { }, 'tag': 'emulator', 'tail': '\n ', 'text': '/usr/libexec/qemu-kvm'}, [], <Element 'features' at 0x2369390>, { '_children': [ <Element 'cpuselection' at 0x23693d0>, <Element 'deviceboot' at 0x2369410>, <Element 'disksnapshot' at 0x2369450>, <Element 'acpi' at 0x2369490>, <Element 'apic' at 0x23694d0>, <Element 'pae' at 0x2369510>, <Element 'nonpae' at 0x2369550>], 'attrib': { }, 'tag': 'features', 'tail': '\n ', 'text': '\n '}, [ <Element 'cpuselection' at 0x23693d0>, <Element 'deviceboot' at 0x2369410>, <Element 'disksnapshot' at 0x2369450>, <Element 'acpi' at 0x2369490>, <Element 'apic' at 0x23694d0>, <Element 'pae' at 0x2369510>, <Element 'nonpae' at 0x2369550>], <Element 'cpuselection' at 0x23693d0>, { '_children': [], 'attrib': { }, 'tag': 'cpuselection', 'tail': '\n '}, [], <Element 'deviceboot' at 0x2369410>, { '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail': '\n '}, [], <Element 'disksnapshot' at 0x2369450>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'no'}, 'tag': 'disksnapshot', 'tail': '\n '}, [], <Element 'acpi' at 0x2369490>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'yes'}, 'tag': 'acpi', 'tail': '\n '}, [], <Element 'apic' at 0x23694d0>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'no'}, 'tag': 'apic', 'tail': '\n '}, [], <Element 'pae' at 0x2369510>, { '_children': [], 'attrib': { }, 'tag': 'pae', 'tail': '\n '}, [], <Element 'nonpae' at 0x2369550>, { '_children': [], 'attrib': { }, 'tag': 'nonpae', 'tail': '\n '}, [], <Element 'guest' at 0x2369590>, { '_children': [ <Element 'os_type' at 0x23695d0>, <Element 'arch' at 0x2369610>, <Element 'features' at 0x2369bd0>], 'attrib': { }, 'tag': 'guest', 'tail': '\n\n', 'text': '\n '}, [ <Element 'os_type' at 0x23695d0>, <Element 'arch' at 0x2369610>, <Element 'features' at 0x2369bd0>], <Element 'os_type' at 0x23695d0>, { '_children': [], 'attrib': { }, 'tag': 'os_type', 'tail': '\n ', 'text': 'hvm'}, [], <Element 'arch' at 0x2369610>, { '_children': [ <Element 'wordsize' at 0x2369650>, <Element 'emulator' at 0x2369690>, <Element 'machine' at 0x2369710>, <Element 'machine' at 0x2369750>, <Element 'machine' at 0x2369790>, <Element 'machine' at 0x23697d0>, <Element 'machine' at 0x2369810>, <Element 'machine' at 0x2369850>, <Element 'machine' at 0x2369890>, <Element 'machine' at 0x23698d0>, <Element 'machine' at 0x2369910>, <Element 'machine' at 0x2369950>, <Element 'machine' at 0x2369990>, <Element 'machine' at 0x23699d0>, <Element 'machine' at 0x2369a10>, <Element 'machine' at 0x2369a50>, <Element 'machine' at 0x2369a90>, <Element 'domain' at 0x2369ad0>, <Element 'domain' at 0x2369b10>], 'attrib': { 'name': 'x86_64'}, 'tag': 'arch', 'tail': '\n ', 'text': '\n '}, [ <Element 'wordsize' at 0x2369650>, <Element 'emulator' at 0x2369690>, <Element 'machine' at 0x2369710>, <Element 'machine' at 0x2369750>, <Element 'machine' at 0x2369790>, <Element 'machine' at 0x23697d0>, <Element 'machine' at 0x2369810>, <Element 'machine' at 0x2369850>, <Element 'machine' at 0x2369890>, <Element 'machine' at 0x23698d0>, <Element 'machine' at 0x2369910>, <Element 'machine' at 0x2369950>, <Element 'machine' at 0x2369990>, <Element 'machine' at 0x23699d0>, <Element 'machine' at 0x2369a10>, <Element 'machine' at 0x2369a50>, <Element 'machine' at 0x2369a90>, <Element 'domain' at 0x2369ad0>, <Element 'domain' at 0x2369b10>], <Element 'wordsize' at 0x2369650>, { '_children': [], 'attrib': { }, 'tag': 'wordsize', 'tail': '\n ', 'text': '64'}, [], <Element 'emulator' at 0x2369690>, { '_children': [], 'attrib': { }, 'tag': 'emulator', 'tail': '\n ', 'text': '/usr/libexec/qemu-kvm'}, [], <Element 'machine' at 0x2369710>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.2.0'}, [], <Element 'machine' at 0x2369750>, { '_children': [], 'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc'}, [], <Element 'machine' at 0x2369790>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.0.0'}, [], <Element 'machine' at 0x23697d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.1.0'}, [], <Element 'machine' at 0x2369810>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.3.0'}, [], <Element 'machine' at 0x2369850>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.2.0'}, [], <Element 'machine' at 0x2369890>, { '_children': [], 'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'q35'}, [], <Element 'machine' at 0x23698d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.4.0'}, [], <Element 'machine' at 0x2369910>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.0.0'}, [], <Element 'machine' at 0x2369950>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.1.0'}, [], <Element 'machine' at 0x2369990>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.5.0'}, [], <Element 'machine' at 0x23699d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.6.0'}, [], <Element 'machine' at 0x2369a10>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.1.0'}, [], <Element 'machine' at 0x2369a50>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.0.0'}, [], <Element 'machine' at 0x2369a90>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.2.0'}, [], <Element 'domain' at 0x2369ad0>, { '_children': [], 'attrib': { 'type': 'qemu'}, 'tag': 'domain', 'tail': '\n '}, [], <Element 'domain' at 0x2369b10>, { '_children': [<Element 'emulator' at 0x2369b50>], 'attrib': { 'type': 'kvm'}, 'tag': 'domain', 'tail': '\n ', 'text': '\n '}, [<Element 'emulator' at 0x2369b50>], <Element 'emulator' at 0x2369b50>, { '_children': [], 'attrib': { }, 'tag': 'emulator', 'tail': '\n ', 'text': '/usr/libexec/qemu-kvm'}, [], <Element 'features' at 0x2369bd0>, { '_children': [ <Element 'cpuselection' at 0x2369c10>, <Element 'deviceboot' at 0x2369c50>, <Element 'disksnapshot' at 0x2369c90>, <Element 'acpi' at 0x2369cd0>, <Element 'apic' at 0x2369d10>], 'attrib': { }, 'tag': 'features', 'tail': '\n ', 'text': '\n '}, [ <Element 'cpuselection' at 0x2369c10>, <Element 'deviceboot' at 0x2369c50>, <Element 'disksnapshot' at 0x2369c90>, <Element 'acpi' at 0x2369cd0>, <Element 'apic' at 0x2369d10>], <Element 'cpuselection' at 0x2369c10>, { '_children': [], 'attrib': { }, 'tag': 'cpuselection', 'tail': '\n '}, [], <Element 'deviceboot' at 0x2369c50>, { '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail': '\n '}, [], <Element 'disksnapshot' at 0x2369c90>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'no'}, 'tag': 'disksnapshot', 'tail': '\n '}, [], <Element 'acpi' at 0x2369cd0>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'yes'}, 'tag': 'acpi', 'tail': '\n '}, [], <Element 'apic' at 0x2369d10>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'no'}, 'tag': 'apic', 'tail': '\n '}, [], [], (<type '_ctypes.Array'>,), { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>, '__doc__': None, '__module__': 'vdsm.netlink', '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>, '_length_': 60, '_type_': <class 'ctypes.c_char'>, 'raw': <attribute 'raw' of 'c_char_Array_60' objects>, 'value': <attribute 'value' of 'c_char_Array_60' objects>}, <class 'vdsm.netlink.c_char_Array_60'>, <attribute '__dict__' of 'c_char_Array_60' objects>, <attribute '__weakref__' of 'c_char_Array_60' objects>, ( <class 'vdsm.netlink.c_char_Array_60'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_60' objects>, <attribute 'value' of 'c_char_Array_60' objects>, <xml.etree.ElementTree.XMLParser object at 0x2357f50>, { '_doctype': None, '_error': <class 'xml.parsers.expat.ExpatError'>, '_names': { u'acpi': 'acpi', u'apic': 'apic', u'arch': 'arch', u'baselabel': 'baselabel', u'canonical': 'canonical', u'capabilities': 'capabilities', u'cell': 'cell', u'cells': 'cells', u'core_id': 'core_id', u'cores': 'cores', u'cpu': 'cpu', u'cpus': 'cpus', u'cpuselection': 'cpuselection', u'default': 'default', u'deviceboot': 'deviceboot', u'disksnapshot': 'disksnapshot', u'distances': 'distances', u'doi': 'doi', u'domain': 'domain', u'emulator': 'emulator', u'feature': 'feature', u'features': 'features', u'guest': 'guest', u'host': 'host', u'id': 'id', u'live': 'live', u'machine': 'machine', u'maxCpus': 'maxCpus', u'memory': 'memory', u'migration_features': 'migration_features', u'model': 'model', u'name': 'name', u'nonpae': 'nonpae', u'num': 'num', u'os_type': 'os_type', u'pae': 'pae', u'pages': 'pages', u'power_management': 'power_management', u'secmodel': 'secmodel', u'sibling': 'sibling', u'siblings': 'siblings', u'size': 'size', u'socket_id': 'socket_id', u'sockets': 'sockets', u'suspend_disk': 'suspend_disk', u'suspend_hybrid': 'suspend_hybrid', u'suspend_mem': 'suspend_mem', u'threads': 'threads', u'toggle': 'toggle', u'topology': 'topology', u'type': 'type', u'unit': 'unit', u'uri_transport': 'uri_transport', u'uri_transports': 'uri_transports', u'uuid': 'uuid', u'value': 'value', u'vendor': 'vendor', u'wordsize': 'wordsize'}, '_target': <xml.etree.ElementTree.TreeBuilder object at 0x2357f10>, 'entity': { }, 'parser': <pyexpat.xmlparser object at 0x22e5360>, 'version': 'Expat 2.1.0'}, <pyexpat.xmlparser object at 0x22e5360>, <xml.etree.ElementTree.TreeBuilder object at 0x2357f10>, <bound method XMLParser._start_list of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._end of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._pi of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._data of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._comment of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._default of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, { '_data': [], '_elem': [], '_factory': <class 'xml.etree.ElementTree.Element'>, '_last': <Element 'capabilities' at 0x2357f90>, '_tail': 1}, [], <Element 'cell' at 0x23646d0>, <Element 'cells' at 0x2364690>, <Element 'topology' at 0x2364650>, <Element 'host' at 0x2357fd0>, <Element 'capabilities' at 0x2357f90>, { '_children': [ <Element 'memory' at 0x2364710>, <Element 'pages' at 0x2364750>, <Element 'pages' at 0x2364790>, <Element 'distances' at 0x23647d0>, <Element 'cpus' at 0x2364850>], 'attrib': { 'id': '0'}, 'tag': 'cell', 'tail': '\n ', 'text': '\n '}, { '_children': [<Element 'cell' at 0x23646d0>], 'attrib': { 'num': '1'}, 'tag': 'cells', 'tail': '\n ', 'text': '\n '}, { '_children': [<Element 'cells' at 0x2364690>], 'attrib': { }, 'tag': 'topology', 'tail': '\n ', 'text': '\n '}, { '_children': [ <Element 'uuid' at 0x2364050>, <Element 'cpu' at 0x2364090>, <Element 'power_management' at 0x2364390>, <Element 'migration_features' at 0x23644d0>, <Element 'topology' at 0x2364650>, <Element 'secmodel' at 0x2364a90>, <Element 'secmodel' at 0x2364bd0>], 'attrib': { }, 'tag': 'host', 'tail': '\n\n ', 'text': '\n '}, { '_children': [ <Element 'host' at 0x2357fd0>, <Element 'guest' at 0x2364d10>, <Element 'guest' at 0x2369590>], 'attrib': { }, 'tag': 'capabilities', 'text': '\n\n '}, [ <Element 'memory' at 0x2364710>, <Element 'pages' at 0x2364750>, <Element 'pages' at 0x2364790>, <Element 'distances' at 0x23647d0>, <Element 'cpus' at 0x2364850>], [<Element 'cell' at 0x23646d0>], [<Element 'cells' at 0x2364690>], [ <Element 'uuid' at 0x2364050>, <Element 'cpu' at 0x2364090>, <Element 'power_management' at 0x2364390>, <Element 'migration_features' at 0x23644d0>, <Element 'topology' at 0x2364650>, <Element 'secmodel' at 0x2364a90>, <Element 'secmodel' at 0x2364bd0>], [ <Element 'host' at 0x2357fd0>, <Element 'guest' at 0x2364d10>, <Element 'guest' at 0x2369590>], <Element 'migration_features' at 0x23644d0>, <Element 'power_management' at 0x2364390>, <Element 'cpu' at 0x2364090>, <Element 'uuid' at 0x2364050>, { '_children': [ <Element 'live' at 0x2364510>, <Element 'uri_transports' at 0x2364590>], 'attrib': { }, 'tag': 'migration_features', 'tail': '\n ', 'text': '\n '}, { '_children': [ <Element 'suspend_mem' at 0x23643d0>, <Element 'suspend_disk' at 0x2364410>, <Element 'suspend_hybrid' at 0x2364490>], 'attrib': { }, 'tag': 'power_management', 'tail': '\n ', 'text': '\n '}, { '_children': [ <Element 'arch' at 0x23640d0>, <Element 'model' at 0x2364110>, <Element 'vendor' at 0x2364150>, <Element 'topology' at 0x2364190>, <Element 'feature' at 0x23641d0>, <Element 'feature' at 0x2364210>, <Element 'feature' at 0x2364250>, <Element 'feature' at 0x2364290>, <Element 'feature' at 0x23642d0>, <Element 'pages' at 0x2364310>, <Element 'pages' at 0x2364350>], 'attrib': { }, 'tag': 'cpu', 'tail': '\n ', 'text': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'uuid', 'tail': '\n ', 'text': 'a920d963-c18f-4fda-aa5a-deee9bbcde9d'}, [<Element 'live' at 0x2364510>, <Element 'uri_transports' at 0x2364590>], [ <Element 'suspend_mem' at 0x23643d0>, <Element 'suspend_disk' at 0x2364410>, <Element 'suspend_hybrid' at 0x2364490>], [ <Element 'arch' at 0x23640d0>, <Element 'model' at 0x2364110>, <Element 'vendor' at 0x2364150>, <Element 'topology' at 0x2364190>, <Element 'feature' at 0x23641d0>, <Element 'feature' at 0x2364210>, <Element 'feature' at 0x2364250>, <Element 'feature' at 0x2364290>, <Element 'feature' at 0x23642d0>, <Element 'pages' at 0x2364310>, <Element 'pages' at 0x2364350>], [], <Element 'uri_transports' at 0x2364590>, <Element 'live' at 0x2364510>, <Element 'suspend_hybrid' at 0x2364490>, <Element 'suspend_disk' at 0x2364410>, <Element 'suspend_mem' at 0x23643d0>, <Element 'pages' at 0x2364350>, <Element 'pages' at 0x2364310>, <Element 'feature' at 0x23642d0>, <Element 'feature' at 0x2364290>, <Element 'feature' at 0x2364250>, <Element 'feature' at 0x2364210>, <Element 'feature' at 0x23641d0>, <Element 'topology' at 0x2364190>, <Element 'vendor' at 0x2364150>, <Element 'model' at 0x2364110>, <Element 'arch' at 0x23640d0>, { '_children': [ <Element 'uri_transport' at 0x23645d0>, <Element 'uri_transport' at 0x2364610>], 'attrib': { }, 'tag': 'uri_transports', 'tail': '\n ', 'text': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'live', 'tail': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'suspend_hybrid', 'tail': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'suspend_disk', 'tail': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'suspend_mem', 'tail': '\n '}, { '_children': [], 'attrib': { 'size': '2048', 'unit': 'KiB'}, 'tag': 'pages', 'tail': '\n '}, { '_children': [], 'attrib': { 'size': '4', 'unit': 'KiB'}, 'tag': 'pages', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'ss'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'vmx'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'pcid'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'osxsave'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'hypervisor'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'cores': '1', 'sockets': '8', 'threads': '1'}, 'tag': 'topology', 'tail': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'vendor', 'tail': '\n ', 'text': 'Intel'}, { '_children': [], 'attrib': { }, 'tag': 'model', 'tail': '\n ', 'text': 'IvyBridge'}, { '_children': [], 'attrib': { }, 'tag': 'arch', 'tail': '\n ', 'text': 'x86_64'}, [ <Element 'uri_transport' at 0x23645d0>, <Element 'uri_transport' at 0x2364610>], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], <Element 'uri_transport' at 0x2364610>, <Element 'uri_transport' at 0x23645d0>, { '_children': [], 'attrib': { }, 'tag': 'uri_transport', 'tail': '\n ', 'text': 'rdma'}, { '_children': [], 'attrib': { }, 'tag': 'uri_transport', 'tail': '\n ', 'text': 'tcp'}, [], [], (<type '_ctypes.Array'>,), { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>, '__doc__': None, '__module__': 'vdsm.netlink', '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>, '_length_': 60, '_type_': <class 'ctypes.c_char'>, 'raw': <attribute 'raw' of 'c_char_Array_60' objects>, 'value': <attribute 'value' of 'c_char_Array_60' objects>}, <class 'vdsm.netlink.c_char_Array_60'>, <attribute '__dict__' of 'c_char_Array_60' objects>, <attribute '__weakref__' of 'c_char_Array_60' objects>, ( <class 'vdsm.netlink.c_char_Array_60'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_60' objects>, <attribute 'value' of 'c_char_Array_60' objects>]