Re: [ovirt-devel] [VDSM] health: Introduce Vdsm health monitoring

Hi all, Continuing the leak investigation Francesco and Milan are working on, I posted this patch, adding leak health monitoring to vdsm [1]. This patch currently monitor collectible objects, and we may add other interesting stuff later if needed. To enable the monitor, you must set: [devel] health_monitor_enable = true In vdsm.conf, and restart vdsm. Here is an example logs - this was logged 60 seconds after starting vdsm, when running as spm: Thread-13::DEBUG::2016-01-14 18:53:43,239::health::84::health::(_check) Checking health Thread-13::DEBUG::2016-01-14 18:53:43,272::health::86::health::(_check) Collected 460 objects Thread-13::WARNING::2016-01-14 18:53:43,277::health::100::health::(_check) Found 5873 uncollectible objects, 10 suspects implementing __del__: ['<pthreading.Lock object at 0x7f012c115320>', '<pthreading.Lock object at 0x7f012c115440>', '<pthreading.Lock object at 0x7f012c1153b0>', '<pthreading.Lock object at 0x7f012c1154d0>', '<pthread.Cond object at 0x7f012c111350>', '<pthread.Cond object at 0x7f012c1112d0>', '<pthread.Cond object at 0x7f012c111250>', '<pthread.Cond object at 0x7f012c111850>', '<pthread.Cond object at 0x7f012c1117d0>', '<pthread.Cond object at 0x7f012c111750>'] We found 460 uncollectible objects, 10 of them are objects implementing __del__. Having such object in a reference cycle make the entire cycle uncollectible. The pthreading objects in this log grow as vdsm continue to run, even without any interaction with engine. The pthreading objects leak is caused by a cycle in storage.misc.RWLock, fixed in patch [2]. After solving the pthreading leak, we still have many uncollectible objects in the logs, listing them, it seems that the suspect is hiding inside a collection (dict, list, tuple) and does not appear in the gc.garbage list. I will address this in later patch, searching in the entire object graph instead of the top level of gc.garbage. See end of this mail for list of these objects. I printed them like this: Enable manhole debugging shell: [devel] manhole_enable = true Connect to vdsm using manhole: nc -U /run/vdsm/vdsmd.manhole Print garbage: >>> import pprint >>> import gc >>> with open('/tmp/garbage.txt', 'w') as f: pprint.pprint(gc.garbage, stream=f, indent=4) These objects looks related to libvirt - Francesco, can you take a look? [1] https://gerrit.ovirt.org/51708 [2] https://gerrit.ovirt.org/51868 Nir --- garbage after patch [2] --- [ (<type '_ctypes.Array'>,), { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>, '__doc__': None, '__module__': 'vdsm.netlink', '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>, '_length_': 60, '_type_': <class 'ctypes.c_char'>, 'raw': <attribute 'raw' of 'c_char_Array_60' objects>, 'value': <attribute 'value' of 'c_char_Array_60' objects>}, <class 'vdsm.netlink.c_char_Array_60'>, <attribute '__dict__' of 'c_char_Array_60' objects>, <attribute '__weakref__' of 'c_char_Array_60' objects>, ( <class 'vdsm.netlink.c_char_Array_60'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_60' objects>, <attribute 'value' of 'c_char_Array_60' objects>, <cell at 0x23662f0: bool object at 0x7f1e086c8c40>, <cell at 0x2366328: NoneType object at 0x7f1e086dcf00>, <cell at 0x2366360: function object at 0x233f488>, ( <cell at 0x23662f0: bool object at 0x7f1e086c8c40>, <cell at 0x2366328: NoneType object at 0x7f1e086dcf00>, <cell at 0x2366360: function object at 0x233f488>), <function wrapMethod at 0x233f488>, [ [ <Recursion on list with id=34033464>, <Recursion on list with id=34033464>, 'class'], [ <Recursion on list with id=34033464>, <Recursion on list with id=34033464>, 'class'], None], [ [ <Recursion on list with id=33979064>, <Recursion on list with id=33979064>, None], [ <Recursion on list with id=33979064>, <Recursion on list with id=33979064>, None], 'class'], [ [ <Recursion on list with id=35535256>, <Recursion on list with id=35535256>, 'class'], [ <Recursion on list with id=35535256>, <Recursion on list with id=35535256>, 'class'], None], [ [ <Recursion on list with id=35535832>, <Recursion on list with id=35535832>, None], [ <Recursion on list with id=35535832>, <Recursion on list with id=35535832>, None], 'class'], [ [ <Recursion on list with id=35556312>, <Recursion on list with id=35556312>, 'class'], [ <Recursion on list with id=35556312>, <Recursion on list with id=35556312>, 'class'], None], [ [ <Recursion on list with id=35556600>, <Recursion on list with id=35556600>, None], [ <Recursion on list with id=35556600>, <Recursion on list with id=35556600>, None], 'class'], [ [ <Recursion on list with id=35556960>, <Recursion on list with id=35556960>, 'class'], [ <Recursion on list with id=35556960>, <Recursion on list with id=35556960>, 'class'], None], [ [ <Recursion on list with id=35561912>, <Recursion on list with id=35561912>, None], [ <Recursion on list with id=35561912>, <Recursion on list with id=35561912>, None], 'class'], [ [ <Recursion on list with id=35583904>, <Recursion on list with id=35583904>, 'class'], [ <Recursion on list with id=35583904>, <Recursion on list with id=35583904>, 'class'], None], [ [ <Recursion on list with id=35584912>, <Recursion on list with id=35584912>, None], [ <Recursion on list with id=35584912>, <Recursion on list with id=35584912>, None], 'class'], [ [ <Recursion on list with id=36340368>, <Recursion on list with id=36340368>, 'class'], [ <Recursion on list with id=36340368>, <Recursion on list with id=36340368>, 'class'], None], [ [ <Recursion on list with id=36340872>, <Recursion on list with id=36340872>, None], [ <Recursion on list with id=36340872>, <Recursion on list with id=36340872>, None], 'class'], [ [ <Recursion on list with id=36364872>, <Recursion on list with id=36364872>, 'class'], [ <Recursion on list with id=36364872>, <Recursion on list with id=36364872>, 'class'], None], [ [ <Recursion on list with id=36365520>, <Recursion on list with id=36365520>, None], [ <Recursion on list with id=36365520>, <Recursion on list with id=36365520>, None], 'class'], [ [ <Recursion on list with id=36457216>, <Recursion on list with id=36457216>, 'class'], [ <Recursion on list with id=36457216>, <Recursion on list with id=36457216>, 'class'], None], [ [ <Recursion on list with id=36458008>, <Recursion on list with id=36458008>, None], [ <Recursion on list with id=36458008>, <Recursion on list with id=36458008>, None], 'class'], [ [ <Recursion on list with id=36460520>, <Recursion on list with id=36460520>, 'class'], [ <Recursion on list with id=36460520>, <Recursion on list with id=36460520>, 'class'], None], [ [ <Recursion on list with id=36461168>, <Recursion on list with id=36461168>, None], [ <Recursion on list with id=36461168>, <Recursion on list with id=36461168>, None], 'class'], [ [ <Recursion on list with id=36562344>, <Recursion on list with id=36562344>, 'class'], [ <Recursion on list with id=36562344>, <Recursion on list with id=36562344>, 'class'], None], [ [ <Recursion on list with id=36563280>, <Recursion on list with id=36563280>, None], [ <Recursion on list with id=36563280>, <Recursion on list with id=36563280>, None], 'class'], <Element 'memory' at 0x2364710>, { '_children': [], 'attrib': { 'unit': 'KiB'}, 'tag': 'memory', 'tail': '\n ', 'text': '4095476'}, [], <Element 'pages' at 0x2364750>, { '_children': [], 'attrib': { 'size': '4', 'unit': 'KiB'}, 'tag': 'pages', 'tail': '\n ', 'text': '1023869'}, [], <Element 'pages' at 0x2364790>, { '_children': [], 'attrib': { 'size': '2048', 'unit': 'KiB'}, 'tag': 'pages', 'tail': '\n ', 'text': '0'}, [], <Element 'distances' at 0x23647d0>, { '_children': [<Element 'sibling' at 0x2364810>], 'attrib': { }, 'tag': 'distances', 'tail': '\n ', 'text': '\n '}, [<Element 'sibling' at 0x2364810>], <Element 'sibling' at 0x2364810>, { '_children': [], 'attrib': { 'id': '0', 'value': '10'}, 'tag': 'sibling', 'tail': '\n '}, [], <Element 'cpus' at 0x2364850>, { '_children': [ <Element 'cpu' at 0x2364890>, <Element 'cpu' at 0x23648d0>, <Element 'cpu' at 0x2364910>, <Element 'cpu' at 0x2364950>, <Element 'cpu' at 0x2364990>, <Element 'cpu' at 0x23649d0>, <Element 'cpu' at 0x2364a10>, <Element 'cpu' at 0x2364a50>], 'attrib': { 'num': '8'}, 'tag': 'cpus', 'tail': '\n ', 'text': '\n '}, [ <Element 'cpu' at 0x2364890>, <Element 'cpu' at 0x23648d0>, <Element 'cpu' at 0x2364910>, <Element 'cpu' at 0x2364950>, <Element 'cpu' at 0x2364990>, <Element 'cpu' at 0x23649d0>, <Element 'cpu' at 0x2364a10>, <Element 'cpu' at 0x2364a50>], <Element 'cpu' at 0x2364890>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '0', 'siblings': '0', 'socket_id': '0'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x23648d0>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '1', 'siblings': '1', 'socket_id': '1'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364910>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '2', 'siblings': '2', 'socket_id': '2'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364950>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '3', 'siblings': '3', 'socket_id': '3'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364990>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '4', 'siblings': '4', 'socket_id': '4'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x23649d0>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '5', 'siblings': '5', 'socket_id': '5'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364a10>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '6', 'siblings': '6', 'socket_id': '6'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364a50>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '7', 'siblings': '7', 'socket_id': '7'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'secmodel' at 0x2364a90>, { '_children': [ <Element 'model' at 0x2364ad0>, <Element 'doi' at 0x2364b10>, <Element 'baselabel' at 0x2364b50>, <Element 'baselabel' at 0x2364b90>], 'attrib': { }, 'tag': 'secmodel', 'tail': '\n ', 'text': '\n '}, [ <Element 'model' at 0x2364ad0>, <Element 'doi' at 0x2364b10>, <Element 'baselabel' at 0x2364b50>, <Element 'baselabel' at 0x2364b90>], <Element 'model' at 0x2364ad0>, { '_children': [], 'attrib': { }, 'tag': 'model', 'tail': '\n ', 'text': 'selinux'}, [], <Element 'doi' at 0x2364b10>, { '_children': [], 'attrib': { }, 'tag': 'doi', 'tail': '\n ', 'text': '0'}, [], <Element 'baselabel' at 0x2364b50>, { '_children': [], 'attrib': { 'type': 'kvm'}, 'tag': 'baselabel', 'tail': '\n ', 'text': 'system_u:system_r:svirt_t:s0'}, [], <Element 'baselabel' at 0x2364b90>, { '_children': [], 'attrib': { 'type': 'qemu'}, 'tag': 'baselabel', 'tail': '\n ', 'text': 'system_u:system_r:svirt_tcg_t:s0'}, [], <Element 'secmodel' at 0x2364bd0>, { '_children': [ <Element 'model' at 0x2364c10>, <Element 'doi' at 0x2364c50>, <Element 'baselabel' at 0x2364c90>, <Element 'baselabel' at 0x2364cd0>], 'attrib': { }, 'tag': 'secmodel', 'tail': '\n ', 'text': '\n '}, [ <Element 'model' at 0x2364c10>, <Element 'doi' at 0x2364c50>, <Element 'baselabel' at 0x2364c90>, <Element 'baselabel' at 0x2364cd0>], <Element 'model' at 0x2364c10>, { '_children': [], 'attrib': { }, 'tag': 'model', 'tail': '\n ', 'text': 'dac'}, [], <Element 'doi' at 0x2364c50>, { '_children': [], 'attrib': { }, 'tag': 'doi', 'tail': '\n ', 'text': '0'}, [], <Element 'baselabel' at 0x2364c90>, { '_children': [], 'attrib': { 'type': 'kvm'}, 'tag': 'baselabel', 'tail': '\n ', 'text': '+107:+107'}, [], <Element 'baselabel' at 0x2364cd0>, { '_children': [], 'attrib': { 'type': 'qemu'}, 'tag': 'baselabel', 'tail': '\n ', 'text': '+107:+107'}, [], <Element 'guest' at 0x2364d10>, { '_children': [ <Element 'os_type' at 0x2364d50>, <Element 'arch' at 0x2364d90>, <Element 'features' at 0x2369390>], 'attrib': { }, 'tag': 'guest', 'tail': '\n\n ', 'text': '\n '}, [ <Element 'os_type' at 0x2364d50>, <Element 'arch' at 0x2364d90>, <Element 'features' at 0x2369390>], <Element 'os_type' at 0x2364d50>, { '_children': [], 'attrib': { }, 'tag': 'os_type', 'tail': '\n ', 'text': 'hvm'}, [], <Element 'arch' at 0x2364d90>, { '_children': [ <Element 'wordsize' at 0x2364dd0>, <Element 'emulator' at 0x2364e10>, <Element 'machine' at 0x2364e90>, <Element 'machine' at 0x2364ed0>, <Element 'machine' at 0x2364f10>, <Element 'machine' at 0x2364f50>, <Element 'machine' at 0x2364f90>, <Element 'machine' at 0x2364fd0>, <Element 'machine' at 0x2369050>, <Element 'machine' at 0x2369090>, <Element 'machine' at 0x23690d0>, <Element 'machine' at 0x2369110>, <Element 'machine' at 0x2369150>, <Element 'machine' at 0x2369190>, <Element 'machine' at 0x23691d0>, <Element 'machine' at 0x2369210>, <Element 'machine' at 0x2369250>, <Element 'domain' at 0x2369290>, <Element 'domain' at 0x23692d0>], 'attrib': { 'name': 'i686'}, 'tag': 'arch', 'tail': '\n ', 'text': '\n '}, [ <Element 'wordsize' at 0x2364dd0>, <Element 'emulator' at 0x2364e10>, <Element 'machine' at 0x2364e90>, <Element 'machine' at 0x2364ed0>, <Element 'machine' at 0x2364f10>, <Element 'machine' at 0x2364f50>, <Element 'machine' at 0x2364f90>, <Element 'machine' at 0x2364fd0>, <Element 'machine' at 0x2369050>, <Element 'machine' at 0x2369090>, <Element 'machine' at 0x23690d0>, <Element 'machine' at 0x2369110>, <Element 'machine' at 0x2369150>, <Element 'machine' at 0x2369190>, <Element 'machine' at 0x23691d0>, <Element 'machine' at 0x2369210>, <Element 'machine' at 0x2369250>, <Element 'domain' at 0x2369290>, <Element 'domain' at 0x23692d0>], <Element 'wordsize' at 0x2364dd0>, { '_children': [], 'attrib': { }, 'tag': 'wordsize', 'tail': '\n ', 'text': '32'}, [], <Element 'emulator' at 0x2364e10>, { '_children': [], 'attrib': { }, 'tag': 'emulator', 'tail': '\n ', 'text': '/usr/libexec/qemu-kvm'}, [], <Element 'machine' at 0x2364e90>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.2.0'}, [], <Element 'machine' at 0x2364ed0>, { '_children': [], 'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc'}, [], <Element 'machine' at 0x2364f10>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.0.0'}, [], <Element 'machine' at 0x2364f50>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.1.0'}, [], <Element 'machine' at 0x2364f90>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.3.0'}, [], <Element 'machine' at 0x2364fd0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.2.0'}, [], <Element 'machine' at 0x2369050>, { '_children': [], 'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'q35'}, [], <Element 'machine' at 0x2369090>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.4.0'}, [], <Element 'machine' at 0x23690d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.0.0'}, [], <Element 'machine' at 0x2369110>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.1.0'}, [], <Element 'machine' at 0x2369150>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.5.0'}, [], <Element 'machine' at 0x2369190>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.6.0'}, [], <Element 'machine' at 0x23691d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.1.0'}, [], <Element 'machine' at 0x2369210>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.0.0'}, [], <Element 'machine' at 0x2369250>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.2.0'}, [], <Element 'domain' at 0x2369290>, { '_children': [], 'attrib': { 'type': 'qemu'}, 'tag': 'domain', 'tail': '\n '}, [], <Element 'domain' at 0x23692d0>, { '_children': [<Element 'emulator' at 0x2369310>], 'attrib': { 'type': 'kvm'}, 'tag': 'domain', 'tail': '\n ', 'text': '\n '}, [<Element 'emulator' at 0x2369310>], <Element 'emulator' at 0x2369310>, { '_children': [], 'attrib': { }, 'tag': 'emulator', 'tail': '\n ', 'text': '/usr/libexec/qemu-kvm'}, [], <Element 'features' at 0x2369390>, { '_children': [ <Element 'cpuselection' at 0x23693d0>, <Element 'deviceboot' at 0x2369410>, <Element 'disksnapshot' at 0x2369450>, <Element 'acpi' at 0x2369490>, <Element 'apic' at 0x23694d0>, <Element 'pae' at 0x2369510>, <Element 'nonpae' at 0x2369550>], 'attrib': { }, 'tag': 'features', 'tail': '\n ', 'text': '\n '}, [ <Element 'cpuselection' at 0x23693d0>, <Element 'deviceboot' at 0x2369410>, <Element 'disksnapshot' at 0x2369450>, <Element 'acpi' at 0x2369490>, <Element 'apic' at 0x23694d0>, <Element 'pae' at 0x2369510>, <Element 'nonpae' at 0x2369550>], <Element 'cpuselection' at 0x23693d0>, { '_children': [], 'attrib': { }, 'tag': 'cpuselection', 'tail': '\n '}, [], <Element 'deviceboot' at 0x2369410>, { '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail': '\n '}, [], <Element 'disksnapshot' at 0x2369450>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'no'}, 'tag': 'disksnapshot', 'tail': '\n '}, [], <Element 'acpi' at 0x2369490>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'yes'}, 'tag': 'acpi', 'tail': '\n '}, [], <Element 'apic' at 0x23694d0>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'no'}, 'tag': 'apic', 'tail': '\n '}, [], <Element 'pae' at 0x2369510>, { '_children': [], 'attrib': { }, 'tag': 'pae', 'tail': '\n '}, [], <Element 'nonpae' at 0x2369550>, { '_children': [], 'attrib': { }, 'tag': 'nonpae', 'tail': '\n '}, [], <Element 'guest' at 0x2369590>, { '_children': [ <Element 'os_type' at 0x23695d0>, <Element 'arch' at 0x2369610>, <Element 'features' at 0x2369bd0>], 'attrib': { }, 'tag': 'guest', 'tail': '\n\n', 'text': '\n '}, [ <Element 'os_type' at 0x23695d0>, <Element 'arch' at 0x2369610>, <Element 'features' at 0x2369bd0>], <Element 'os_type' at 0x23695d0>, { '_children': [], 'attrib': { }, 'tag': 'os_type', 'tail': '\n ', 'text': 'hvm'}, [], <Element 'arch' at 0x2369610>, { '_children': [ <Element 'wordsize' at 0x2369650>, <Element 'emulator' at 0x2369690>, <Element 'machine' at 0x2369710>, <Element 'machine' at 0x2369750>, <Element 'machine' at 0x2369790>, <Element 'machine' at 0x23697d0>, <Element 'machine' at 0x2369810>, <Element 'machine' at 0x2369850>, <Element 'machine' at 0x2369890>, <Element 'machine' at 0x23698d0>, <Element 'machine' at 0x2369910>, <Element 'machine' at 0x2369950>, <Element 'machine' at 0x2369990>, <Element 'machine' at 0x23699d0>, <Element 'machine' at 0x2369a10>, <Element 'machine' at 0x2369a50>, <Element 'machine' at 0x2369a90>, <Element 'domain' at 0x2369ad0>, <Element 'domain' at 0x2369b10>], 'attrib': { 'name': 'x86_64'}, 'tag': 'arch', 'tail': '\n ', 'text': '\n '}, [ <Element 'wordsize' at 0x2369650>, <Element 'emulator' at 0x2369690>, <Element 'machine' at 0x2369710>, <Element 'machine' at 0x2369750>, <Element 'machine' at 0x2369790>, <Element 'machine' at 0x23697d0>, <Element 'machine' at 0x2369810>, <Element 'machine' at 0x2369850>, <Element 'machine' at 0x2369890>, <Element 'machine' at 0x23698d0>, <Element 'machine' at 0x2369910>, <Element 'machine' at 0x2369950>, <Element 'machine' at 0x2369990>, <Element 'machine' at 0x23699d0>, <Element 'machine' at 0x2369a10>, <Element 'machine' at 0x2369a50>, <Element 'machine' at 0x2369a90>, <Element 'domain' at 0x2369ad0>, <Element 'domain' at 0x2369b10>], <Element 'wordsize' at 0x2369650>, { '_children': [], 'attrib': { }, 'tag': 'wordsize', 'tail': '\n ', 'text': '64'}, [], <Element 'emulator' at 0x2369690>, { '_children': [], 'attrib': { }, 'tag': 'emulator', 'tail': '\n ', 'text': '/usr/libexec/qemu-kvm'}, [], <Element 'machine' at 0x2369710>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.2.0'}, [], <Element 'machine' at 0x2369750>, { '_children': [], 'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc'}, [], <Element 'machine' at 0x2369790>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.0.0'}, [], <Element 'machine' at 0x23697d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.1.0'}, [], <Element 'machine' at 0x2369810>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.3.0'}, [], <Element 'machine' at 0x2369850>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.2.0'}, [], <Element 'machine' at 0x2369890>, { '_children': [], 'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'q35'}, [], <Element 'machine' at 0x23698d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.4.0'}, [], <Element 'machine' at 0x2369910>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.0.0'}, [], <Element 'machine' at 0x2369950>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.1.0'}, [], <Element 'machine' at 0x2369990>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.5.0'}, [], <Element 'machine' at 0x23699d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.6.0'}, [], <Element 'machine' at 0x2369a10>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.1.0'}, [], <Element 'machine' at 0x2369a50>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.0.0'}, [], <Element 'machine' at 0x2369a90>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.2.0'}, [], <Element 'domain' at 0x2369ad0>, { '_children': [], 'attrib': { 'type': 'qemu'}, 'tag': 'domain', 'tail': '\n '}, [], <Element 'domain' at 0x2369b10>, { '_children': [<Element 'emulator' at 0x2369b50>], 'attrib': { 'type': 'kvm'}, 'tag': 'domain', 'tail': '\n ', 'text': '\n '}, [<Element 'emulator' at 0x2369b50>], <Element 'emulator' at 0x2369b50>, { '_children': [], 'attrib': { }, 'tag': 'emulator', 'tail': '\n ', 'text': '/usr/libexec/qemu-kvm'}, [], <Element 'features' at 0x2369bd0>, { '_children': [ <Element 'cpuselection' at 0x2369c10>, <Element 'deviceboot' at 0x2369c50>, <Element 'disksnapshot' at 0x2369c90>, <Element 'acpi' at 0x2369cd0>, <Element 'apic' at 0x2369d10>], 'attrib': { }, 'tag': 'features', 'tail': '\n ', 'text': '\n '}, [ <Element 'cpuselection' at 0x2369c10>, <Element 'deviceboot' at 0x2369c50>, <Element 'disksnapshot' at 0x2369c90>, <Element 'acpi' at 0x2369cd0>, <Element 'apic' at 0x2369d10>], <Element 'cpuselection' at 0x2369c10>, { '_children': [], 'attrib': { }, 'tag': 'cpuselection', 'tail': '\n '}, [], <Element 'deviceboot' at 0x2369c50>, { '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail': '\n '}, [], <Element 'disksnapshot' at 0x2369c90>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'no'}, 'tag': 'disksnapshot', 'tail': '\n '}, [], <Element 'acpi' at 0x2369cd0>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'yes'}, 'tag': 'acpi', 'tail': '\n '}, [], <Element 'apic' at 0x2369d10>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'no'}, 'tag': 'apic', 'tail': '\n '}, [], [], (<type '_ctypes.Array'>,), { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>, '__doc__': None, '__module__': 'vdsm.netlink', '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>, '_length_': 60, '_type_': <class 'ctypes.c_char'>, 'raw': <attribute 'raw' of 'c_char_Array_60' objects>, 'value': <attribute 'value' of 'c_char_Array_60' objects>}, <class 'vdsm.netlink.c_char_Array_60'>, <attribute '__dict__' of 'c_char_Array_60' objects>, <attribute '__weakref__' of 'c_char_Array_60' objects>, ( <class 'vdsm.netlink.c_char_Array_60'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_60' objects>, <attribute 'value' of 'c_char_Array_60' objects>, <xml.etree.ElementTree.XMLParser object at 0x2357f50>, { '_doctype': None, '_error': <class 'xml.parsers.expat.ExpatError'>, '_names': { u'acpi': 'acpi', u'apic': 'apic', u'arch': 'arch', u'baselabel': 'baselabel', u'canonical': 'canonical', u'capabilities': 'capabilities', u'cell': 'cell', u'cells': 'cells', u'core_id': 'core_id', u'cores': 'cores', u'cpu': 'cpu', u'cpus': 'cpus', u'cpuselection': 'cpuselection', u'default': 'default', u'deviceboot': 'deviceboot', u'disksnapshot': 'disksnapshot', u'distances': 'distances', u'doi': 'doi', u'domain': 'domain', u'emulator': 'emulator', u'feature': 'feature', u'features': 'features', u'guest': 'guest', u'host': 'host', u'id': 'id', u'live': 'live', u'machine': 'machine', u'maxCpus': 'maxCpus', u'memory': 'memory', u'migration_features': 'migration_features', u'model': 'model', u'name': 'name', u'nonpae': 'nonpae', u'num': 'num', u'os_type': 'os_type', u'pae': 'pae', u'pages': 'pages', u'power_management': 'power_management', u'secmodel': 'secmodel', u'sibling': 'sibling', u'siblings': 'siblings', u'size': 'size', u'socket_id': 'socket_id', u'sockets': 'sockets', u'suspend_disk': 'suspend_disk', u'suspend_hybrid': 'suspend_hybrid', u'suspend_mem': 'suspend_mem', u'threads': 'threads', u'toggle': 'toggle', u'topology': 'topology', u'type': 'type', u'unit': 'unit', u'uri_transport': 'uri_transport', u'uri_transports': 'uri_transports', u'uuid': 'uuid', u'value': 'value', u'vendor': 'vendor', u'wordsize': 'wordsize'}, '_target': <xml.etree.ElementTree.TreeBuilder object at 0x2357f10>, 'entity': { }, 'parser': <pyexpat.xmlparser object at 0x22e5360>, 'version': 'Expat 2.1.0'}, <pyexpat.xmlparser object at 0x22e5360>, <xml.etree.ElementTree.TreeBuilder object at 0x2357f10>, <bound method XMLParser._start_list of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._end of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._pi of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._data of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._comment of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._default of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, { '_data': [], '_elem': [], '_factory': <class 'xml.etree.ElementTree.Element'>, '_last': <Element 'capabilities' at 0x2357f90>, '_tail': 1}, [], <Element 'cell' at 0x23646d0>, <Element 'cells' at 0x2364690>, <Element 'topology' at 0x2364650>, <Element 'host' at 0x2357fd0>, <Element 'capabilities' at 0x2357f90>, { '_children': [ <Element 'memory' at 0x2364710>, <Element 'pages' at 0x2364750>, <Element 'pages' at 0x2364790>, <Element 'distances' at 0x23647d0>, <Element 'cpus' at 0x2364850>], 'attrib': { 'id': '0'}, 'tag': 'cell', 'tail': '\n ', 'text': '\n '}, { '_children': [<Element 'cell' at 0x23646d0>], 'attrib': { 'num': '1'}, 'tag': 'cells', 'tail': '\n ', 'text': '\n '}, { '_children': [<Element 'cells' at 0x2364690>], 'attrib': { }, 'tag': 'topology', 'tail': '\n ', 'text': '\n '}, { '_children': [ <Element 'uuid' at 0x2364050>, <Element 'cpu' at 0x2364090>, <Element 'power_management' at 0x2364390>, <Element 'migration_features' at 0x23644d0>, <Element 'topology' at 0x2364650>, <Element 'secmodel' at 0x2364a90>, <Element 'secmodel' at 0x2364bd0>], 'attrib': { }, 'tag': 'host', 'tail': '\n\n ', 'text': '\n '}, { '_children': [ <Element 'host' at 0x2357fd0>, <Element 'guest' at 0x2364d10>, <Element 'guest' at 0x2369590>], 'attrib': { }, 'tag': 'capabilities', 'text': '\n\n '}, [ <Element 'memory' at 0x2364710>, <Element 'pages' at 0x2364750>, <Element 'pages' at 0x2364790>, <Element 'distances' at 0x23647d0>, <Element 'cpus' at 0x2364850>], [<Element 'cell' at 0x23646d0>], [<Element 'cells' at 0x2364690>], [ <Element 'uuid' at 0x2364050>, <Element 'cpu' at 0x2364090>, <Element 'power_management' at 0x2364390>, <Element 'migration_features' at 0x23644d0>, <Element 'topology' at 0x2364650>, <Element 'secmodel' at 0x2364a90>, <Element 'secmodel' at 0x2364bd0>], [ <Element 'host' at 0x2357fd0>, <Element 'guest' at 0x2364d10>, <Element 'guest' at 0x2369590>], <Element 'migration_features' at 0x23644d0>, <Element 'power_management' at 0x2364390>, <Element 'cpu' at 0x2364090>, <Element 'uuid' at 0x2364050>, { '_children': [ <Element 'live' at 0x2364510>, <Element 'uri_transports' at 0x2364590>], 'attrib': { }, 'tag': 'migration_features', 'tail': '\n ', 'text': '\n '}, { '_children': [ <Element 'suspend_mem' at 0x23643d0>, <Element 'suspend_disk' at 0x2364410>, <Element 'suspend_hybrid' at 0x2364490>], 'attrib': { }, 'tag': 'power_management', 'tail': '\n ', 'text': '\n '}, { '_children': [ <Element 'arch' at 0x23640d0>, <Element 'model' at 0x2364110>, <Element 'vendor' at 0x2364150>, <Element 'topology' at 0x2364190>, <Element 'feature' at 0x23641d0>, <Element 'feature' at 0x2364210>, <Element 'feature' at 0x2364250>, <Element 'feature' at 0x2364290>, <Element 'feature' at 0x23642d0>, <Element 'pages' at 0x2364310>, <Element 'pages' at 0x2364350>], 'attrib': { }, 'tag': 'cpu', 'tail': '\n ', 'text': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'uuid', 'tail': '\n ', 'text': 'a920d963-c18f-4fda-aa5a-deee9bbcde9d'}, [<Element 'live' at 0x2364510>, <Element 'uri_transports' at 0x2364590>], [ <Element 'suspend_mem' at 0x23643d0>, <Element 'suspend_disk' at 0x2364410>, <Element 'suspend_hybrid' at 0x2364490>], [ <Element 'arch' at 0x23640d0>, <Element 'model' at 0x2364110>, <Element 'vendor' at 0x2364150>, <Element 'topology' at 0x2364190>, <Element 'feature' at 0x23641d0>, <Element 'feature' at 0x2364210>, <Element 'feature' at 0x2364250>, <Element 'feature' at 0x2364290>, <Element 'feature' at 0x23642d0>, <Element 'pages' at 0x2364310>, <Element 'pages' at 0x2364350>], [], <Element 'uri_transports' at 0x2364590>, <Element 'live' at 0x2364510>, <Element 'suspend_hybrid' at 0x2364490>, <Element 'suspend_disk' at 0x2364410>, <Element 'suspend_mem' at 0x23643d0>, <Element 'pages' at 0x2364350>, <Element 'pages' at 0x2364310>, <Element 'feature' at 0x23642d0>, <Element 'feature' at 0x2364290>, <Element 'feature' at 0x2364250>, <Element 'feature' at 0x2364210>, <Element 'feature' at 0x23641d0>, <Element 'topology' at 0x2364190>, <Element 'vendor' at 0x2364150>, <Element 'model' at 0x2364110>, <Element 'arch' at 0x23640d0>, { '_children': [ <Element 'uri_transport' at 0x23645d0>, <Element 'uri_transport' at 0x2364610>], 'attrib': { }, 'tag': 'uri_transports', 'tail': '\n ', 'text': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'live', 'tail': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'suspend_hybrid', 'tail': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'suspend_disk', 'tail': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'suspend_mem', 'tail': '\n '}, { '_children': [], 'attrib': { 'size': '2048', 'unit': 'KiB'}, 'tag': 'pages', 'tail': '\n '}, { '_children': [], 'attrib': { 'size': '4', 'unit': 'KiB'}, 'tag': 'pages', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'ss'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'vmx'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'pcid'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'osxsave'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'hypervisor'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'cores': '1', 'sockets': '8', 'threads': '1'}, 'tag': 'topology', 'tail': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'vendor', 'tail': '\n ', 'text': 'Intel'}, { '_children': [], 'attrib': { }, 'tag': 'model', 'tail': '\n ', 'text': 'IvyBridge'}, { '_children': [], 'attrib': { }, 'tag': 'arch', 'tail': '\n ', 'text': 'x86_64'}, [ <Element 'uri_transport' at 0x23645d0>, <Element 'uri_transport' at 0x2364610>], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], <Element 'uri_transport' at 0x2364610>, <Element 'uri_transport' at 0x23645d0>, { '_children': [], 'attrib': { }, 'tag': 'uri_transport', 'tail': '\n ', 'text': 'rdma'}, { '_children': [], 'attrib': { }, 'tag': 'uri_transport', 'tail': '\n ', 'text': 'tcp'}, [], [], (<type '_ctypes.Array'>,), { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>, '__doc__': None, '__module__': 'vdsm.netlink', '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>, '_length_': 60, '_type_': <class 'ctypes.c_char'>, 'raw': <attribute 'raw' of 'c_char_Array_60' objects>, 'value': <attribute 'value' of 'c_char_Array_60' objects>}, <class 'vdsm.netlink.c_char_Array_60'>, <attribute '__dict__' of 'c_char_Array_60' objects>, <attribute '__weakref__' of 'c_char_Array_60' objects>, ( <class 'vdsm.netlink.c_char_Array_60'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_60' objects>, <attribute 'value' of 'c_char_Array_60' objects>]