Hi all, Continuing the leak investigation Francesco and Milan are working on, I posted this patch, adding leak health monitoring to vdsm [1]. This patch currently monitor collectible objects, and we may add other interesting stuff later if needed. To enable the monitor, you must set: [devel] health_monitor_enable = true In vdsm.conf, and restart vdsm. Here is an example logs - this was logged 60 seconds after starting vdsm, when running as spm: Thread-13::DEBUG::2016-01-14 18:53:43,239::health::84::health::(_check) Checking health Thread-13::DEBUG::2016-01-14 18:53:43,272::health::86::health::(_check) Collected 460 objects Thread-13::WARNING::2016-01-14 18:53:43,277::health::100::health::(_check) Found 5873 uncollectible objects, 10 suspects implementing __del__: ['<pthreading.Lock object at 0x7f012c115320>', '<pthreading.Lock object at 0x7f012c115440>', '<pthreading.Lock object at 0x7f012c1153b0>', '<pthreading.Lock object at 0x7f012c1154d0>', '<pthread.Cond object at 0x7f012c111350>', '<pthread.Cond object at 0x7f012c1112d0>', '<pthread.Cond object at 0x7f012c111250>', '<pthread.Cond object at 0x7f012c111850>', '<pthread.Cond object at 0x7f012c1117d0>', '<pthread.Cond object at 0x7f012c111750>'] We found 460 uncollectible objects, 10 of them are objects implementing __del__. Having such object in a reference cycle make the entire cycle uncollectible. The pthreading objects in this log grow as vdsm continue to run, even without any interaction with engine. The pthreading objects leak is caused by a cycle in storage.misc.RWLock, fixed in patch [2]. After solving the pthreading leak, we still have many uncollectible objects in the logs, listing them, it seems that the suspect is hiding inside a collection (dict, list, tuple) and does not appear in the gc.garbage list. I will address this in later patch, searching in the entire object graph instead of the top level of gc.garbage. See end of this mail for list of these objects. I printed them like this: Enable manhole debugging shell: [devel] manhole_enable = true Connect to vdsm using manhole: nc -U /run/vdsm/vdsmd.manhole Print garbage: >>> import pprint >>> import gc >>> with open('/tmp/garbage.txt', 'w') as f: pprint.pprint(gc.garbage, stream=f, indent=4) These objects looks related to libvirt - Francesco, can you take a look? [1] https://gerrit.ovirt.org/51708 [2] https://gerrit.ovirt.org/51868 Nir --- garbage after patch [2] --- [ (<type '_ctypes.Array'>,), { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>, '__doc__': None, '__module__': 'vdsm.netlink', '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>, '_length_': 60, '_type_': <class 'ctypes.c_char'>, 'raw': <attribute 'raw' of 'c_char_Array_60' objects>, 'value': <attribute 'value' of 'c_char_Array_60' objects>}, <class 'vdsm.netlink.c_char_Array_60'>, <attribute '__dict__' of 'c_char_Array_60' objects>, <attribute '__weakref__' of 'c_char_Array_60' objects>, ( <class 'vdsm.netlink.c_char_Array_60'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_60' objects>, <attribute 'value' of 'c_char_Array_60' objects>, <cell at 0x23662f0: bool object at 0x7f1e086c8c40>, <cell at 0x2366328: NoneType object at 0x7f1e086dcf00>, <cell at 0x2366360: function object at 0x233f488>, ( <cell at 0x23662f0: bool object at 0x7f1e086c8c40>, <cell at 0x2366328: NoneType object at 0x7f1e086dcf00>, <cell at 0x2366360: function object at 0x233f488>), <function wrapMethod at 0x233f488>, [ [ <Recursion on list with id=34033464>, <Recursion on list with id=34033464>, 'class'], [ <Recursion on list with id=34033464>, <Recursion on list with id=34033464>, 'class'], None], [ [ <Recursion on list with id=33979064>, <Recursion on list with id=33979064>, None], [ <Recursion on list with id=33979064>, <Recursion on list with id=33979064>, None], 'class'], [ [ <Recursion on list with id=35535256>, <Recursion on list with id=35535256>, 'class'], [ <Recursion on list with id=35535256>, <Recursion on list with id=35535256>, 'class'], None], [ [ <Recursion on list with id=35535832>, <Recursion on list with id=35535832>, None], [ <Recursion on list with id=35535832>, <Recursion on list with id=35535832>, None], 'class'], [ [ <Recursion on list with id=35556312>, <Recursion on list with id=35556312>, 'class'], [ <Recursion on list with id=35556312>, <Recursion on list with id=35556312>, 'class'], None], [ [ <Recursion on list with id=35556600>, <Recursion on list with id=35556600>, None], [ <Recursion on list with id=35556600>, <Recursion on list with id=35556600>, None], 'class'], [ [ <Recursion on list with id=35556960>, <Recursion on list with id=35556960>, 'class'], [ <Recursion on list with id=35556960>, <Recursion on list with id=35556960>, 'class'], None], [ [ <Recursion on list with id=35561912>, <Recursion on list with id=35561912>, None], [ <Recursion on list with id=35561912>, <Recursion on list with id=35561912>, None], 'class'], [ [ <Recursion on list with id=35583904>, <Recursion on list with id=35583904>, 'class'], [ <Recursion on list with id=35583904>, <Recursion on list with id=35583904>, 'class'], None], [ [ <Recursion on list with id=35584912>, <Recursion on list with id=35584912>, None], [ <Recursion on list with id=35584912>, <Recursion on list with id=35584912>, None], 'class'], [ [ <Recursion on list with id=36340368>, <Recursion on list with id=36340368>, 'class'], [ <Recursion on list with id=36340368>, <Recursion on list with id=36340368>, 'class'], None], [ [ <Recursion on list with id=36340872>, <Recursion on list with id=36340872>, None], [ <Recursion on list with id=36340872>, <Recursion on list with id=36340872>, None], 'class'], [ [ <Recursion on list with id=36364872>, <Recursion on list with id=36364872>, 'class'], [ <Recursion on list with id=36364872>, <Recursion on list with id=36364872>, 'class'], None], [ [ <Recursion on list with id=36365520>, <Recursion on list with id=36365520>, None], [ <Recursion on list with id=36365520>, <Recursion on list with id=36365520>, None], 'class'], [ [ <Recursion on list with id=36457216>, <Recursion on list with id=36457216>, 'class'], [ <Recursion on list with id=36457216>, <Recursion on list with id=36457216>, 'class'], None], [ [ <Recursion on list with id=36458008>, <Recursion on list with id=36458008>, None], [ <Recursion on list with id=36458008>, <Recursion on list with id=36458008>, None], 'class'], [ [ <Recursion on list with id=36460520>, <Recursion on list with id=36460520>, 'class'], [ <Recursion on list with id=36460520>, <Recursion on list with id=36460520>, 'class'], None], [ [ <Recursion on list with id=36461168>, <Recursion on list with id=36461168>, None], [ <Recursion on list with id=36461168>, <Recursion on list with id=36461168>, None], 'class'], [ [ <Recursion on list with id=36562344>, <Recursion on list with id=36562344>, 'class'], [ <Recursion on list with id=36562344>, <Recursion on list with id=36562344>, 'class'], None], [ [ <Recursion on list with id=36563280>, <Recursion on list with id=36563280>, None], [ <Recursion on list with id=36563280>, <Recursion on list with id=36563280>, None], 'class'], <Element 'memory' at 0x2364710>, { '_children': [], 'attrib': { 'unit': 'KiB'}, 'tag': 'memory', 'tail': '\n ', 'text': '4095476'}, [], <Element 'pages' at 0x2364750>, { '_children': [], 'attrib': { 'size': '4', 'unit': 'KiB'}, 'tag': 'pages', 'tail': '\n ', 'text': '1023869'}, [], <Element 'pages' at 0x2364790>, { '_children': [], 'attrib': { 'size': '2048', 'unit': 'KiB'}, 'tag': 'pages', 'tail': '\n ', 'text': '0'}, [], <Element 'distances' at 0x23647d0>, { '_children': [<Element 'sibling' at 0x2364810>], 'attrib': { }, 'tag': 'distances', 'tail': '\n ', 'text': '\n '}, [<Element 'sibling' at 0x2364810>], <Element 'sibling' at 0x2364810>, { '_children': [], 'attrib': { 'id': '0', 'value': '10'}, 'tag': 'sibling', 'tail': '\n '}, [], <Element 'cpus' at 0x2364850>, { '_children': [ <Element 'cpu' at 0x2364890>, <Element 'cpu' at 0x23648d0>, <Element 'cpu' at 0x2364910>, <Element 'cpu' at 0x2364950>, <Element 'cpu' at 0x2364990>, <Element 'cpu' at 0x23649d0>, <Element 'cpu' at 0x2364a10>, <Element 'cpu' at 0x2364a50>], 'attrib': { 'num': '8'}, 'tag': 'cpus', 'tail': '\n ', 'text': '\n '}, [ <Element 'cpu' at 0x2364890>, <Element 'cpu' at 0x23648d0>, <Element 'cpu' at 0x2364910>, <Element 'cpu' at 0x2364950>, <Element 'cpu' at 0x2364990>, <Element 'cpu' at 0x23649d0>, <Element 'cpu' at 0x2364a10>, <Element 'cpu' at 0x2364a50>], <Element 'cpu' at 0x2364890>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '0', 'siblings': '0', 'socket_id': '0'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x23648d0>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '1', 'siblings': '1', 'socket_id': '1'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364910>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '2', 'siblings': '2', 'socket_id': '2'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364950>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '3', 'siblings': '3', 'socket_id': '3'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364990>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '4', 'siblings': '4', 'socket_id': '4'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x23649d0>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '5', 'siblings': '5', 'socket_id': '5'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364a10>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '6', 'siblings': '6', 'socket_id': '6'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364a50>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '7', 'siblings': '7', 'socket_id': '7'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'secmodel' at 0x2364a90>, { '_children': [ <Element 'model' at 0x2364ad0>, <Element 'doi' at 0x2364b10>, <Element 'baselabel' at 0x2364b50>, <Element 'baselabel' at 0x2364b90>], 'attrib': { }, 'tag': 'secmodel', 'tail': '\n ', 'text': '\n '}, [ <Element 'model' at 0x2364ad0>, <Element 'doi' at 0x2364b10>, <Element 'baselabel' at 0x2364b50>, <Element 'baselabel' at 0x2364b90>], <Element 'model' at 0x2364ad0>, { '_children': [], 'attrib': { }, 'tag': 'model', 'tail': '\n ', 'text': 'selinux'}, [], <Element 'doi' at 0x2364b10>, { '_children': [], 'attrib': { }, 'tag': 'doi', 'tail': '\n ', 'text': '0'}, [], <Element 'baselabel' at 0x2364b50>, { '_children': [], 'attrib': { 'type': 'kvm'}, 'tag': 'baselabel', 'tail': '\n ', 'text': 'system_u:system_r:svirt_t:s0'}, [], <Element 'baselabel' at 0x2364b90>, { '_children': [], 'attrib': { 'type': 'qemu'}, 'tag': 'baselabel', 'tail': '\n ', 'text': 'system_u:system_r:svirt_tcg_t:s0'}, [], <Element 'secmodel' at 0x2364bd0>, { '_children': [ <Element 'model' at 0x2364c10>, <Element 'doi' at 0x2364c50>, <Element 'baselabel' at 0x2364c90>, <Element 'baselabel' at 0x2364cd0>], 'attrib': { }, 'tag': 'secmodel', 'tail': '\n ', 'text': '\n '}, [ <Element 'model' at 0x2364c10>, <Element 'doi' at 0x2364c50>, <Element 'baselabel' at 0x2364c90>, <Element 'baselabel' at 0x2364cd0>], <Element 'model' at 0x2364c10>, { '_children': [], 'attrib': { }, 'tag': 'model', 'tail': '\n ', 'text': 'dac'}, [], <Element 'doi' at 0x2364c50>, { '_children': [], 'attrib': { }, 'tag': 'doi', 'tail': '\n ', 'text': '0'}, [], <Element 'baselabel' at 0x2364c90>, { '_children': [], 'attrib': { 'type': 'kvm'}, 'tag': 'baselabel', 'tail': '\n ', 'text': '+107:+107'}, [], <Element 'baselabel' at 0x2364cd0>, { '_children': [], 'attrib': { 'type': 'qemu'}, 'tag': 'baselabel', 'tail': '\n ', 'text': '+107:+107'}, [], <Element 'guest' at 0x2364d10>, { '_children': [ <Element 'os_type' at 0x2364d50>, <Element 'arch' at 0x2364d90>, <Element 'features' at 0x2369390>], 'attrib': { }, 'tag': 'guest', 'tail': '\n\n ', 'text': '\n '}, [ <Element 'os_type' at 0x2364d50>, <Element 'arch' at 0x2364d90>, <Element 'features' at 0x2369390>], <Element 'os_type' at 0x2364d50>, { '_children': [], 'attrib': { }, 'tag': 'os_type', 'tail': '\n ', 'text': 'hvm'}, [], <Element 'arch' at 0x2364d90>, { '_children': [ <Element 'wordsize' at 0x2364dd0>, <Element 'emulator' at 0x2364e10>, <Element 'machine' at 0x2364e90>, <Element 'machine' at 0x2364ed0>, <Element 'machine' at 0x2364f10>, <Element 'machine' at 0x2364f50>, <Element 'machine' at 0x2364f90>, <Element 'machine' at 0x2364fd0>, <Element 'machine' at 0x2369050>, <Element 'machine' at 0x2369090>, <Element 'machine' at 0x23690d0>, <Element 'machine' at 0x2369110>, <Element 'machine' at 0x2369150>, <Element 'machine' at 0x2369190>, <Element 'machine' at 0x23691d0>, <Element 'machine' at 0x2369210>, <Element 'machine' at 0x2369250>, <Element 'domain' at 0x2369290>, <Element 'domain' at 0x23692d0>], 'attrib': { 'name': 'i686'}, 'tag': 'arch', 'tail': '\n ', 'text': '\n '}, [ <Element 'wordsize' at 0x2364dd0>, <Element 'emulator' at 0x2364e10>, <Element 'machine' at 0x2364e90>, <Element 'machine' at 0x2364ed0>, <Element 'machine' at 0x2364f10>, <Element 'machine' at 0x2364f50>, <Element 'machine' at 0x2364f90>, <Element 'machine' at 0x2364fd0>, <Element 'machine' at 0x2369050>, <Element 'machine' at 0x2369090>, <Element 'machine' at 0x23690d0>, <Element 'machine' at 0x2369110>, <Element 'machine' at 0x2369150>, <Element 'machine' at 0x2369190>, <Element 'machine' at 0x23691d0>, <Element 'machine' at 0x2369210>, <Element 'machine' at 0x2369250>, <Element 'domain' at 0x2369290>, <Element 'domain' at 0x23692d0>], <Element 'wordsize' at 0x2364dd0>, { '_children': [], 'attrib': { }, 'tag': 'wordsize', 'tail': '\n ', 'text': '32'}, [], <Element 'emulator' at 0x2364e10>, { '_children': [], 'attrib': { }, 'tag': 'emulator', 'tail': '\n ', 'text': '/usr/libexec/qemu-kvm'}, [], <Element 'machine' at 0x2364e90>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.2.0'}, [], <Element 'machine' at 0x2364ed0>, { '_children': [], 'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc'}, [], <Element 'machine' at 0x2364f10>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.0.0'}, [], <Element 'machine' at 0x2364f50>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.1.0'}, [], <Element 'machine' at 0x2364f90>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.3.0'}, [], <Element 'machine' at 0x2364fd0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.2.0'}, [], <Element 'machine' at 0x2369050>, { '_children': [], 'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'q35'}, [], <Element 'machine' at 0x2369090>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.4.0'}, [], <Element 'machine' at 0x23690d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.0.0'}, [], <Element 'machine' at 0x2369110>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.1.0'}, [], <Element 'machine' at 0x2369150>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.5.0'}, [], <Element 'machine' at 0x2369190>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.6.0'}, [], <Element 'machine' at 0x23691d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.1.0'}, [], <Element 'machine' at 0x2369210>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.0.0'}, [], <Element 'machine' at 0x2369250>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.2.0'}, [], <Element 'domain' at 0x2369290>, { '_children': [], 'attrib': { 'type': 'qemu'}, 'tag': 'domain', 'tail': '\n '}, [], <Element 'domain' at 0x23692d0>, { '_children': [<Element 'emulator' at 0x2369310>], 'attrib': { 'type': 'kvm'}, 'tag': 'domain', 'tail': '\n ', 'text': '\n '}, [<Element 'emulator' at 0x2369310>], <Element 'emulator' at 0x2369310>, { '_children': [], 'attrib': { }, 'tag': 'emulator', 'tail': '\n ', 'text': '/usr/libexec/qemu-kvm'}, [], <Element 'features' at 0x2369390>, { '_children': [ <Element 'cpuselection' at 0x23693d0>, <Element 'deviceboot' at 0x2369410>, <Element 'disksnapshot' at 0x2369450>, <Element 'acpi' at 0x2369490>, <Element 'apic' at 0x23694d0>, <Element 'pae' at 0x2369510>, <Element 'nonpae' at 0x2369550>], 'attrib': { }, 'tag': 'features', 'tail': '\n ', 'text': '\n '}, [ <Element 'cpuselection' at 0x23693d0>, <Element 'deviceboot' at 0x2369410>, <Element 'disksnapshot' at 0x2369450>, <Element 'acpi' at 0x2369490>, <Element 'apic' at 0x23694d0>, <Element 'pae' at 0x2369510>, <Element 'nonpae' at 0x2369550>], <Element 'cpuselection' at 0x23693d0>, { '_children': [], 'attrib': { }, 'tag': 'cpuselection', 'tail': '\n '}, [], <Element 'deviceboot' at 0x2369410>, { '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail': '\n '}, [], <Element 'disksnapshot' at 0x2369450>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'no'}, 'tag': 'disksnapshot', 'tail': '\n '}, [], <Element 'acpi' at 0x2369490>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'yes'}, 'tag': 'acpi', 'tail': '\n '}, [], <Element 'apic' at 0x23694d0>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'no'}, 'tag': 'apic', 'tail': '\n '}, [], <Element 'pae' at 0x2369510>, { '_children': [], 'attrib': { }, 'tag': 'pae', 'tail': '\n '}, [], <Element 'nonpae' at 0x2369550>, { '_children': [], 'attrib': { }, 'tag': 'nonpae', 'tail': '\n '}, [], <Element 'guest' at 0x2369590>, { '_children': [ <Element 'os_type' at 0x23695d0>, <Element 'arch' at 0x2369610>, <Element 'features' at 0x2369bd0>], 'attrib': { }, 'tag': 'guest', 'tail': '\n\n', 'text': '\n '}, [ <Element 'os_type' at 0x23695d0>, <Element 'arch' at 0x2369610>, <Element 'features' at 0x2369bd0>], <Element 'os_type' at 0x23695d0>, { '_children': [], 'attrib': { }, 'tag': 'os_type', 'tail': '\n ', 'text': 'hvm'}, [], <Element 'arch' at 0x2369610>, { '_children': [ <Element 'wordsize' at 0x2369650>, <Element 'emulator' at 0x2369690>, <Element 'machine' at 0x2369710>, <Element 'machine' at 0x2369750>, <Element 'machine' at 0x2369790>, <Element 'machine' at 0x23697d0>, <Element 'machine' at 0x2369810>, <Element 'machine' at 0x2369850>, <Element 'machine' at 0x2369890>, <Element 'machine' at 0x23698d0>, <Element 'machine' at 0x2369910>, <Element 'machine' at 0x2369950>, <Element 'machine' at 0x2369990>, <Element 'machine' at 0x23699d0>, <Element 'machine' at 0x2369a10>, <Element 'machine' at 0x2369a50>, <Element 'machine' at 0x2369a90>, <Element 'domain' at 0x2369ad0>, <Element 'domain' at 0x2369b10>], 'attrib': { 'name': 'x86_64'}, 'tag': 'arch', 'tail': '\n ', 'text': '\n '}, [ <Element 'wordsize' at 0x2369650>, <Element 'emulator' at 0x2369690>, <Element 'machine' at 0x2369710>, <Element 'machine' at 0x2369750>, <Element 'machine' at 0x2369790>, <Element 'machine' at 0x23697d0>, <Element 'machine' at 0x2369810>, <Element 'machine' at 0x2369850>, <Element 'machine' at 0x2369890>, <Element 'machine' at 0x23698d0>, <Element 'machine' at 0x2369910>, <Element 'machine' at 0x2369950>, <Element 'machine' at 0x2369990>, <Element 'machine' at 0x23699d0>, <Element 'machine' at 0x2369a10>, <Element 'machine' at 0x2369a50>, <Element 'machine' at 0x2369a90>, <Element 'domain' at 0x2369ad0>, <Element 'domain' at 0x2369b10>], <Element 'wordsize' at 0x2369650>, { '_children': [], 'attrib': { }, 'tag': 'wordsize', 'tail': '\n ', 'text': '64'}, [], <Element 'emulator' at 0x2369690>, { '_children': [], 'attrib': { }, 'tag': 'emulator', 'tail': '\n ', 'text': '/usr/libexec/qemu-kvm'}, [], <Element 'machine' at 0x2369710>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.2.0'}, [], <Element 'machine' at 0x2369750>, { '_children': [], 'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc'}, [], <Element 'machine' at 0x2369790>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.0.0'}, [], <Element 'machine' at 0x23697d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.1.0'}, [], <Element 'machine' at 0x2369810>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.3.0'}, [], <Element 'machine' at 0x2369850>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.2.0'}, [], <Element 'machine' at 0x2369890>, { '_children': [], 'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'q35'}, [], <Element 'machine' at 0x23698d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.4.0'}, [], <Element 'machine' at 0x2369910>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.0.0'}, [], <Element 'machine' at 0x2369950>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.1.0'}, [], <Element 'machine' at 0x2369990>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.5.0'}, [], <Element 'machine' at 0x23699d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.6.0'}, [], <Element 'machine' at 0x2369a10>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.1.0'}, [], <Element 'machine' at 0x2369a50>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.0.0'}, [], <Element 'machine' at 0x2369a90>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.2.0'}, [], <Element 'domain' at 0x2369ad0>, { '_children': [], 'attrib': { 'type': 'qemu'}, 'tag': 'domain', 'tail': '\n '}, [], <Element 'domain' at 0x2369b10>, { '_children': [<Element 'emulator' at 0x2369b50>], 'attrib': { 'type': 'kvm'}, 'tag': 'domain', 'tail': '\n ', 'text': '\n '}, [<Element 'emulator' at 0x2369b50>], <Element 'emulator' at 0x2369b50>, { '_children': [], 'attrib': { }, 'tag': 'emulator', 'tail': '\n ', 'text': '/usr/libexec/qemu-kvm'}, [], <Element 'features' at 0x2369bd0>, { '_children': [ <Element 'cpuselection' at 0x2369c10>, <Element 'deviceboot' at 0x2369c50>, <Element 'disksnapshot' at 0x2369c90>, <Element 'acpi' at 0x2369cd0>, <Element 'apic' at 0x2369d10>], 'attrib': { }, 'tag': 'features', 'tail': '\n ', 'text': '\n '}, [ <Element 'cpuselection' at 0x2369c10>, <Element 'deviceboot' at 0x2369c50>, <Element 'disksnapshot' at 0x2369c90>, <Element 'acpi' at 0x2369cd0>, <Element 'apic' at 0x2369d10>], <Element 'cpuselection' at 0x2369c10>, { '_children': [], 'attrib': { }, 'tag': 'cpuselection', 'tail': '\n '}, [], <Element 'deviceboot' at 0x2369c50>, { '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail': '\n '}, [], <Element 'disksnapshot' at 0x2369c90>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'no'}, 'tag': 'disksnapshot', 'tail': '\n '}, [], <Element 'acpi' at 0x2369cd0>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'yes'}, 'tag': 'acpi', 'tail': '\n '}, [], <Element 'apic' at 0x2369d10>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'no'}, 'tag': 'apic', 'tail': '\n '}, [], [], (<type '_ctypes.Array'>,), { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>, '__doc__': None, '__module__': 'vdsm.netlink', '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>, '_length_': 60, '_type_': <class 'ctypes.c_char'>, 'raw': <attribute 'raw' of 'c_char_Array_60' objects>, 'value': <attribute 'value' of 'c_char_Array_60' objects>}, <class 'vdsm.netlink.c_char_Array_60'>, <attribute '__dict__' of 'c_char_Array_60' objects>, <attribute '__weakref__' of 'c_char_Array_60' objects>, ( <class 'vdsm.netlink.c_char_Array_60'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_60' objects>, <attribute 'value' of 'c_char_Array_60' objects>, <xml.etree.ElementTree.XMLParser object at 0x2357f50>, { '_doctype': None, '_error': <class 'xml.parsers.expat.ExpatError'>, '_names': { u'acpi': 'acpi', u'apic': 'apic', u'arch': 'arch', u'baselabel': 'baselabel', u'canonical': 'canonical', u'capabilities': 'capabilities', u'cell': 'cell', u'cells': 'cells', u'core_id': 'core_id', u'cores': 'cores', u'cpu': 'cpu', u'cpus': 'cpus', u'cpuselection': 'cpuselection', u'default': 'default', u'deviceboot': 'deviceboot', u'disksnapshot': 'disksnapshot', u'distances': 'distances', u'doi': 'doi', u'domain': 'domain', u'emulator': 'emulator', u'feature': 'feature', u'features': 'features', u'guest': 'guest', u'host': 'host', u'id': 'id', u'live': 'live', u'machine': 'machine', u'maxCpus': 'maxCpus', u'memory': 'memory', u'migration_features': 'migration_features', u'model': 'model', u'name': 'name', u'nonpae': 'nonpae', u'num': 'num', u'os_type': 'os_type', u'pae': 'pae', u'pages': 'pages', u'power_management': 'power_management', u'secmodel': 'secmodel', u'sibling': 'sibling', u'siblings': 'siblings', u'size': 'size', u'socket_id': 'socket_id', u'sockets': 'sockets', u'suspend_disk': 'suspend_disk', u'suspend_hybrid': 'suspend_hybrid', u'suspend_mem': 'suspend_mem', u'threads': 'threads', u'toggle': 'toggle', u'topology': 'topology', u'type': 'type', u'unit': 'unit', u'uri_transport': 'uri_transport', u'uri_transports': 'uri_transports', u'uuid': 'uuid', u'value': 'value', u'vendor': 'vendor', u'wordsize': 'wordsize'}, '_target': <xml.etree.ElementTree.TreeBuilder object at 0x2357f10>, 'entity': { }, 'parser': <pyexpat.xmlparser object at 0x22e5360>, 'version': 'Expat 2.1.0'}, <pyexpat.xmlparser object at 0x22e5360>, <xml.etree.ElementTree.TreeBuilder object at 0x2357f10>, <bound method XMLParser._start_list of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._end of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._pi of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._data of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._comment of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._default of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, { '_data': [], '_elem': [], '_factory': <class 'xml.etree.ElementTree.Element'>, '_last': <Element 'capabilities' at 0x2357f90>, '_tail': 1}, [], <Element 'cell' at 0x23646d0>, <Element 'cells' at 0x2364690>, <Element 'topology' at 0x2364650>, <Element 'host' at 0x2357fd0>, <Element 'capabilities' at 0x2357f90>, { '_children': [ <Element 'memory' at 0x2364710>, <Element 'pages' at 0x2364750>, <Element 'pages' at 0x2364790>, <Element 'distances' at 0x23647d0>, <Element 'cpus' at 0x2364850>], 'attrib': { 'id': '0'}, 'tag': 'cell', 'tail': '\n ', 'text': '\n '}, { '_children': [<Element 'cell' at 0x23646d0>], 'attrib': { 'num': '1'}, 'tag': 'cells', 'tail': '\n ', 'text': '\n '}, { '_children': [<Element 'cells' at 0x2364690>], 'attrib': { }, 'tag': 'topology', 'tail': '\n ', 'text': '\n '}, { '_children': [ <Element 'uuid' at 0x2364050>, <Element 'cpu' at 0x2364090>, <Element 'power_management' at 0x2364390>, <Element 'migration_features' at 0x23644d0>, <Element 'topology' at 0x2364650>, <Element 'secmodel' at 0x2364a90>, <Element 'secmodel' at 0x2364bd0>], 'attrib': { }, 'tag': 'host', 'tail': '\n\n ', 'text': '\n '}, { '_children': [ <Element 'host' at 0x2357fd0>, <Element 'guest' at 0x2364d10>, <Element 'guest' at 0x2369590>], 'attrib': { }, 'tag': 'capabilities', 'text': '\n\n '}, [ <Element 'memory' at 0x2364710>, <Element 'pages' at 0x2364750>, <Element 'pages' at 0x2364790>, <Element 'distances' at 0x23647d0>, <Element 'cpus' at 0x2364850>], [<Element 'cell' at 0x23646d0>], [<Element 'cells' at 0x2364690>], [ <Element 'uuid' at 0x2364050>, <Element 'cpu' at 0x2364090>, <Element 'power_management' at 0x2364390>, <Element 'migration_features' at 0x23644d0>, <Element 'topology' at 0x2364650>, <Element 'secmodel' at 0x2364a90>, <Element 'secmodel' at 0x2364bd0>], [ <Element 'host' at 0x2357fd0>, <Element 'guest' at 0x2364d10>, <Element 'guest' at 0x2369590>], <Element 'migration_features' at 0x23644d0>, <Element 'power_management' at 0x2364390>, <Element 'cpu' at 0x2364090>, <Element 'uuid' at 0x2364050>, { '_children': [ <Element 'live' at 0x2364510>, <Element 'uri_transports' at 0x2364590>], 'attrib': { }, 'tag': 'migration_features', 'tail': '\n ', 'text': '\n '}, { '_children': [ <Element 'suspend_mem' at 0x23643d0>, <Element 'suspend_disk' at 0x2364410>, <Element 'suspend_hybrid' at 0x2364490>], 'attrib': { }, 'tag': 'power_management', 'tail': '\n ', 'text': '\n '}, { '_children': [ <Element 'arch' at 0x23640d0>, <Element 'model' at 0x2364110>, <Element 'vendor' at 0x2364150>, <Element 'topology' at 0x2364190>, <Element 'feature' at 0x23641d0>, <Element 'feature' at 0x2364210>, <Element 'feature' at 0x2364250>, <Element 'feature' at 0x2364290>, <Element 'feature' at 0x23642d0>, <Element 'pages' at 0x2364310>, <Element 'pages' at 0x2364350>], 'attrib': { }, 'tag': 'cpu', 'tail': '\n ', 'text': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'uuid', 'tail': '\n ', 'text': 'a920d963-c18f-4fda-aa5a-deee9bbcde9d'}, [<Element 'live' at 0x2364510>, <Element 'uri_transports' at 0x2364590>], [ <Element 'suspend_mem' at 0x23643d0>, <Element 'suspend_disk' at 0x2364410>, <Element 'suspend_hybrid' at 0x2364490>], [ <Element 'arch' at 0x23640d0>, <Element 'model' at 0x2364110>, <Element 'vendor' at 0x2364150>, <Element 'topology' at 0x2364190>, <Element 'feature' at 0x23641d0>, <Element 'feature' at 0x2364210>, <Element 'feature' at 0x2364250>, <Element 'feature' at 0x2364290>, <Element 'feature' at 0x23642d0>, <Element 'pages' at 0x2364310>, <Element 'pages' at 0x2364350>], [], <Element 'uri_transports' at 0x2364590>, <Element 'live' at 0x2364510>, <Element 'suspend_hybrid' at 0x2364490>, <Element 'suspend_disk' at 0x2364410>, <Element 'suspend_mem' at 0x23643d0>, <Element 'pages' at 0x2364350>, <Element 'pages' at 0x2364310>, <Element 'feature' at 0x23642d0>, <Element 'feature' at 0x2364290>, <Element 'feature' at 0x2364250>, <Element 'feature' at 0x2364210>, <Element 'feature' at 0x23641d0>, <Element 'topology' at 0x2364190>, <Element 'vendor' at 0x2364150>, <Element 'model' at 0x2364110>, <Element 'arch' at 0x23640d0>, { '_children': [ <Element 'uri_transport' at 0x23645d0>, <Element 'uri_transport' at 0x2364610>], 'attrib': { }, 'tag': 'uri_transports', 'tail': '\n ', 'text': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'live', 'tail': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'suspend_hybrid', 'tail': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'suspend_disk', 'tail': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'suspend_mem', 'tail': '\n '}, { '_children': [], 'attrib': { 'size': '2048', 'unit': 'KiB'}, 'tag': 'pages', 'tail': '\n '}, { '_children': [], 'attrib': { 'size': '4', 'unit': 'KiB'}, 'tag': 'pages', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'ss'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'vmx'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'pcid'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'osxsave'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'hypervisor'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'cores': '1', 'sockets': '8', 'threads': '1'}, 'tag': 'topology', 'tail': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'vendor', 'tail': '\n ', 'text': 'Intel'}, { '_children': [], 'attrib': { }, 'tag': 'model', 'tail': '\n ', 'text': 'IvyBridge'}, { '_children': [], 'attrib': { }, 'tag': 'arch', 'tail': '\n ', 'text': 'x86_64'}, [ <Element 'uri_transport' at 0x23645d0>, <Element 'uri_transport' at 0x2364610>], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], <Element 'uri_transport' at 0x2364610>, <Element 'uri_transport' at 0x23645d0>, { '_children': [], 'attrib': { }, 'tag': 'uri_transport', 'tail': '\n ', 'text': 'rdma'}, { '_children': [], 'attrib': { }, 'tag': 'uri_transport', 'tail': '\n ', 'text': 'tcp'}, [], [], (<type '_ctypes.Array'>,), { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>, '__doc__': None, '__module__': 'vdsm.netlink', '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>, '_length_': 60, '_type_': <class 'ctypes.c_char'>, 'raw': <attribute 'raw' of 'c_char_Array_60' objects>, 'value': <attribute 'value' of 'c_char_Array_60' objects>}, <class 'vdsm.netlink.c_char_Array_60'>, <attribute '__dict__' of 'c_char_Array_60' objects>, <attribute '__weakref__' of 'c_char_Array_60' objects>, ( <class 'vdsm.netlink.c_char_Array_60'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_60' objects>, <attribute 'value' of 'c_char_Array_60' objects>]

From: "Nir Soffer" <nsoffer(a)redhat.com&gt; To: "devel" <devel(a)ovirt.org&gt;, "Francesco Romani" <fromani(a)redhat.com&gt; Sent: Friday, January 15, 2016 9:52:13 AM Subject: Re: [VDSM] health: Introduce Vdsm health monitoring I tested with https://gerrit.ovirt.org/#/c/51630 libvirt.virDomain object does not leak any more, but pthreading object are still leaking, and the number of uncollectible objects is still very high (12,915). Each time we start and stop a vm, we leak about 6000 objects which are part of a reference cycle, and each time we create and leak 27 pthreading locks/conditions which are probably the reason for this leak (because they implement __del__ or reference object implementing __del__.

I tested with https://gerrit.ovirt.org/#/c/51630 libvirt.virDomain object does not leak any more, but pthreading object are still leaking, and the number of uncollectible objects is still very high (12,915). Each time we start and stop a vm, we leak about 6000 objects which are part of a reference cycle, and each time we create and leak 27 pthreading locks/conditions which are probably the reason for this leak (because they implement __del__ or reference object implementing __del__. When vdsm is idle, we leak 18 objects per minute, we don't know yet why. # tail -f /var/log/vdsm/vdsm.log | grep health Vdsm was connected to storage pool here, running as the spm... Thread-15::DEBUG::2016-01-15 10:00:00,159::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:00:00,178::health::86::health::(_check) Collected 468 objects Thread-15::WARNING::2016-01-15 10:00:00,182::health::100::health::(_check) Found 5953 uncollectible objects, 0 suspects implementing __del__: [] Thread-15::DEBUG::2016-01-15 10:01:00,182::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:01:00,207::health::86::health::(_check) Collected 202 objects Thread-15::WARNING::2016-01-15 10:01:00,211::health::100::health::(_check) Found 6169 uncollectible objects, 23 suspects implementing __del__: ['<pthreading.Lock object at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>', '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>', '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>', '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>', '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>', '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>', '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>', '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>'] vm was started here... Thread-15::DEBUG::2016-01-15 10:02:00,212::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:02:00,243::health::86::health::(_check) Collected 1689 objects Thread-15::WARNING::2016-01-15 10:02:00,252::health::100::health::(_check) Found 11107 uncollectible objects, 23 suspects implementing __del__: ['<pthreading.Lock object at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>', '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>', '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>', '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>', '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>', '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>', '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>', '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>'] Thread-15::DEBUG::2016-01-15 10:03:00,254::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:03:00,285::health::86::health::(_check) Collected 57 objects Thread-15::WARNING::2016-01-15 10:03:00,294::health::100::health::(_check) Found 11162 uncollectible objects, 23 suspects implementing __del__: ['<pthreading.Lock object at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>', '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>', '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>', '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>', '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>', '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>', '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>', '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>'] vm was stopped here... Thread-15::DEBUG::2016-01-15 10:04:00,294::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:04:00,324::health::86::health::(_check) Collected 1755 objects Thread-15::WARNING::2016-01-15 10:04:00,335::health::100::health::(_check) Found 12915 uncollectible objects, 50 suspects implementing __del__: ['<pthreading.Lock object at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>', '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>', '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>', '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>', '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>', '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>', '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>', '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>', '<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>', '<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>', '<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>', '<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at 0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>', '<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at 0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>', '<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at 0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>', '<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at 0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>', '<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at 0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>', '<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>'] Skipping some logs, no activity in engine... Thread-15::DEBUG::2016-01-15 10:23:01,082::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:23:01,113::health::86::health::(_check) Collected 18 objects Thread-15::WARNING::2016-01-15 10:23:01,124::health::100::health::(_check) Found 13267 uncollectible objects, 50 suspects implementing __del__: ['<pthreading.Lock object at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>', '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>', '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>', '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>', '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>', '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>', '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>', '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>', '<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>', '<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>', '<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>', '<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at 0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>', '<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at 0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>', '<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at 0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>', '<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at 0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>', '<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at 0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>', '<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>'] Thread-15::DEBUG::2016-01-15 10:24:01,125::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:24:01,156::health::86::health::(_check) Collected 18 objects Thread-15::WARNING::2016-01-15 10:24:01,167::health::100::health::(_check) Found 13283 uncollectible objects, 50 suspects implementing __del__: ['<pthreading.Lock object at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>', '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>', '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>', '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>', '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>', '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>', '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>', '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>', '<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>', '<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>', '<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>', '<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at 0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>', '<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at 0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>', '<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at 0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>', '<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at 0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>', '<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at 0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>', '<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>'] vm was started here... Thread-15::DEBUG::2016-01-15 10:25:01,167::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:25:01,204::health::86::health::(_check) Collected 1537 objects Thread-15::WARNING::2016-01-15 10:25:01,219::health::100::health::(_check) Found 18264 uncollectible objects, 50 suspects implementing __del__: ['<pthreading.Lock object at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>', '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>', '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>', '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>', '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>', '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>', '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>', '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>', '<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>', '<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>', '<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>', '<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at 0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>', '<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at 0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>', '<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at 0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>', '<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at 0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>', '<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at 0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>', '<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>'] vm was ask to stop here... Thread-15::DEBUG::2016-01-15 10:26:01,219::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:26:01,255::health::86::health::(_check) Collected 57 objects Thread-15::WARNING::2016-01-15 10:26:01,270::health::100::health::(_check) Found 18319 uncollectible objects, 50 suspects implementing __del__: ['<pthreading.Lock object at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>', '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>', '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>', '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>', '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>', '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>', '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>', '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>', '<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>', '<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>', '<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>', '<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at 0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>', '<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at 0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>', '<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at 0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>', '<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at 0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>', '<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at 0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>', '<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>'] vm was shutdown here... Thread-15::DEBUG::2016-01-15 10:27:01,270::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:27:01,306::health::86::health::(_check) Collected 1752 objects Thread-15::WARNING::2016-01-15 10:27:01,322::health::100::health::(_check) Found 20069 uncollectible objects, 77 suspects implementing __del__: ['<pthreading.Lock object at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>', '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>', '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>', '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>', '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>', '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>', '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>', '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>', '<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>', '<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>', '<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>', '<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at 0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>', '<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at 0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>', '<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at 0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>', '<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at 0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>', '<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at 0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>', '<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>', '<pthreading.Lock object at 0x7fa904078710>', '<pthreading.Lock object at 0x7fa9b40af440>', '<pthreading.Lock object at 0x7fa90405b200>', '<pthreading.Lock object at 0x7fa904078998>', '<pthreading.Lock object at 0x7fa9a407cfc8>', '<pthreading.Lock object at 0x7fa904078638>', '<pthreading.Lock object at 0x7fa904078560>', '<pthreading.Lock object at 0x7fa904078200>', '<pthread.Cond object at 0x7fa9a4142d10>', '<pthreading.Lock object at 0x7fa9040785f0>', '<pthread.Cond object at 0x7fa9b40d1a50>', '<pthreading.Lock object at 0x7fa90405b9e0>', '<pthread.Cond object at 0x7fa9a4142610>', '<pthreading.Lock object at 0x7fa9040787a0>', '<pthread.Cond object at 0x7fa9b40d1f10>', '<pthreading.Lock object at 0x7fa9040786c8>', '<pthread.Cond object at 0x7fa9a4142b10>', '<pthreading.Lock object at 0x7fa904078368>', '<pthread.Cond object at 0x7fa9b40d1790>', '<pthreading.Lock object at 0x7fa904078290>', '<pthread.Cond object at 0x7fa9b40d14d0>', '<pthreading.Lock object at 0x7fa904078248>', '<pthread.Cond object at 0x7fa9b40d1150>', '<pthread.Cond object at 0x7fa9a41360d0>', '<pthreading.Lock object at 0x7fa9441a8d40>', '<pthreading.Lock object at 0x7fa944049b00>', '<pthread.Cond object at 0x7fa9a4136410>'] On Thu, Jan 14, 2016 at 8:00 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > Here are example logs with running and stopping one vm: > > # tail -f /var/log/vdsm/vdsm.log | grep health > Thread-13::DEBUG::2016-01-14 > 19:42:04,625::health::84::health::(_check) Checking health > Thread-13::DEBUG::2016-01-14 > 19:42:04,651::health::86::health::(_check) Collected 25 objects > Thread-13::WARNING::2016-01-14 > 19:42:04,656::health::100::health::(_check) Found 6085 uncollectible > objects, 0 suspects implementing __del__: [] > > Vm was started here... > > Thread-13::DEBUG::2016-01-14 > 19:43:04,657::health::84::health::(_check) Checking health > Thread-13::DEBUG::2016-01-14 > 19:43:04,688::health::86::health::(_check) Collected 1680 objects > Thread-13::WARNING::2016-01-14 > 19:43:04,697::health::100::health::(_check) Found 11041 uncollectible > objects, 0 suspects implementing __del__: [] > Thread-13::DEBUG::2016-01-14 > 19:44:04,698::health::84::health::(_check) Checking health > Thread-13::DEBUG::2016-01-14 > 19:44:04,729::health::86::health::(_check) Collected 57 objects > Thread-13::WARNING::2016-01-14 > 19:44:04,737::health::100::health::(_check) Found 11096 uncollectible > objects, 0 suspects implementing __del__: [] > Thread-13::DEBUG::2016-01-14 > 19:45:04,738::health::84::health::(_check) Checking health > Thread-13::DEBUG::2016-01-14 > 19:45:04,769::health::86::health::(_check) Collected 69 objects > Thread-13::WARNING::2016-01-14 > 19:45:04,778::health::100::health::(_check) Found 11163 uncollectible > objects, 0 suspects implementing __del__: [] > > Vm was shutdown here... > > Thread-13::DEBUG::2016-01-14 > 19:46:04,779::health::84::health::(_check) Checking health > Thread-13::DEBUG::2016-01-14 > 19:46:04,809::health::86::health::(_check) Collected 2272 objects > Thread-13::WARNING::2016-01-14 > 19:46:04,819::health::100::health::(_check) Found 13464 uncollectible > objects, 28 suspects implementing __del__: ['<pthreading.Lock object > at 0x7f1d8016f908>', '<pthreading.Lock object at 0x7f1d801804d0>', > '<pthreading.Lock object at 0x7f1cf8010f38>', '<pthreading.Lock object > at 0x7f1d8016f7a0>', '<pthreading.Lock object at 0x7f1d20003680>', > '<pthreading.Lock object at 0x7f1da8112998>', '<pthreading.Lock object > at 0x7f1d8016fa28>', '<pthreading.Lock object at 0x7f1da8112830>', > '<pthread.Cond object at 0x7f1da811cf50>', '<pthreading.Lock object at > 0x7f1d8016f200>', '<pthread.Cond object at 0x7f1da811ca50>', > '<pthreading.Lock object at 0x7f1cf8010fc8>', '<pthread.Cond object at > 0x7f1da8115c50>', '<pthreading.Lock object at 0x7f1da8112878>', > '<pthread.Cond object at 0x7f1da811c550>', '<pthreading.Lock object at > 0x7f1d8016f9e0>', '<pthread.Cond object at 0x7f1da811ca90>', > '<pthreading.Lock object at 0x7f1d8016f680>', '<pthread.Cond object at > 0x7f1da811cc50>', '<libvirt.virDomain object at 0x1ba8210>', > '<pthreading.Lock object at 0x7f1d8016f758>', '<pthread.Cond object at > 0x7f1da811cb50>', '<pthreading.Lock object at 0x7f1d8016fb90>', > '<pthread.Cond object at 0x7f1da811cc10>', '<pthread.Cond object at > 0x233ed50>', '<pthreading.Lock object at 0x7f1ce0227c20>', > '<pthreading.Lock object at 0x7f1ce0227ab8>', '<pthread.Cond object at > 0x233ed10>'] > > Need to test with should be fixed by https://gerrit.ovirt.org/51630, > which should > eliminate '<libvirt.virDomain object at 0x1ba8210>', and maybe also > the pthreading > objects. > > Nir > > On Thu, Jan 14, 2016 at 7:38 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >> Hi all, >> >> Continuing the leak investigation Francesco and Milan are working on, >> I posted this patch, adding leak health monitoring to vdsm [1]. This patch >> currently monitor collectible objects, and we may add other interesting >> stuff later if needed. >> >> To enable the monitor, you must set: >> >> [devel] >> health_monitor_enable = true >> >> In vdsm.conf, and restart vdsm. >> >> Here is an example logs - this was logged 60 seconds after starting vdsm, >> when running as spm: >> >> Thread-13::DEBUG::2016-01-14 >> 18:53:43,239::health::84::health::(_check) Checking health >> Thread-13::DEBUG::2016-01-14 >> 18:53:43,272::health::86::health::(_check) Collected 460 objects >> Thread-13::WARNING::2016-01-14 >> 18:53:43,277::health::100::health::(_check) Found 5873 uncollectible >> objects, 10 suspects implementing __del__: ['<pthreading.Lock object >> at 0x7f012c115320>', '<pthreading.Lock object at 0x7f012c115440>', >> '<pthreading.Lock object at 0x7f012c1153b0>', '<pthreading.Lock object >> at 0x7f012c1154d0>', '<pthread.Cond object at 0x7f012c111350>', >> '<pthread.Cond object at 0x7f012c1112d0>', '<pthread.Cond object at >> 0x7f012c111250>', '<pthread.Cond object at 0x7f012c111850>', >> '<pthread.Cond object at 0x7f012c1117d0>', '<pthread.Cond object at >> 0x7f012c111750>'] >> >> We found 460 uncollectible objects, 10 of them are objects implementing __del__. >> Having such object in a reference cycle make the entire cycle uncollectible. >> >> The pthreading objects in this log grow as vdsm continue to run, even without >> any interaction with engine. The pthreading objects leak is caused by a cycle >> in storage.misc.RWLock, fixed in patch [2]. >> >> After solving the pthreading leak, we still have many uncollectible >> objects in the logs, >> listing them, it seems that the suspect is hiding inside a collection >> (dict, list, tuple) >> and does not appear in the gc.garbage list. I will address this in later patch, >> searching in the entire object graph instead of the top level of gc.garbage. >> >> See end of this mail for list of these objects. I printed them like this: >> >> Enable manhole debugging shell: >> >> [devel] >> manhole_enable = true >> >> Connect to vdsm using manhole: >> >> nc -U /run/vdsm/vdsmd.manhole >> >> Print garbage: >> >>>>> import pprint >>>>> import gc >>>>> with open('/tmp/garbage.txt', 'w') as f: >> pprint.pprint(gc.garbage, stream=f, indent=4) >> >> These objects looks related to libvirt - Francesco, can you take a look? >> >> [1] https://gerrit.ovirt.org/51708 >> [2] https://gerrit.ovirt.org/51868 >> >> Nir >> >> --- garbage after patch [2] --- >> >> [ (<type '_ctypes.Array'>,), >> { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>, >> '__doc__': None, >> '__module__': 'vdsm.netlink', >> '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>, >> '_length_': 60, >> '_type_': <class 'ctypes.c_char'>, >> 'raw': <attribute 'raw' of 'c_char_Array_60' objects>, >> 'value': <attribute 'value' of 'c_char_Array_60' objects>}, >> <class 'vdsm.netlink.c_char_Array_60'>, >> <attribute '__dict__' of 'c_char_Array_60' objects>, >> <attribute '__weakref__' of 'c_char_Array_60' objects>, >> ( <class 'vdsm.netlink.c_char_Array_60'>, >> <type '_ctypes.Array'>, >> <type '_ctypes._CData'>, >> <type 'object'>), >> <attribute 'raw' of 'c_char_Array_60' objects>, >> <attribute 'value' of 'c_char_Array_60' objects>, >> <cell at 0x23662f0: bool object at 0x7f1e086c8c40>, >> <cell at 0x2366328: NoneType object at 0x7f1e086dcf00>, >> <cell at 0x2366360: function object at 0x233f488>, >> ( <cell at 0x23662f0: bool object at 0x7f1e086c8c40>, >> <cell at 0x2366328: NoneType object at 0x7f1e086dcf00>, >> <cell at 0x2366360: function object at 0x233f488>), >> <function wrapMethod at 0x233f488>, >> [ [ <Recursion on list with id=34033464>, >> <Recursion on list with id=34033464>, >> 'class'], >> [ <Recursion on list with id=34033464>, >> <Recursion on list with id=34033464>, >> 'class'], >> None], >> [ [ <Recursion on list with id=33979064>, >> <Recursion on list with id=33979064>, >> None], >> [ <Recursion on list with id=33979064>, >> <Recursion on list with id=33979064>, >> None], >> 'class'], >> [ [ <Recursion on list with id=35535256>, >> <Recursion on list with id=35535256>, >> 'class'], >> [ <Recursion on list with id=35535256>, >> <Recursion on list with id=35535256>, >> 'class'], >> None], >> [ [ <Recursion on list with id=35535832>, >> <Recursion on list with id=35535832>, >> None], >> [ <Recursion on list with id=35535832>, >> <Recursion on list with id=35535832>, >> None], >> 'class'], >> [ [ <Recursion on list with id=35556312>, >> <Recursion on list with id=35556312>, >> 'class'], >> [ <Recursion on list with id=35556312>, >> <Recursion on list with id=35556312>, >> 'class'], >> None], >> [ [ <Recursion on list with id=35556600>, >> <Recursion on list with id=35556600>, >> None], >> [ <Recursion on list with id=35556600>, >> <Recursion on list with id=35556600>, >> None], >> 'class'], >> [ [ <Recursion on list with id=35556960>, >> <Recursion on list with id=35556960>, >> 'class'], >> [ <Recursion on list with id=35556960>, >> <Recursion on list with id=35556960>, >> 'class'], >> None], >> [ [ <Recursion on list with id=35561912>, >> <Recursion on list with id=35561912>, >> None], >> [ <Recursion on list with id=35561912>, >> <Recursion on list with id=35561912>, >> None], >> 'class'], >> [ [ <Recursion on list with id=35583904>, >> <Recursion on list with id=35583904>, >> 'class'], >> [ <Recursion on list with id=35583904>, >> <Recursion on list with id=35583904>, >> 'class'], >> None], >> [ [ <Recursion on list with id=35584912>, >> <Recursion on list with id=35584912>, >> None], >> [ <Recursion on list with id=35584912>, >> <Recursion on list with id=35584912>, >> None], >> 'class'], >> [ [ <Recursion on list with id=36340368>, >> <Recursion on list with id=36340368>, >> 'class'], >> [ <Recursion on list with id=36340368>, >> <Recursion on list with id=36340368>, >> 'class'], >> None], >> [ [ <Recursion on list with id=36340872>, >> <Recursion on list with id=36340872>, >> None], >> [ <Recursion on list with id=36340872>, >> <Recursion on list with id=36340872>, >> None], >> 'class'], >> [ [ <Recursion on list with id=36364872>, >> <Recursion on list with id=36364872>, >> 'class'], >> [ <Recursion on list with id=36364872>, >> <Recursion on list with id=36364872>, >> 'class'], >> None], >> [ [ <Recursion on list with id=36365520>, >> <Recursion on list with id=36365520>, >> None], >> [ <Recursion on list with id=36365520>, >> <Recursion on list with id=36365520>, >> None], >> 'class'], >> [ [ <Recursion on list with id=36457216>, >> <Recursion on list with id=36457216>, >> 'class'], >> [ <Recursion on list with id=36457216>, >> <Recursion on list with id=36457216>, >> 'class'], >> None], >> [ [ <Recursion on list with id=36458008>, >> <Recursion on list with id=36458008>, >> None], >> [ <Recursion on list with id=36458008>, >> <Recursion on list with id=36458008>, >> None], >> 'class'], >> [ [ <Recursion on list with id=36460520>, >> <Recursion on list with id=36460520>, >> 'class'], >> [ <Recursion on list with id=36460520>, >> <Recursion on list with id=36460520>, >> 'class'], >> None], >> [ [ <Recursion on list with id=36461168>, >> <Recursion on list with id=36461168>, >> None], >> [ <Recursion on list with id=36461168>, >> <Recursion on list with id=36461168>, >> None], >> 'class'], >> [ [ <Recursion on list with id=36562344>, >> <Recursion on list with id=36562344>, >> 'class'], >> [ <Recursion on list with id=36562344>, >> <Recursion on list with id=36562344>, >> 'class'], >> None], >> [ [ <Recursion on list with id=36563280>, >> <Recursion on list with id=36563280>, >> None], >> [ <Recursion on list with id=36563280>, >> <Recursion on list with id=36563280>, >> None], >> 'class'], >> <Element 'memory' at 0x2364710>, >> { '_children': [], >> 'attrib': { 'unit': 'KiB'}, >> 'tag': 'memory', >> 'tail': '\n ', >> 'text': '4095476'}, >> [], >> <Element 'pages' at 0x2364750>, >> { '_children': [], >> 'attrib': { 'size': '4', 'unit': 'KiB'}, >> 'tag': 'pages', >> 'tail': '\n ', >> 'text': '1023869'}, >> [], >> <Element 'pages' at 0x2364790>, >> { '_children': [], >> 'attrib': { 'size': '2048', 'unit': 'KiB'}, >> 'tag': 'pages', >> 'tail': '\n ', >> 'text': '0'}, >> [], >> <Element 'distances' at 0x23647d0>, >> { '_children': [<Element 'sibling' at 0x2364810>], >> 'attrib': { }, >> 'tag': 'distances', >> 'tail': '\n ', >> 'text': '\n '}, >> [<Element 'sibling' at 0x2364810>], >> <Element 'sibling' at 0x2364810>, >> { '_children': [], >> 'attrib': { 'id': '0', 'value': '10'}, >> 'tag': 'sibling', >> 'tail': '\n '}, >> [], >> <Element 'cpus' at 0x2364850>, >> { '_children': [ <Element 'cpu' at 0x2364890>, >> <Element 'cpu' at 0x23648d0>, >> <Element 'cpu' at 0x2364910>, >> <Element 'cpu' at 0x2364950>, >> <Element 'cpu' at 0x2364990>, >> <Element 'cpu' at 0x23649d0>, >> <Element 'cpu' at 0x2364a10>, >> <Element 'cpu' at 0x2364a50>], >> 'attrib': { 'num': '8'}, >> 'tag': 'cpus', >> 'tail': '\n ', >> 'text': '\n '}, >> [ <Element 'cpu' at 0x2364890>, >> <Element 'cpu' at 0x23648d0>, >> <Element 'cpu' at 0x2364910>, >> <Element 'cpu' at 0x2364950>, >> <Element 'cpu' at 0x2364990>, >> <Element 'cpu' at 0x23649d0>, >> <Element 'cpu' at 0x2364a10>, >> <Element 'cpu' at 0x2364a50>], >> <Element 'cpu' at 0x2364890>, >> { '_children': [], >> 'attrib': { 'core_id': '0', >> 'id': '0', >> 'siblings': '0', >> 'socket_id': '0'}, >> 'tag': 'cpu', >> 'tail': '\n '}, >> [], >> <Element 'cpu' at 0x23648d0>, >> { '_children': [], >> 'attrib': { 'core_id': '0', >> 'id': '1', >> 'siblings': '1', >> 'socket_id': '1'}, >> 'tag': 'cpu', >> 'tail': '\n '}, >> [], >> <Element 'cpu' at 0x2364910>, >> { '_children': [], >> 'attrib': { 'core_id': '0', >> 'id': '2', >> 'siblings': '2', >> 'socket_id': '2'}, >> 'tag': 'cpu', >> 'tail': '\n '}, >> [], >> <Element 'cpu' at 0x2364950>, >> { '_children': [], >> 'attrib': { 'core_id': '0', >> 'id': '3', >> 'siblings': '3', >> 'socket_id': '3'}, >> 'tag': 'cpu', >> 'tail': '\n '}, >> [], >> <Element 'cpu' at 0x2364990>, >> { '_children': [], >> 'attrib': { 'core_id': '0', >> 'id': '4', >> 'siblings': '4', >> 'socket_id': '4'}, >> 'tag': 'cpu', >> 'tail': '\n '}, >> [], >> <Element 'cpu' at 0x23649d0>, >> { '_children': [], >> 'attrib': { 'core_id': '0', >> 'id': '5', >> 'siblings': '5', >> 'socket_id': '5'}, >> 'tag': 'cpu', >> 'tail': '\n '}, >> [], >> <Element 'cpu' at 0x2364a10>, >> { '_children': [], >> 'attrib': { 'core_id': '0', >> 'id': '6', >> 'siblings': '6', >> 'socket_id': '6'}, >> 'tag': 'cpu', >> 'tail': '\n '}, >> [], >> <Element 'cpu' at 0x2364a50>, >> { '_children': [], >> 'attrib': { 'core_id': '0', >> 'id': '7', >> 'siblings': '7', >> 'socket_id': '7'}, >> 'tag': 'cpu', >> 'tail': '\n '}, >> [], >> <Element 'secmodel' at 0x2364a90>, >> { '_children': [ <Element 'model' at 0x2364ad0>, >> <Element 'doi' at 0x2364b10>, >> <Element 'baselabel' at 0x2364b50>, >> <Element 'baselabel' at 0x2364b90>], >> 'attrib': { }, >> 'tag': 'secmodel', >> 'tail': '\n ', >> 'text': '\n '}, >> [ <Element 'model' at 0x2364ad0>, >> <Element 'doi' at 0x2364b10>, >> <Element 'baselabel' at 0x2364b50>, >> <Element 'baselabel' at 0x2364b90>], >> <Element 'model' at 0x2364ad0>, >> { '_children': [], >> 'attrib': { }, >> 'tag': 'model', >> 'tail': '\n ', >> 'text': 'selinux'}, >> [], >> <Element 'doi' at 0x2364b10>, >> { '_children': [], >> 'attrib': { }, >> 'tag': 'doi', >> 'tail': '\n ', >> 'text': '0'}, >> [], >> <Element 'baselabel' at 0x2364b50>, >> { '_children': [], >> 'attrib': { 'type': 'kvm'}, >> 'tag': 'baselabel', >> 'tail': '\n ', >> 'text': 'system_u:system_r:svirt_t:s0'}, >> [], >> <Element 'baselabel' at 0x2364b90>, >> { '_children': [], >> 'attrib': { 'type': 'qemu'}, >> 'tag': 'baselabel', >> 'tail': '\n ', >> 'text': 'system_u:system_r:svirt_tcg_t:s0'}, >> [], >> <Element 'secmodel' at 0x2364bd0>, >> { '_children': [ <Element 'model' at 0x2364c10>, >> <Element 'doi' at 0x2364c50>, >> <Element 'baselabel' at 0x2364c90>, >> <Element 'baselabel' at 0x2364cd0>], >> 'attrib': { }, >> 'tag': 'secmodel', >> 'tail': '\n ', >> 'text': '\n '}, >> [ <Element 'model' at 0x2364c10>, >> <Element 'doi' at 0x2364c50>, >> <Element 'baselabel' at 0x2364c90>, >> <Element 'baselabel' at 0x2364cd0>], >> <Element 'model' at 0x2364c10>, >> { '_children': [], >> 'attrib': { }, >> 'tag': 'model', >> 'tail': '\n ', >> 'text': 'dac'}, >> [], >> <Element 'doi' at 0x2364c50>, >> { '_children': [], >> 'attrib': { }, >> 'tag': 'doi', >> 'tail': '\n ', >> 'text': '0'}, >> [], >> <Element 'baselabel' at 0x2364c90>, >> { '_children': [], >> 'attrib': { 'type': 'kvm'}, >> 'tag': 'baselabel', >> 'tail': '\n ', >> 'text': '+107:+107'}, >> [], >> <Element 'baselabel' at 0x2364cd0>, >> { '_children': [], >> 'attrib': { 'type': 'qemu'}, >> 'tag': 'baselabel', >> 'tail': '\n ', >> 'text': '+107:+107'}, >> [], >> <Element 'guest' at 0x2364d10>, >> { '_children': [ <Element 'os_type' at 0x2364d50>, >> <Element 'arch' at 0x2364d90>, >> <Element 'features' at 0x2369390>], >> 'attrib': { }, >> 'tag': 'guest', >> 'tail': '\n\n ', >> 'text': '\n '}, >> [ <Element 'os_type' at 0x2364d50>, >> <Element 'arch' at 0x2364d90>, >> <Element 'features' at 0x2369390>], >> <Element 'os_type' at 0x2364d50>, >> { '_children': [], >> 'attrib': { }, >> 'tag': 'os_type', >> 'tail': '\n ', >> 'text': 'hvm'}, >> [], >> <Element 'arch' at 0x2364d90>, >> { '_children': [ <Element 'wordsize' at 0x2364dd0>, >> <Element 'emulator' at 0x2364e10>, >> <Element 'machine' at 0x2364e90>, >> <Element 'machine' at 0x2364ed0>, >> <Element 'machine' at 0x2364f10>, >> <Element 'machine' at 0x2364f50>, >> <Element 'machine' at 0x2364f90>, >> <Element 'machine' at 0x2364fd0>, >> <Element 'machine' at 0x2369050>, >> <Element 'machine' at 0x2369090>, >> <Element 'machine' at 0x23690d0>, >> <Element 'machine' at 0x2369110>, >> <Element 'machine' at 0x2369150>, >> <Element 'machine' at 0x2369190>, >> <Element 'machine' at 0x23691d0>, >> <Element 'machine' at 0x2369210>, >> <Element 'machine' at 0x2369250>, >> <Element 'domain' at 0x2369290>, >> <Element 'domain' at 0x23692d0>], >> 'attrib': { 'name': 'i686'}, >> 'tag': 'arch', >> 'tail': '\n ', >> 'text': '\n '}, >> [ <Element 'wordsize' at 0x2364dd0>, >> <Element 'emulator' at 0x2364e10>, >> <Element 'machine' at 0x2364e90>, >> <Element 'machine' at 0x2364ed0>, >> <Element 'machine' at 0x2364f10>, >> <Element 'machine' at 0x2364f50>, >> <Element 'machine' at 0x2364f90>, >> <Element 'machine' at 0x2364fd0>, >> <Element 'machine' at 0x2369050>, >> <Element 'machine' at 0x2369090>, >> <Element 'machine' at 0x23690d0>, >> <Element 'machine' at 0x2369110>, >> <Element 'machine' at 0x2369150>, >> <Element 'machine' at 0x2369190>, >> <Element 'machine' at 0x23691d0>, >> <Element 'machine' at 0x2369210>, >> <Element 'machine' at 0x2369250>, >> <Element 'domain' at 0x2369290>, >> <Element 'domain' at 0x23692d0>], >> <Element 'wordsize' at 0x2364dd0>, >> { '_children': [], >> 'attrib': { }, >> 'tag': 'wordsize', >> 'tail': '\n ', >> 'text': '32'}, >> [], >> <Element 'emulator' at 0x2364e10>, >> { '_children': [], >> 'attrib': { }, >> 'tag': 'emulator', >> 'tail': '\n ', >> 'text': '/usr/libexec/qemu-kvm'}, >> [], >> <Element 'machine' at 0x2364e90>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'pc-i440fx-rhel7.2.0'}, >> [], >> <Element 'machine' at 0x2364ed0>, >> { '_children': [], >> 'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'pc'}, >> [], >> <Element 'machine' at 0x2364f10>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'pc-i440fx-rhel7.0.0'}, >> [], >> <Element 'machine' at 0x2364f50>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'pc-q35-rhel7.1.0'}, >> [], >> <Element 'machine' at 0x2364f90>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'rhel6.3.0'}, >> [], >> <Element 'machine' at 0x2364fd0>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'pc-q35-rhel7.2.0'}, >> [], >> <Element 'machine' at 0x2369050>, >> { '_children': [], >> 'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'q35'}, >> [], >> <Element 'machine' at 0x2369090>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'rhel6.4.0'}, >> [], >> <Element 'machine' at 0x23690d0>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'rhel6.0.0'}, >> [], >> <Element 'machine' at 0x2369110>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'pc-i440fx-rhel7.1.0'}, >> [], >> <Element 'machine' at 0x2369150>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'rhel6.5.0'}, >> [], >> <Element 'machine' at 0x2369190>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'rhel6.6.0'}, >> [], >> <Element 'machine' at 0x23691d0>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'rhel6.1.0'}, >> [], >> <Element 'machine' at 0x2369210>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'pc-q35-rhel7.0.0'}, >> [], >> <Element 'machine' at 0x2369250>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'rhel6.2.0'}, >> [], >> <Element 'domain' at 0x2369290>, >> { '_children': [], >> 'attrib': { 'type': 'qemu'}, >> 'tag': 'domain', >> 'tail': '\n '}, >> [], >> <Element 'domain' at 0x23692d0>, >> { '_children': [<Element 'emulator' at 0x2369310>], >> 'attrib': { 'type': 'kvm'}, >> 'tag': 'domain', >> 'tail': '\n ', >> 'text': '\n '}, >> [<Element 'emulator' at 0x2369310>], >> <Element 'emulator' at 0x2369310>, >> { '_children': [], >> 'attrib': { }, >> 'tag': 'emulator', >> 'tail': '\n ', >> 'text': '/usr/libexec/qemu-kvm'}, >> [], >> <Element 'features' at 0x2369390>, >> { '_children': [ <Element 'cpuselection' at 0x23693d0>, >> <Element 'deviceboot' at 0x2369410>, >> <Element 'disksnapshot' at 0x2369450>, >> <Element 'acpi' at 0x2369490>, >> <Element 'apic' at 0x23694d0>, >> <Element 'pae' at 0x2369510>, >> <Element 'nonpae' at 0x2369550>], >> 'attrib': { }, >> 'tag': 'features', >> 'tail': '\n ', >> 'text': '\n '}, >> [ <Element 'cpuselection' at 0x23693d0>, >> <Element 'deviceboot' at 0x2369410>, >> <Element 'disksnapshot' at 0x2369450>, >> <Element 'acpi' at 0x2369490>, >> <Element 'apic' at 0x23694d0>, >> <Element 'pae' at 0x2369510>, >> <Element 'nonpae' at 0x2369550>], >> <Element 'cpuselection' at 0x23693d0>, >> { '_children': [], 'attrib': { }, 'tag': 'cpuselection', >> 'tail': '\n '}, >> [], >> <Element 'deviceboot' at 0x2369410>, >> { '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail': >> '\n '}, >> [], >> <Element 'disksnapshot' at 0x2369450>, >> { '_children': [], >> 'attrib': { 'default': 'on', 'toggle': 'no'}, >> 'tag': 'disksnapshot', >> 'tail': '\n '}, >> [], >> <Element 'acpi' at 0x2369490>, >> { '_children': [], >> 'attrib': { 'default': 'on', 'toggle': 'yes'}, >> 'tag': 'acpi', >> 'tail': '\n '}, >> [], >> <Element 'apic' at 0x23694d0>, >> { '_children': [], >> 'attrib': { 'default': 'on', 'toggle': 'no'}, >> 'tag': 'apic', >> 'tail': '\n '}, >> [], >> <Element 'pae' at 0x2369510>, >> { '_children': [], 'attrib': { }, 'tag': 'pae', 'tail': '\n '}, >> [], >> <Element 'nonpae' at 0x2369550>, >> { '_children': [], 'attrib': { }, 'tag': 'nonpae', 'tail': '\n '}, >> [], >> <Element 'guest' at 0x2369590>, >> { '_children': [ <Element 'os_type' at 0x23695d0>, >> <Element 'arch' at 0x2369610>, >> <Element 'features' at 0x2369bd0>], >> 'attrib': { }, >> 'tag': 'guest', >> 'tail': '\n\n', >> 'text': '\n '}, >> [ <Element 'os_type' at 0x23695d0>, >> <Element 'arch' at 0x2369610>, >> <Element 'features' at 0x2369bd0>], >> <Element 'os_type' at 0x23695d0>, >> { '_children': [], >> 'attrib': { }, >> 'tag': 'os_type', >> 'tail': '\n ', >> 'text': 'hvm'}, >> [], >> <Element 'arch' at 0x2369610>, >> { '_children': [ <Element 'wordsize' at 0x2369650>, >> <Element 'emulator' at 0x2369690>, >> <Element 'machine' at 0x2369710>, >> <Element 'machine' at 0x2369750>, >> <Element 'machine' at 0x2369790>, >> <Element 'machine' at 0x23697d0>, >> <Element 'machine' at 0x2369810>, >> <Element 'machine' at 0x2369850>, >> <Element 'machine' at 0x2369890>, >> <Element 'machine' at 0x23698d0>, >> <Element 'machine' at 0x2369910>, >> <Element 'machine' at 0x2369950>, >> <Element 'machine' at 0x2369990>, >> <Element 'machine' at 0x23699d0>, >> <Element 'machine' at 0x2369a10>, >> <Element 'machine' at 0x2369a50>, >> <Element 'machine' at 0x2369a90>, >> <Element 'domain' at 0x2369ad0>, >> <Element 'domain' at 0x2369b10>], >> 'attrib': { 'name': 'x86_64'}, >> 'tag': 'arch', >> 'tail': '\n ', >> 'text': '\n '}, >> [ <Element 'wordsize' at 0x2369650>, >> <Element 'emulator' at 0x2369690>, >> <Element 'machine' at 0x2369710>, >> <Element 'machine' at 0x2369750>, >> <Element 'machine' at 0x2369790>, >> <Element 'machine' at 0x23697d0>, >> <Element 'machine' at 0x2369810>, >> <Element 'machine' at 0x2369850>, >> <Element 'machine' at 0x2369890>, >> <Element 'machine' at 0x23698d0>, >> <Element 'machine' at 0x2369910>, >> <Element 'machine' at 0x2369950>, >> <Element 'machine' at 0x2369990>, >> <Element 'machine' at 0x23699d0>, >> <Element 'machine' at 0x2369a10>, >> <Element 'machine' at 0x2369a50>, >> <Element 'machine' at 0x2369a90>, >> <Element 'domain' at 0x2369ad0>, >> <Element 'domain' at 0x2369b10>], >> <Element 'wordsize' at 0x2369650>, >> { '_children': [], >> 'attrib': { }, >> 'tag': 'wordsize', >> 'tail': '\n ', >> 'text': '64'}, >> [], >> <Element 'emulator' at 0x2369690>, >> { '_children': [], >> 'attrib': { }, >> 'tag': 'emulator', >> 'tail': '\n ', >> 'text': '/usr/libexec/qemu-kvm'}, >> [], >> <Element 'machine' at 0x2369710>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'pc-i440fx-rhel7.2.0'}, >> [], >> <Element 'machine' at 0x2369750>, >> { '_children': [], >> 'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'pc'}, >> [], >> <Element 'machine' at 0x2369790>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'pc-i440fx-rhel7.0.0'}, >> [], >> <Element 'machine' at 0x23697d0>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'pc-q35-rhel7.1.0'}, >> [], >> <Element 'machine' at 0x2369810>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'rhel6.3.0'}, >> [], >> <Element 'machine' at 0x2369850>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'pc-q35-rhel7.2.0'}, >> [], >> <Element 'machine' at 0x2369890>, >> { '_children': [], >> 'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'q35'}, >> [], >> <Element 'machine' at 0x23698d0>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'rhel6.4.0'}, >> [], >> <Element 'machine' at 0x2369910>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'rhel6.0.0'}, >> [], >> <Element 'machine' at 0x2369950>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'pc-i440fx-rhel7.1.0'}, >> [], >> <Element 'machine' at 0x2369990>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'rhel6.5.0'}, >> [], >> <Element 'machine' at 0x23699d0>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'rhel6.6.0'}, >> [], >> <Element 'machine' at 0x2369a10>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'rhel6.1.0'}, >> [], >> <Element 'machine' at 0x2369a50>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'pc-q35-rhel7.0.0'}, >> [], >> <Element 'machine' at 0x2369a90>, >> { '_children': [], >> 'attrib': { 'maxCpus': '240'}, >> 'tag': 'machine', >> 'tail': '\n ', >> 'text': 'rhel6.2.0'}, >> [], >> <Element 'domain' at 0x2369ad0>, >> { '_children': [], >> 'attrib': { 'type': 'qemu'}, >> 'tag': 'domain', >> 'tail': '\n '}, >> [], >> <Element 'domain' at 0x2369b10>, >> { '_children': [<Element 'emulator' at 0x2369b50>], >> 'attrib': { 'type': 'kvm'}, >> 'tag': 'domain', >> 'tail': '\n ', >> 'text': '\n '}, >> [<Element 'emulator' at 0x2369b50>], >> <Element 'emulator' at 0x2369b50>, >> { '_children': [], >> 'attrib': { }, >> 'tag': 'emulator', >> 'tail': '\n ', >> 'text': '/usr/libexec/qemu-kvm'}, >> [], >> <Element 'features' at 0x2369bd0>, >> { '_children': [ <Element 'cpuselection' at 0x2369c10>, >> <Element 'deviceboot' at 0x2369c50>, >> <Element 'disksnapshot' at 0x2369c90>, >> <Element 'acpi' at 0x2369cd0>, >> <Element 'apic' at 0x2369d10>], >> 'attrib': { }, >> 'tag': 'features', >> 'tail': '\n ', >> 'text': '\n '}, >> [ <Element 'cpuselection' at 0x2369c10>, >> <Element 'deviceboot' at 0x2369c50>, >> <Element 'disksnapshot' at 0x2369c90>, >> <Element 'acpi' at 0x2369cd0>, >> <Element 'apic' at 0x2369d10>], >> <Element 'cpuselection' at 0x2369c10>, >> { '_children': [], 'attrib': { }, 'tag': 'cpuselection', >> 'tail': '\n '}, >> [], >> <Element 'deviceboot' at 0x2369c50>, >> { '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail': >> '\n '}, >> [], >> <Element 'disksnapshot' at 0x2369c90>, >> { '_children': [], >> 'attrib': { 'default': 'on', 'toggle': 'no'}, >> 'tag': 'disksnapshot', >> 'tail': '\n '}, >> [], >> <Element 'acpi' at 0x2369cd0>, >> { '_children': [], >> 'attrib': { 'default': 'on', 'toggle': 'yes'}, >> 'tag': 'acpi', >> 'tail': '\n '}, >> [], >> <Element 'apic' at 0x2369d10>, >> { '_children': [], >> 'attrib': { 'default': 'on', 'toggle': 'no'}, >> 'tag': 'apic', >> 'tail': '\n '}, >> [], >> [], >> (<type '_ctypes.Array'>,), >> { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>, >> '__doc__': None, >> '__module__': 'vdsm.netlink', >> '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>, >> '_length_': 60, >> '_type_': <class 'ctypes.c_char'>, >> 'raw': <attribute 'raw' of 'c_char_Array_60' objects>, >> 'value': <attribute 'value' of 'c_char_Array_60' objects>}, >> <class 'vdsm.netlink.c_char_Array_60'>, >> <attribute '__dict__' of 'c_char_Array_60' objects>, >> <attribute '__weakref__' of 'c_char_Array_60' objects>, >> ( <class 'vdsm.netlink.c_char_Array_60'>, >> <type '_ctypes.Array'>, >> <type '_ctypes._CData'>, >> <type 'object'>), >> <attribute 'raw' of 'c_char_Array_60' objects>, >> <attribute 'value' of 'c_char_Array_60' objects>, >> <xml.etree.ElementTree.XMLParser object at 0x2357f50>, >> { '_doctype': None, >> '_error': <class 'xml.parsers.expat.ExpatError'>, >> '_names': { u'acpi': 'acpi', >> u'apic': 'apic', >> u'arch': 'arch', >> u'baselabel': 'baselabel', >> u'canonical': 'canonical', >> u'capabilities': 'capabilities', >> u'cell': 'cell', >> u'cells': 'cells', >> u'core_id': 'core_id', >> u'cores': 'cores', >> u'cpu': 'cpu', >> u'cpus': 'cpus', >> u'cpuselection': 'cpuselection', >> u'default': 'default', >> u'deviceboot': 'deviceboot', >> u'disksnapshot': 'disksnapshot', >> u'distances': 'distances', >> u'doi': 'doi', >> u'domain': 'domain', >> u'emulator': 'emulator', >> u'feature': 'feature', >> u'features': 'features', >> u'guest': 'guest', >> u'host': 'host', >> u'id': 'id', >> u'live': 'live', >> u'machine': 'machine', >> u'maxCpus': 'maxCpus', >> u'memory': 'memory', >> u'migration_features': 'migration_features', >> u'model': 'model', >> u'name': 'name', >> u'nonpae': 'nonpae', >> u'num': 'num', >> u'os_type': 'os_type', >> u'pae': 'pae', >> u'pages': 'pages', >> u'power_management': 'power_management', >> u'secmodel': 'secmodel', >> u'sibling': 'sibling', >> u'siblings': 'siblings', >> u'size': 'size', >> u'socket_id': 'socket_id', >> u'sockets': 'sockets', >> u'suspend_disk': 'suspend_disk', >> u'suspend_hybrid': 'suspend_hybrid', >> u'suspend_mem': 'suspend_mem', >> u'threads': 'threads', >> u'toggle': 'toggle', >> u'topology': 'topology', >> u'type': 'type', >> u'unit': 'unit', >> u'uri_transport': 'uri_transport', >> u'uri_transports': 'uri_transports', >> u'uuid': 'uuid', >> u'value': 'value', >> u'vendor': 'vendor', >> u'wordsize': 'wordsize'}, >> '_target': <xml.etree.ElementTree.TreeBuilder object at 0x2357f10>, >> 'entity': { }, >> 'parser': <pyexpat.xmlparser object at 0x22e5360>, >> 'version': 'Expat 2.1.0'}, >> <pyexpat.xmlparser object at 0x22e5360>, >> <xml.etree.ElementTree.TreeBuilder object at 0x2357f10>, >> <bound method XMLParser._start_list of >> <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, >> <bound method XMLParser._end of <xml.etree.ElementTree.XMLParser >> object at 0x2357f50>>, >> <bound method XMLParser._pi of <xml.etree.ElementTree.XMLParser >> object at 0x2357f50>>, >> <bound method XMLParser._data of <xml.etree.ElementTree.XMLParser >> object at 0x2357f50>>, >> <bound method XMLParser._comment of >> <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, >> <bound method XMLParser._default of >> <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, >> { '_data': [], >> '_elem': [], >> '_factory': <class 'xml.etree.ElementTree.Element'>, >> '_last': <Element 'capabilities' at 0x2357f90>, >> '_tail': 1}, >> [], >> <Element 'cell' at 0x23646d0>, >> <Element 'cells' at 0x2364690>, >> <Element 'topology' at 0x2364650>, >> <Element 'host' at 0x2357fd0>, >> <Element 'capabilities' at 0x2357f90>, >> { '_children': [ <Element 'memory' at 0x2364710>, >> <Element 'pages' at 0x2364750>, >> <Element 'pages' at 0x2364790>, >> <Element 'distances' at 0x23647d0>, >> <Element 'cpus' at 0x2364850>], >> 'attrib': { 'id': '0'}, >> 'tag': 'cell', >> 'tail': '\n ', >> 'text': '\n '}, >> { '_children': [<Element 'cell' at 0x23646d0>], >> 'attrib': { 'num': '1'}, >> 'tag': 'cells', >> 'tail': '\n ', >> 'text': '\n '}, >> { '_children': [<Element 'cells' at 0x2364690>], >> 'attrib': { }, >> 'tag': 'topology', >> 'tail': '\n ', >> 'text': '\n '}, >> { '_children': [ <Element 'uuid' at 0x2364050>, >> <Element 'cpu' at 0x2364090>, >> <Element 'power_management' at 0x2364390>, >> <Element 'migration_features' at 0x23644d0>, >> <Element 'topology' at 0x2364650>, >> <Element 'secmodel' at 0x2364a90>, >> <Element 'secmodel' at 0x2364bd0>], >> 'attrib': { }, >> 'tag': 'host', >> 'tail': '\n\n ', >> 'text': '\n '}, >> { '_children': [ <Element 'host' at 0x2357fd0>, >> <Element 'guest' at 0x2364d10>, >> <Element 'guest' at 0x2369590>], >> 'attrib': { }, >> 'tag': 'capabilities', >> 'text': '\n\n '}, >> [ <Element 'memory' at 0x2364710>, >> <Element 'pages' at 0x2364750>, >> <Element 'pages' at 0x2364790>, >> <Element 'distances' at 0x23647d0>, >> <Element 'cpus' at 0x2364850>], >> [<Element 'cell' at 0x23646d0>], >> [<Element 'cells' at 0x2364690>], >> [ <Element 'uuid' at 0x2364050>, >> <Element 'cpu' at 0x2364090>, >> <Element 'power_management' at 0x2364390>, >> <Element 'migration_features' at 0x23644d0>, >> <Element 'topology' at 0x2364650>, >> <Element 'secmodel' at 0x2364a90>, >> <Element 'secmodel' at 0x2364bd0>], >> [ <Element 'host' at 0x2357fd0>, >> <Element 'guest' at 0x2364d10>, >> <Element 'guest' at 0x2369590>], >> <Element 'migration_features' at 0x23644d0>, >> <Element 'power_management' at 0x2364390>, >> <Element 'cpu' at 0x2364090>, >> <Element 'uuid' at 0x2364050>, >> { '_children': [ <Element 'live' at 0x2364510>, >> <Element 'uri_transports' at 0x2364590>], >> 'attrib': { }, >> 'tag': 'migration_features', >> 'tail': '\n ', >> 'text': '\n '}, >> { '_children': [ <Element 'suspend_mem' at 0x23643d0>, >> <Element 'suspend_disk' at 0x2364410>, >> <Element 'suspend_hybrid' at 0x2364490>], >> 'attrib': { }, >> 'tag': 'power_management', >> 'tail': '\n ', >> 'text': '\n '}, >> { '_children': [ <Element 'arch' at 0x23640d0>, >> <Element 'model' at 0x2364110>, >> <Element 'vendor' at 0x2364150>, >> <Element 'topology' at 0x2364190>, >> <Element 'feature' at 0x23641d0>, >> <Element 'feature' at 0x2364210>, >> <Element 'feature' at 0x2364250>, >> <Element 'feature' at 0x2364290>, >> <Element 'feature' at 0x23642d0>, >> <Element 'pages' at 0x2364310>, >> <Element 'pages' at 0x2364350>], >> 'attrib': { }, >> 'tag': 'cpu', >> 'tail': '\n ', >> 'text': '\n '}, >> { '_children': [], >> 'attrib': { }, >> 'tag': 'uuid', >> 'tail': '\n ', >> 'text': 'a920d963-c18f-4fda-aa5a-deee9bbcde9d'}, >> [<Element 'live' at 0x2364510>, <Element 'uri_transports' at 0x2364590>], >> [ <Element 'suspend_mem' at 0x23643d0>, >> <Element 'suspend_disk' at 0x2364410>, >> <Element 'suspend_hybrid' at 0x2364490>], >> [ <Element 'arch' at 0x23640d0>, >> <Element 'model' at 0x2364110>, >> <Element 'vendor' at 0x2364150>, >> <Element 'topology' at 0x2364190>, >> <Element 'feature' at 0x23641d0>, >> <Element 'feature' at 0x2364210>, >> <Element 'feature' at 0x2364250>, >> <Element 'feature' at 0x2364290>, >> <Element 'feature' at 0x23642d0>, >> <Element 'pages' at 0x2364310>, >> <Element 'pages' at 0x2364350>], >> [], >> <Element 'uri_transports' at 0x2364590>, >> <Element 'live' at 0x2364510>, >> <Element 'suspend_hybrid' at 0x2364490>, >> <Element 'suspend_disk' at 0x2364410>, >> <Element 'suspend_mem' at 0x23643d0>, >> <Element 'pages' at 0x2364350>, >> <Element 'pages' at 0x2364310>, >> <Element 'feature' at 0x23642d0>, >> <Element 'feature' at 0x2364290>, >> <Element 'feature' at 0x2364250>, >> <Element 'feature' at 0x2364210>, >> <Element 'feature' at 0x23641d0>, >> <Element 'topology' at 0x2364190>, >> <Element 'vendor' at 0x2364150>, >> <Element 'model' at 0x2364110>, >> <Element 'arch' at 0x23640d0>, >> { '_children': [ <Element 'uri_transport' at 0x23645d0>, >> <Element 'uri_transport' at 0x2364610>], >> 'attrib': { }, >> 'tag': 'uri_transports', >> 'tail': '\n ', >> 'text': '\n '}, >> { '_children': [], 'attrib': { }, 'tag': 'live', 'tail': '\n '}, >> { '_children': [], 'attrib': { }, 'tag': 'suspend_hybrid', >> 'tail': '\n '}, >> { '_children': [], 'attrib': { }, 'tag': 'suspend_disk', >> 'tail': '\n '}, >> { '_children': [], 'attrib': { }, 'tag': 'suspend_mem', >> 'tail': '\n '}, >> { '_children': [], >> 'attrib': { 'size': '2048', 'unit': 'KiB'}, >> 'tag': 'pages', >> 'tail': '\n '}, >> { '_children': [], >> 'attrib': { 'size': '4', 'unit': 'KiB'}, >> 'tag': 'pages', >> 'tail': '\n '}, >> { '_children': [], >> 'attrib': { 'name': 'ss'}, >> 'tag': 'feature', >> 'tail': '\n '}, >> { '_children': [], >> 'attrib': { 'name': 'vmx'}, >> 'tag': 'feature', >> 'tail': '\n '}, >> { '_children': [], >> 'attrib': { 'name': 'pcid'}, >> 'tag': 'feature', >> 'tail': '\n '}, >> { '_children': [], >> 'attrib': { 'name': 'osxsave'}, >> 'tag': 'feature', >> 'tail': '\n '}, >> { '_children': [], >> 'attrib': { 'name': 'hypervisor'}, >> 'tag': 'feature', >> 'tail': '\n '}, >> { '_children': [], >> 'attrib': { 'cores': '1', 'sockets': '8', 'threads': '1'}, >> 'tag': 'topology', >> 'tail': '\n '}, >> { '_children': [], >> 'attrib': { }, >> 'tag': 'vendor', >> 'tail': '\n ', >> 'text': 'Intel'}, >> { '_children': [], >> 'attrib': { }, >> 'tag': 'model', >> 'tail': '\n ', >> 'text': 'IvyBridge'}, >> { '_children': [], >> 'attrib': { }, >> 'tag': 'arch', >> 'tail': '\n ', >> 'text': 'x86_64'}, >> [ <Element 'uri_transport' at 0x23645d0>, >> <Element 'uri_transport' at 0x2364610>], >> [], >> [], >> [], >> [], >> [], >> [], >> [], >> [], >> [], >> [], >> [], >> [], >> [], >> [], >> [], >> <Element 'uri_transport' at 0x2364610>, >> <Element 'uri_transport' at 0x23645d0>, >> { '_children': [], >> 'attrib': { }, >> 'tag': 'uri_transport', >> 'tail': '\n ', >> 'text': 'rdma'}, >> { '_children': [], >> 'attrib': { }, >> 'tag': 'uri_transport', >> 'tail': '\n ', >> 'text': 'tcp'}, >> [], >> [], >> (<type '_ctypes.Array'>,), >> { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>, >> '__doc__': None, >> '__module__': 'vdsm.netlink', >> '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>, >> '_length_': 60, >> '_type_': <class 'ctypes.c_char'>, >> 'raw': <attribute 'raw' of 'c_char_Array_60' objects>, >> 'value': <attribute 'value' of 'c_char_Array_60' objects>}, >> <class 'vdsm.netlink.c_char_Array_60'>, >> <attribute '__dict__' of 'c_char_Array_60' objects>, >> <attribute '__weakref__' of 'c_char_Array_60' objects>, >> ( <class 'vdsm.netlink.c_char_Array_60'>, >> <type '_ctypes.Array'>, >> <type '_ctypes._CData'>, >> <type 'object'>), >> <attribute 'raw' of 'c_char_Array_60' objects>, >> <attribute 'value' of 'c_char_Array_60' objects>]