[ovirt-devel] [VDSM] health: Introduce Vdsm health monitoring
Nir Soffer
nsoffer at redhat.com
Thu Jan 14 17:38:36 UTC 2016
Hi all,
Continuing the leak investigation Francesco and Milan are working on,
I posted this patch, adding leak health monitoring to vdsm [1]. This patch
currently monitor collectible objects, and we may add other interesting
stuff later if needed.
To enable the monitor, you must set:
[devel]
health_monitor_enable = true
In vdsm.conf, and restart vdsm.
Here is an example logs - this was logged 60 seconds after starting vdsm,
when running as spm:
Thread-13::DEBUG::2016-01-14
18:53:43,239::health::84::health::(_check) Checking health
Thread-13::DEBUG::2016-01-14
18:53:43,272::health::86::health::(_check) Collected 460 objects
Thread-13::WARNING::2016-01-14
18:53:43,277::health::100::health::(_check) Found 5873 uncollectible
objects, 10 suspects implementing __del__: ['<pthreading.Lock object
at 0x7f012c115320>', '<pthreading.Lock object at 0x7f012c115440>',
'<pthreading.Lock object at 0x7f012c1153b0>', '<pthreading.Lock object
at 0x7f012c1154d0>', '<pthread.Cond object at 0x7f012c111350>',
'<pthread.Cond object at 0x7f012c1112d0>', '<pthread.Cond object at
0x7f012c111250>', '<pthread.Cond object at 0x7f012c111850>',
'<pthread.Cond object at 0x7f012c1117d0>', '<pthread.Cond object at
0x7f012c111750>']
We found 460 uncollectible objects, 10 of them are objects implementing __del__.
Having such object in a reference cycle make the entire cycle uncollectible.
The pthreading objects in this log grow as vdsm continue to run, even without
any interaction with engine. The pthreading objects leak is caused by a cycle
in storage.misc.RWLock, fixed in patch [2].
After solving the pthreading leak, we still have many uncollectible
objects in the logs,
listing them, it seems that the suspect is hiding inside a collection
(dict, list, tuple)
and does not appear in the gc.garbage list. I will address this in later patch,
searching in the entire object graph instead of the top level of gc.garbage.
See end of this mail for list of these objects. I printed them like this:
Enable manhole debugging shell:
[devel]
manhole_enable = true
Connect to vdsm using manhole:
nc -U /run/vdsm/vdsmd.manhole
Print garbage:
>>> import pprint
>>> import gc
>>> with open('/tmp/garbage.txt', 'w') as f:
pprint.pprint(gc.garbage, stream=f, indent=4)
These objects looks related to libvirt - Francesco, can you take a look?
[1] https://gerrit.ovirt.org/51708
[2] https://gerrit.ovirt.org/51868
Nir
--- garbage after patch [2] ---
[ (<type '_ctypes.Array'>,),
{ '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>,
'__doc__': None,
'__module__': 'vdsm.netlink',
'__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>,
'_length_': 60,
'_type_': <class 'ctypes.c_char'>,
'raw': <attribute 'raw' of 'c_char_Array_60' objects>,
'value': <attribute 'value' of 'c_char_Array_60' objects>},
<class 'vdsm.netlink.c_char_Array_60'>,
<attribute '__dict__' of 'c_char_Array_60' objects>,
<attribute '__weakref__' of 'c_char_Array_60' objects>,
( <class 'vdsm.netlink.c_char_Array_60'>,
<type '_ctypes.Array'>,
<type '_ctypes._CData'>,
<type 'object'>),
<attribute 'raw' of 'c_char_Array_60' objects>,
<attribute 'value' of 'c_char_Array_60' objects>,
<cell at 0x23662f0: bool object at 0x7f1e086c8c40>,
<cell at 0x2366328: NoneType object at 0x7f1e086dcf00>,
<cell at 0x2366360: function object at 0x233f488>,
( <cell at 0x23662f0: bool object at 0x7f1e086c8c40>,
<cell at 0x2366328: NoneType object at 0x7f1e086dcf00>,
<cell at 0x2366360: function object at 0x233f488>),
<function wrapMethod at 0x233f488>,
[ [ <Recursion on list with id=34033464>,
<Recursion on list with id=34033464>,
'class'],
[ <Recursion on list with id=34033464>,
<Recursion on list with id=34033464>,
'class'],
None],
[ [ <Recursion on list with id=33979064>,
<Recursion on list with id=33979064>,
None],
[ <Recursion on list with id=33979064>,
<Recursion on list with id=33979064>,
None],
'class'],
[ [ <Recursion on list with id=35535256>,
<Recursion on list with id=35535256>,
'class'],
[ <Recursion on list with id=35535256>,
<Recursion on list with id=35535256>,
'class'],
None],
[ [ <Recursion on list with id=35535832>,
<Recursion on list with id=35535832>,
None],
[ <Recursion on list with id=35535832>,
<Recursion on list with id=35535832>,
None],
'class'],
[ [ <Recursion on list with id=35556312>,
<Recursion on list with id=35556312>,
'class'],
[ <Recursion on list with id=35556312>,
<Recursion on list with id=35556312>,
'class'],
None],
[ [ <Recursion on list with id=35556600>,
<Recursion on list with id=35556600>,
None],
[ <Recursion on list with id=35556600>,
<Recursion on list with id=35556600>,
None],
'class'],
[ [ <Recursion on list with id=35556960>,
<Recursion on list with id=35556960>,
'class'],
[ <Recursion on list with id=35556960>,
<Recursion on list with id=35556960>,
'class'],
None],
[ [ <Recursion on list with id=35561912>,
<Recursion on list with id=35561912>,
None],
[ <Recursion on list with id=35561912>,
<Recursion on list with id=35561912>,
None],
'class'],
[ [ <Recursion on list with id=35583904>,
<Recursion on list with id=35583904>,
'class'],
[ <Recursion on list with id=35583904>,
<Recursion on list with id=35583904>,
'class'],
None],
[ [ <Recursion on list with id=35584912>,
<Recursion on list with id=35584912>,
None],
[ <Recursion on list with id=35584912>,
<Recursion on list with id=35584912>,
None],
'class'],
[ [ <Recursion on list with id=36340368>,
<Recursion on list with id=36340368>,
'class'],
[ <Recursion on list with id=36340368>,
<Recursion on list with id=36340368>,
'class'],
None],
[ [ <Recursion on list with id=36340872>,
<Recursion on list with id=36340872>,
None],
[ <Recursion on list with id=36340872>,
<Recursion on list with id=36340872>,
None],
'class'],
[ [ <Recursion on list with id=36364872>,
<Recursion on list with id=36364872>,
'class'],
[ <Recursion on list with id=36364872>,
<Recursion on list with id=36364872>,
'class'],
None],
[ [ <Recursion on list with id=36365520>,
<Recursion on list with id=36365520>,
None],
[ <Recursion on list with id=36365520>,
<Recursion on list with id=36365520>,
None],
'class'],
[ [ <Recursion on list with id=36457216>,
<Recursion on list with id=36457216>,
'class'],
[ <Recursion on list with id=36457216>,
<Recursion on list with id=36457216>,
'class'],
None],
[ [ <Recursion on list with id=36458008>,
<Recursion on list with id=36458008>,
None],
[ <Recursion on list with id=36458008>,
<Recursion on list with id=36458008>,
None],
'class'],
[ [ <Recursion on list with id=36460520>,
<Recursion on list with id=36460520>,
'class'],
[ <Recursion on list with id=36460520>,
<Recursion on list with id=36460520>,
'class'],
None],
[ [ <Recursion on list with id=36461168>,
<Recursion on list with id=36461168>,
None],
[ <Recursion on list with id=36461168>,
<Recursion on list with id=36461168>,
None],
'class'],
[ [ <Recursion on list with id=36562344>,
<Recursion on list with id=36562344>,
'class'],
[ <Recursion on list with id=36562344>,
<Recursion on list with id=36562344>,
'class'],
None],
[ [ <Recursion on list with id=36563280>,
<Recursion on list with id=36563280>,
None],
[ <Recursion on list with id=36563280>,
<Recursion on list with id=36563280>,
None],
'class'],
<Element 'memory' at 0x2364710>,
{ '_children': [],
'attrib': { 'unit': 'KiB'},
'tag': 'memory',
'tail': '\n ',
'text': '4095476'},
[],
<Element 'pages' at 0x2364750>,
{ '_children': [],
'attrib': { 'size': '4', 'unit': 'KiB'},
'tag': 'pages',
'tail': '\n ',
'text': '1023869'},
[],
<Element 'pages' at 0x2364790>,
{ '_children': [],
'attrib': { 'size': '2048', 'unit': 'KiB'},
'tag': 'pages',
'tail': '\n ',
'text': '0'},
[],
<Element 'distances' at 0x23647d0>,
{ '_children': [<Element 'sibling' at 0x2364810>],
'attrib': { },
'tag': 'distances',
'tail': '\n ',
'text': '\n '},
[<Element 'sibling' at 0x2364810>],
<Element 'sibling' at 0x2364810>,
{ '_children': [],
'attrib': { 'id': '0', 'value': '10'},
'tag': 'sibling',
'tail': '\n '},
[],
<Element 'cpus' at 0x2364850>,
{ '_children': [ <Element 'cpu' at 0x2364890>,
<Element 'cpu' at 0x23648d0>,
<Element 'cpu' at 0x2364910>,
<Element 'cpu' at 0x2364950>,
<Element 'cpu' at 0x2364990>,
<Element 'cpu' at 0x23649d0>,
<Element 'cpu' at 0x2364a10>,
<Element 'cpu' at 0x2364a50>],
'attrib': { 'num': '8'},
'tag': 'cpus',
'tail': '\n ',
'text': '\n '},
[ <Element 'cpu' at 0x2364890>,
<Element 'cpu' at 0x23648d0>,
<Element 'cpu' at 0x2364910>,
<Element 'cpu' at 0x2364950>,
<Element 'cpu' at 0x2364990>,
<Element 'cpu' at 0x23649d0>,
<Element 'cpu' at 0x2364a10>,
<Element 'cpu' at 0x2364a50>],
<Element 'cpu' at 0x2364890>,
{ '_children': [],
'attrib': { 'core_id': '0',
'id': '0',
'siblings': '0',
'socket_id': '0'},
'tag': 'cpu',
'tail': '\n '},
[],
<Element 'cpu' at 0x23648d0>,
{ '_children': [],
'attrib': { 'core_id': '0',
'id': '1',
'siblings': '1',
'socket_id': '1'},
'tag': 'cpu',
'tail': '\n '},
[],
<Element 'cpu' at 0x2364910>,
{ '_children': [],
'attrib': { 'core_id': '0',
'id': '2',
'siblings': '2',
'socket_id': '2'},
'tag': 'cpu',
'tail': '\n '},
[],
<Element 'cpu' at 0x2364950>,
{ '_children': [],
'attrib': { 'core_id': '0',
'id': '3',
'siblings': '3',
'socket_id': '3'},
'tag': 'cpu',
'tail': '\n '},
[],
<Element 'cpu' at 0x2364990>,
{ '_children': [],
'attrib': { 'core_id': '0',
'id': '4',
'siblings': '4',
'socket_id': '4'},
'tag': 'cpu',
'tail': '\n '},
[],
<Element 'cpu' at 0x23649d0>,
{ '_children': [],
'attrib': { 'core_id': '0',
'id': '5',
'siblings': '5',
'socket_id': '5'},
'tag': 'cpu',
'tail': '\n '},
[],
<Element 'cpu' at 0x2364a10>,
{ '_children': [],
'attrib': { 'core_id': '0',
'id': '6',
'siblings': '6',
'socket_id': '6'},
'tag': 'cpu',
'tail': '\n '},
[],
<Element 'cpu' at 0x2364a50>,
{ '_children': [],
'attrib': { 'core_id': '0',
'id': '7',
'siblings': '7',
'socket_id': '7'},
'tag': 'cpu',
'tail': '\n '},
[],
<Element 'secmodel' at 0x2364a90>,
{ '_children': [ <Element 'model' at 0x2364ad0>,
<Element 'doi' at 0x2364b10>,
<Element 'baselabel' at 0x2364b50>,
<Element 'baselabel' at 0x2364b90>],
'attrib': { },
'tag': 'secmodel',
'tail': '\n ',
'text': '\n '},
[ <Element 'model' at 0x2364ad0>,
<Element 'doi' at 0x2364b10>,
<Element 'baselabel' at 0x2364b50>,
<Element 'baselabel' at 0x2364b90>],
<Element 'model' at 0x2364ad0>,
{ '_children': [],
'attrib': { },
'tag': 'model',
'tail': '\n ',
'text': 'selinux'},
[],
<Element 'doi' at 0x2364b10>,
{ '_children': [],
'attrib': { },
'tag': 'doi',
'tail': '\n ',
'text': '0'},
[],
<Element 'baselabel' at 0x2364b50>,
{ '_children': [],
'attrib': { 'type': 'kvm'},
'tag': 'baselabel',
'tail': '\n ',
'text': 'system_u:system_r:svirt_t:s0'},
[],
<Element 'baselabel' at 0x2364b90>,
{ '_children': [],
'attrib': { 'type': 'qemu'},
'tag': 'baselabel',
'tail': '\n ',
'text': 'system_u:system_r:svirt_tcg_t:s0'},
[],
<Element 'secmodel' at 0x2364bd0>,
{ '_children': [ <Element 'model' at 0x2364c10>,
<Element 'doi' at 0x2364c50>,
<Element 'baselabel' at 0x2364c90>,
<Element 'baselabel' at 0x2364cd0>],
'attrib': { },
'tag': 'secmodel',
'tail': '\n ',
'text': '\n '},
[ <Element 'model' at 0x2364c10>,
<Element 'doi' at 0x2364c50>,
<Element 'baselabel' at 0x2364c90>,
<Element 'baselabel' at 0x2364cd0>],
<Element 'model' at 0x2364c10>,
{ '_children': [],
'attrib': { },
'tag': 'model',
'tail': '\n ',
'text': 'dac'},
[],
<Element 'doi' at 0x2364c50>,
{ '_children': [],
'attrib': { },
'tag': 'doi',
'tail': '\n ',
'text': '0'},
[],
<Element 'baselabel' at 0x2364c90>,
{ '_children': [],
'attrib': { 'type': 'kvm'},
'tag': 'baselabel',
'tail': '\n ',
'text': '+107:+107'},
[],
<Element 'baselabel' at 0x2364cd0>,
{ '_children': [],
'attrib': { 'type': 'qemu'},
'tag': 'baselabel',
'tail': '\n ',
'text': '+107:+107'},
[],
<Element 'guest' at 0x2364d10>,
{ '_children': [ <Element 'os_type' at 0x2364d50>,
<Element 'arch' at 0x2364d90>,
<Element 'features' at 0x2369390>],
'attrib': { },
'tag': 'guest',
'tail': '\n\n ',
'text': '\n '},
[ <Element 'os_type' at 0x2364d50>,
<Element 'arch' at 0x2364d90>,
<Element 'features' at 0x2369390>],
<Element 'os_type' at 0x2364d50>,
{ '_children': [],
'attrib': { },
'tag': 'os_type',
'tail': '\n ',
'text': 'hvm'},
[],
<Element 'arch' at 0x2364d90>,
{ '_children': [ <Element 'wordsize' at 0x2364dd0>,
<Element 'emulator' at 0x2364e10>,
<Element 'machine' at 0x2364e90>,
<Element 'machine' at 0x2364ed0>,
<Element 'machine' at 0x2364f10>,
<Element 'machine' at 0x2364f50>,
<Element 'machine' at 0x2364f90>,
<Element 'machine' at 0x2364fd0>,
<Element 'machine' at 0x2369050>,
<Element 'machine' at 0x2369090>,
<Element 'machine' at 0x23690d0>,
<Element 'machine' at 0x2369110>,
<Element 'machine' at 0x2369150>,
<Element 'machine' at 0x2369190>,
<Element 'machine' at 0x23691d0>,
<Element 'machine' at 0x2369210>,
<Element 'machine' at 0x2369250>,
<Element 'domain' at 0x2369290>,
<Element 'domain' at 0x23692d0>],
'attrib': { 'name': 'i686'},
'tag': 'arch',
'tail': '\n ',
'text': '\n '},
[ <Element 'wordsize' at 0x2364dd0>,
<Element 'emulator' at 0x2364e10>,
<Element 'machine' at 0x2364e90>,
<Element 'machine' at 0x2364ed0>,
<Element 'machine' at 0x2364f10>,
<Element 'machine' at 0x2364f50>,
<Element 'machine' at 0x2364f90>,
<Element 'machine' at 0x2364fd0>,
<Element 'machine' at 0x2369050>,
<Element 'machine' at 0x2369090>,
<Element 'machine' at 0x23690d0>,
<Element 'machine' at 0x2369110>,
<Element 'machine' at 0x2369150>,
<Element 'machine' at 0x2369190>,
<Element 'machine' at 0x23691d0>,
<Element 'machine' at 0x2369210>,
<Element 'machine' at 0x2369250>,
<Element 'domain' at 0x2369290>,
<Element 'domain' at 0x23692d0>],
<Element 'wordsize' at 0x2364dd0>,
{ '_children': [],
'attrib': { },
'tag': 'wordsize',
'tail': '\n ',
'text': '32'},
[],
<Element 'emulator' at 0x2364e10>,
{ '_children': [],
'attrib': { },
'tag': 'emulator',
'tail': '\n ',
'text': '/usr/libexec/qemu-kvm'},
[],
<Element 'machine' at 0x2364e90>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'pc-i440fx-rhel7.2.0'},
[],
<Element 'machine' at 0x2364ed0>,
{ '_children': [],
'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'pc'},
[],
<Element 'machine' at 0x2364f10>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'pc-i440fx-rhel7.0.0'},
[],
<Element 'machine' at 0x2364f50>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'pc-q35-rhel7.1.0'},
[],
<Element 'machine' at 0x2364f90>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'rhel6.3.0'},
[],
<Element 'machine' at 0x2364fd0>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'pc-q35-rhel7.2.0'},
[],
<Element 'machine' at 0x2369050>,
{ '_children': [],
'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'q35'},
[],
<Element 'machine' at 0x2369090>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'rhel6.4.0'},
[],
<Element 'machine' at 0x23690d0>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'rhel6.0.0'},
[],
<Element 'machine' at 0x2369110>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'pc-i440fx-rhel7.1.0'},
[],
<Element 'machine' at 0x2369150>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'rhel6.5.0'},
[],
<Element 'machine' at 0x2369190>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'rhel6.6.0'},
[],
<Element 'machine' at 0x23691d0>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'rhel6.1.0'},
[],
<Element 'machine' at 0x2369210>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'pc-q35-rhel7.0.0'},
[],
<Element 'machine' at 0x2369250>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'rhel6.2.0'},
[],
<Element 'domain' at 0x2369290>,
{ '_children': [],
'attrib': { 'type': 'qemu'},
'tag': 'domain',
'tail': '\n '},
[],
<Element 'domain' at 0x23692d0>,
{ '_children': [<Element 'emulator' at 0x2369310>],
'attrib': { 'type': 'kvm'},
'tag': 'domain',
'tail': '\n ',
'text': '\n '},
[<Element 'emulator' at 0x2369310>],
<Element 'emulator' at 0x2369310>,
{ '_children': [],
'attrib': { },
'tag': 'emulator',
'tail': '\n ',
'text': '/usr/libexec/qemu-kvm'},
[],
<Element 'features' at 0x2369390>,
{ '_children': [ <Element 'cpuselection' at 0x23693d0>,
<Element 'deviceboot' at 0x2369410>,
<Element 'disksnapshot' at 0x2369450>,
<Element 'acpi' at 0x2369490>,
<Element 'apic' at 0x23694d0>,
<Element 'pae' at 0x2369510>,
<Element 'nonpae' at 0x2369550>],
'attrib': { },
'tag': 'features',
'tail': '\n ',
'text': '\n '},
[ <Element 'cpuselection' at 0x23693d0>,
<Element 'deviceboot' at 0x2369410>,
<Element 'disksnapshot' at 0x2369450>,
<Element 'acpi' at 0x2369490>,
<Element 'apic' at 0x23694d0>,
<Element 'pae' at 0x2369510>,
<Element 'nonpae' at 0x2369550>],
<Element 'cpuselection' at 0x23693d0>,
{ '_children': [], 'attrib': { }, 'tag': 'cpuselection',
'tail': '\n '},
[],
<Element 'deviceboot' at 0x2369410>,
{ '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail':
'\n '},
[],
<Element 'disksnapshot' at 0x2369450>,
{ '_children': [],
'attrib': { 'default': 'on', 'toggle': 'no'},
'tag': 'disksnapshot',
'tail': '\n '},
[],
<Element 'acpi' at 0x2369490>,
{ '_children': [],
'attrib': { 'default': 'on', 'toggle': 'yes'},
'tag': 'acpi',
'tail': '\n '},
[],
<Element 'apic' at 0x23694d0>,
{ '_children': [],
'attrib': { 'default': 'on', 'toggle': 'no'},
'tag': 'apic',
'tail': '\n '},
[],
<Element 'pae' at 0x2369510>,
{ '_children': [], 'attrib': { }, 'tag': 'pae', 'tail': '\n '},
[],
<Element 'nonpae' at 0x2369550>,
{ '_children': [], 'attrib': { }, 'tag': 'nonpae', 'tail': '\n '},
[],
<Element 'guest' at 0x2369590>,
{ '_children': [ <Element 'os_type' at 0x23695d0>,
<Element 'arch' at 0x2369610>,
<Element 'features' at 0x2369bd0>],
'attrib': { },
'tag': 'guest',
'tail': '\n\n',
'text': '\n '},
[ <Element 'os_type' at 0x23695d0>,
<Element 'arch' at 0x2369610>,
<Element 'features' at 0x2369bd0>],
<Element 'os_type' at 0x23695d0>,
{ '_children': [],
'attrib': { },
'tag': 'os_type',
'tail': '\n ',
'text': 'hvm'},
[],
<Element 'arch' at 0x2369610>,
{ '_children': [ <Element 'wordsize' at 0x2369650>,
<Element 'emulator' at 0x2369690>,
<Element 'machine' at 0x2369710>,
<Element 'machine' at 0x2369750>,
<Element 'machine' at 0x2369790>,
<Element 'machine' at 0x23697d0>,
<Element 'machine' at 0x2369810>,
<Element 'machine' at 0x2369850>,
<Element 'machine' at 0x2369890>,
<Element 'machine' at 0x23698d0>,
<Element 'machine' at 0x2369910>,
<Element 'machine' at 0x2369950>,
<Element 'machine' at 0x2369990>,
<Element 'machine' at 0x23699d0>,
<Element 'machine' at 0x2369a10>,
<Element 'machine' at 0x2369a50>,
<Element 'machine' at 0x2369a90>,
<Element 'domain' at 0x2369ad0>,
<Element 'domain' at 0x2369b10>],
'attrib': { 'name': 'x86_64'},
'tag': 'arch',
'tail': '\n ',
'text': '\n '},
[ <Element 'wordsize' at 0x2369650>,
<Element 'emulator' at 0x2369690>,
<Element 'machine' at 0x2369710>,
<Element 'machine' at 0x2369750>,
<Element 'machine' at 0x2369790>,
<Element 'machine' at 0x23697d0>,
<Element 'machine' at 0x2369810>,
<Element 'machine' at 0x2369850>,
<Element 'machine' at 0x2369890>,
<Element 'machine' at 0x23698d0>,
<Element 'machine' at 0x2369910>,
<Element 'machine' at 0x2369950>,
<Element 'machine' at 0x2369990>,
<Element 'machine' at 0x23699d0>,
<Element 'machine' at 0x2369a10>,
<Element 'machine' at 0x2369a50>,
<Element 'machine' at 0x2369a90>,
<Element 'domain' at 0x2369ad0>,
<Element 'domain' at 0x2369b10>],
<Element 'wordsize' at 0x2369650>,
{ '_children': [],
'attrib': { },
'tag': 'wordsize',
'tail': '\n ',
'text': '64'},
[],
<Element 'emulator' at 0x2369690>,
{ '_children': [],
'attrib': { },
'tag': 'emulator',
'tail': '\n ',
'text': '/usr/libexec/qemu-kvm'},
[],
<Element 'machine' at 0x2369710>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'pc-i440fx-rhel7.2.0'},
[],
<Element 'machine' at 0x2369750>,
{ '_children': [],
'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'pc'},
[],
<Element 'machine' at 0x2369790>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'pc-i440fx-rhel7.0.0'},
[],
<Element 'machine' at 0x23697d0>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'pc-q35-rhel7.1.0'},
[],
<Element 'machine' at 0x2369810>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'rhel6.3.0'},
[],
<Element 'machine' at 0x2369850>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'pc-q35-rhel7.2.0'},
[],
<Element 'machine' at 0x2369890>,
{ '_children': [],
'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'q35'},
[],
<Element 'machine' at 0x23698d0>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'rhel6.4.0'},
[],
<Element 'machine' at 0x2369910>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'rhel6.0.0'},
[],
<Element 'machine' at 0x2369950>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'pc-i440fx-rhel7.1.0'},
[],
<Element 'machine' at 0x2369990>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'rhel6.5.0'},
[],
<Element 'machine' at 0x23699d0>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'rhel6.6.0'},
[],
<Element 'machine' at 0x2369a10>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'rhel6.1.0'},
[],
<Element 'machine' at 0x2369a50>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'pc-q35-rhel7.0.0'},
[],
<Element 'machine' at 0x2369a90>,
{ '_children': [],
'attrib': { 'maxCpus': '240'},
'tag': 'machine',
'tail': '\n ',
'text': 'rhel6.2.0'},
[],
<Element 'domain' at 0x2369ad0>,
{ '_children': [],
'attrib': { 'type': 'qemu'},
'tag': 'domain',
'tail': '\n '},
[],
<Element 'domain' at 0x2369b10>,
{ '_children': [<Element 'emulator' at 0x2369b50>],
'attrib': { 'type': 'kvm'},
'tag': 'domain',
'tail': '\n ',
'text': '\n '},
[<Element 'emulator' at 0x2369b50>],
<Element 'emulator' at 0x2369b50>,
{ '_children': [],
'attrib': { },
'tag': 'emulator',
'tail': '\n ',
'text': '/usr/libexec/qemu-kvm'},
[],
<Element 'features' at 0x2369bd0>,
{ '_children': [ <Element 'cpuselection' at 0x2369c10>,
<Element 'deviceboot' at 0x2369c50>,
<Element 'disksnapshot' at 0x2369c90>,
<Element 'acpi' at 0x2369cd0>,
<Element 'apic' at 0x2369d10>],
'attrib': { },
'tag': 'features',
'tail': '\n ',
'text': '\n '},
[ <Element 'cpuselection' at 0x2369c10>,
<Element 'deviceboot' at 0x2369c50>,
<Element 'disksnapshot' at 0x2369c90>,
<Element 'acpi' at 0x2369cd0>,
<Element 'apic' at 0x2369d10>],
<Element 'cpuselection' at 0x2369c10>,
{ '_children': [], 'attrib': { }, 'tag': 'cpuselection',
'tail': '\n '},
[],
<Element 'deviceboot' at 0x2369c50>,
{ '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail':
'\n '},
[],
<Element 'disksnapshot' at 0x2369c90>,
{ '_children': [],
'attrib': { 'default': 'on', 'toggle': 'no'},
'tag': 'disksnapshot',
'tail': '\n '},
[],
<Element 'acpi' at 0x2369cd0>,
{ '_children': [],
'attrib': { 'default': 'on', 'toggle': 'yes'},
'tag': 'acpi',
'tail': '\n '},
[],
<Element 'apic' at 0x2369d10>,
{ '_children': [],
'attrib': { 'default': 'on', 'toggle': 'no'},
'tag': 'apic',
'tail': '\n '},
[],
[],
(<type '_ctypes.Array'>,),
{ '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>,
'__doc__': None,
'__module__': 'vdsm.netlink',
'__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>,
'_length_': 60,
'_type_': <class 'ctypes.c_char'>,
'raw': <attribute 'raw' of 'c_char_Array_60' objects>,
'value': <attribute 'value' of 'c_char_Array_60' objects>},
<class 'vdsm.netlink.c_char_Array_60'>,
<attribute '__dict__' of 'c_char_Array_60' objects>,
<attribute '__weakref__' of 'c_char_Array_60' objects>,
( <class 'vdsm.netlink.c_char_Array_60'>,
<type '_ctypes.Array'>,
<type '_ctypes._CData'>,
<type 'object'>),
<attribute 'raw' of 'c_char_Array_60' objects>,
<attribute 'value' of 'c_char_Array_60' objects>,
<xml.etree.ElementTree.XMLParser object at 0x2357f50>,
{ '_doctype': None,
'_error': <class 'xml.parsers.expat.ExpatError'>,
'_names': { u'acpi': 'acpi',
u'apic': 'apic',
u'arch': 'arch',
u'baselabel': 'baselabel',
u'canonical': 'canonical',
u'capabilities': 'capabilities',
u'cell': 'cell',
u'cells': 'cells',
u'core_id': 'core_id',
u'cores': 'cores',
u'cpu': 'cpu',
u'cpus': 'cpus',
u'cpuselection': 'cpuselection',
u'default': 'default',
u'deviceboot': 'deviceboot',
u'disksnapshot': 'disksnapshot',
u'distances': 'distances',
u'doi': 'doi',
u'domain': 'domain',
u'emulator': 'emulator',
u'feature': 'feature',
u'features': 'features',
u'guest': 'guest',
u'host': 'host',
u'id': 'id',
u'live': 'live',
u'machine': 'machine',
u'maxCpus': 'maxCpus',
u'memory': 'memory',
u'migration_features': 'migration_features',
u'model': 'model',
u'name': 'name',
u'nonpae': 'nonpae',
u'num': 'num',
u'os_type': 'os_type',
u'pae': 'pae',
u'pages': 'pages',
u'power_management': 'power_management',
u'secmodel': 'secmodel',
u'sibling': 'sibling',
u'siblings': 'siblings',
u'size': 'size',
u'socket_id': 'socket_id',
u'sockets': 'sockets',
u'suspend_disk': 'suspend_disk',
u'suspend_hybrid': 'suspend_hybrid',
u'suspend_mem': 'suspend_mem',
u'threads': 'threads',
u'toggle': 'toggle',
u'topology': 'topology',
u'type': 'type',
u'unit': 'unit',
u'uri_transport': 'uri_transport',
u'uri_transports': 'uri_transports',
u'uuid': 'uuid',
u'value': 'value',
u'vendor': 'vendor',
u'wordsize': 'wordsize'},
'_target': <xml.etree.ElementTree.TreeBuilder object at 0x2357f10>,
'entity': { },
'parser': <pyexpat.xmlparser object at 0x22e5360>,
'version': 'Expat 2.1.0'},
<pyexpat.xmlparser object at 0x22e5360>,
<xml.etree.ElementTree.TreeBuilder object at 0x2357f10>,
<bound method XMLParser._start_list of
<xml.etree.ElementTree.XMLParser object at 0x2357f50>>,
<bound method XMLParser._end of <xml.etree.ElementTree.XMLParser
object at 0x2357f50>>,
<bound method XMLParser._pi of <xml.etree.ElementTree.XMLParser
object at 0x2357f50>>,
<bound method XMLParser._data of <xml.etree.ElementTree.XMLParser
object at 0x2357f50>>,
<bound method XMLParser._comment of
<xml.etree.ElementTree.XMLParser object at 0x2357f50>>,
<bound method XMLParser._default of
<xml.etree.ElementTree.XMLParser object at 0x2357f50>>,
{ '_data': [],
'_elem': [],
'_factory': <class 'xml.etree.ElementTree.Element'>,
'_last': <Element 'capabilities' at 0x2357f90>,
'_tail': 1},
[],
<Element 'cell' at 0x23646d0>,
<Element 'cells' at 0x2364690>,
<Element 'topology' at 0x2364650>,
<Element 'host' at 0x2357fd0>,
<Element 'capabilities' at 0x2357f90>,
{ '_children': [ <Element 'memory' at 0x2364710>,
<Element 'pages' at 0x2364750>,
<Element 'pages' at 0x2364790>,
<Element 'distances' at 0x23647d0>,
<Element 'cpus' at 0x2364850>],
'attrib': { 'id': '0'},
'tag': 'cell',
'tail': '\n ',
'text': '\n '},
{ '_children': [<Element 'cell' at 0x23646d0>],
'attrib': { 'num': '1'},
'tag': 'cells',
'tail': '\n ',
'text': '\n '},
{ '_children': [<Element 'cells' at 0x2364690>],
'attrib': { },
'tag': 'topology',
'tail': '\n ',
'text': '\n '},
{ '_children': [ <Element 'uuid' at 0x2364050>,
<Element 'cpu' at 0x2364090>,
<Element 'power_management' at 0x2364390>,
<Element 'migration_features' at 0x23644d0>,
<Element 'topology' at 0x2364650>,
<Element 'secmodel' at 0x2364a90>,
<Element 'secmodel' at 0x2364bd0>],
'attrib': { },
'tag': 'host',
'tail': '\n\n ',
'text': '\n '},
{ '_children': [ <Element 'host' at 0x2357fd0>,
<Element 'guest' at 0x2364d10>,
<Element 'guest' at 0x2369590>],
'attrib': { },
'tag': 'capabilities',
'text': '\n\n '},
[ <Element 'memory' at 0x2364710>,
<Element 'pages' at 0x2364750>,
<Element 'pages' at 0x2364790>,
<Element 'distances' at 0x23647d0>,
<Element 'cpus' at 0x2364850>],
[<Element 'cell' at 0x23646d0>],
[<Element 'cells' at 0x2364690>],
[ <Element 'uuid' at 0x2364050>,
<Element 'cpu' at 0x2364090>,
<Element 'power_management' at 0x2364390>,
<Element 'migration_features' at 0x23644d0>,
<Element 'topology' at 0x2364650>,
<Element 'secmodel' at 0x2364a90>,
<Element 'secmodel' at 0x2364bd0>],
[ <Element 'host' at 0x2357fd0>,
<Element 'guest' at 0x2364d10>,
<Element 'guest' at 0x2369590>],
<Element 'migration_features' at 0x23644d0>,
<Element 'power_management' at 0x2364390>,
<Element 'cpu' at 0x2364090>,
<Element 'uuid' at 0x2364050>,
{ '_children': [ <Element 'live' at 0x2364510>,
<Element 'uri_transports' at 0x2364590>],
'attrib': { },
'tag': 'migration_features',
'tail': '\n ',
'text': '\n '},
{ '_children': [ <Element 'suspend_mem' at 0x23643d0>,
<Element 'suspend_disk' at 0x2364410>,
<Element 'suspend_hybrid' at 0x2364490>],
'attrib': { },
'tag': 'power_management',
'tail': '\n ',
'text': '\n '},
{ '_children': [ <Element 'arch' at 0x23640d0>,
<Element 'model' at 0x2364110>,
<Element 'vendor' at 0x2364150>,
<Element 'topology' at 0x2364190>,
<Element 'feature' at 0x23641d0>,
<Element 'feature' at 0x2364210>,
<Element 'feature' at 0x2364250>,
<Element 'feature' at 0x2364290>,
<Element 'feature' at 0x23642d0>,
<Element 'pages' at 0x2364310>,
<Element 'pages' at 0x2364350>],
'attrib': { },
'tag': 'cpu',
'tail': '\n ',
'text': '\n '},
{ '_children': [],
'attrib': { },
'tag': 'uuid',
'tail': '\n ',
'text': 'a920d963-c18f-4fda-aa5a-deee9bbcde9d'},
[<Element 'live' at 0x2364510>, <Element 'uri_transports' at 0x2364590>],
[ <Element 'suspend_mem' at 0x23643d0>,
<Element 'suspend_disk' at 0x2364410>,
<Element 'suspend_hybrid' at 0x2364490>],
[ <Element 'arch' at 0x23640d0>,
<Element 'model' at 0x2364110>,
<Element 'vendor' at 0x2364150>,
<Element 'topology' at 0x2364190>,
<Element 'feature' at 0x23641d0>,
<Element 'feature' at 0x2364210>,
<Element 'feature' at 0x2364250>,
<Element 'feature' at 0x2364290>,
<Element 'feature' at 0x23642d0>,
<Element 'pages' at 0x2364310>,
<Element 'pages' at 0x2364350>],
[],
<Element 'uri_transports' at 0x2364590>,
<Element 'live' at 0x2364510>,
<Element 'suspend_hybrid' at 0x2364490>,
<Element 'suspend_disk' at 0x2364410>,
<Element 'suspend_mem' at 0x23643d0>,
<Element 'pages' at 0x2364350>,
<Element 'pages' at 0x2364310>,
<Element 'feature' at 0x23642d0>,
<Element 'feature' at 0x2364290>,
<Element 'feature' at 0x2364250>,
<Element 'feature' at 0x2364210>,
<Element 'feature' at 0x23641d0>,
<Element 'topology' at 0x2364190>,
<Element 'vendor' at 0x2364150>,
<Element 'model' at 0x2364110>,
<Element 'arch' at 0x23640d0>,
{ '_children': [ <Element 'uri_transport' at 0x23645d0>,
<Element 'uri_transport' at 0x2364610>],
'attrib': { },
'tag': 'uri_transports',
'tail': '\n ',
'text': '\n '},
{ '_children': [], 'attrib': { }, 'tag': 'live', 'tail': '\n '},
{ '_children': [], 'attrib': { }, 'tag': 'suspend_hybrid',
'tail': '\n '},
{ '_children': [], 'attrib': { }, 'tag': 'suspend_disk',
'tail': '\n '},
{ '_children': [], 'attrib': { }, 'tag': 'suspend_mem',
'tail': '\n '},
{ '_children': [],
'attrib': { 'size': '2048', 'unit': 'KiB'},
'tag': 'pages',
'tail': '\n '},
{ '_children': [],
'attrib': { 'size': '4', 'unit': 'KiB'},
'tag': 'pages',
'tail': '\n '},
{ '_children': [],
'attrib': { 'name': 'ss'},
'tag': 'feature',
'tail': '\n '},
{ '_children': [],
'attrib': { 'name': 'vmx'},
'tag': 'feature',
'tail': '\n '},
{ '_children': [],
'attrib': { 'name': 'pcid'},
'tag': 'feature',
'tail': '\n '},
{ '_children': [],
'attrib': { 'name': 'osxsave'},
'tag': 'feature',
'tail': '\n '},
{ '_children': [],
'attrib': { 'name': 'hypervisor'},
'tag': 'feature',
'tail': '\n '},
{ '_children': [],
'attrib': { 'cores': '1', 'sockets': '8', 'threads': '1'},
'tag': 'topology',
'tail': '\n '},
{ '_children': [],
'attrib': { },
'tag': 'vendor',
'tail': '\n ',
'text': 'Intel'},
{ '_children': [],
'attrib': { },
'tag': 'model',
'tail': '\n ',
'text': 'IvyBridge'},
{ '_children': [],
'attrib': { },
'tag': 'arch',
'tail': '\n ',
'text': 'x86_64'},
[ <Element 'uri_transport' at 0x23645d0>,
<Element 'uri_transport' at 0x2364610>],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
<Element 'uri_transport' at 0x2364610>,
<Element 'uri_transport' at 0x23645d0>,
{ '_children': [],
'attrib': { },
'tag': 'uri_transport',
'tail': '\n ',
'text': 'rdma'},
{ '_children': [],
'attrib': { },
'tag': 'uri_transport',
'tail': '\n ',
'text': 'tcp'},
[],
[],
(<type '_ctypes.Array'>,),
{ '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>,
'__doc__': None,
'__module__': 'vdsm.netlink',
'__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>,
'_length_': 60,
'_type_': <class 'ctypes.c_char'>,
'raw': <attribute 'raw' of 'c_char_Array_60' objects>,
'value': <attribute 'value' of 'c_char_Array_60' objects>},
<class 'vdsm.netlink.c_char_Array_60'>,
<attribute '__dict__' of 'c_char_Array_60' objects>,
<attribute '__weakref__' of 'c_char_Array_60' objects>,
( <class 'vdsm.netlink.c_char_Array_60'>,
<type '_ctypes.Array'>,
<type '_ctypes._CData'>,
<type 'object'>),
<attribute 'raw' of 'c_char_Array_60' objects>,
<attribute 'value' of 'c_char_Array_60' objects>]
More information about the Devel
mailing list