[ovirt-devel] [VDSM] health: Introduce Vdsm health monitoring
Vinzenz Feenstra
vfeenstr at redhat.com
Thu Jan 14 17:44:18 UTC 2016
> On Jan 14, 2016, at 6:38 PM, Nir Soffer <nsoffer at redhat.com> wrote:
>
> Hi all,
>
> Continuing the leak investigation Francesco and Milan are working on,
> I posted this patch, adding leak health monitoring to vdsm [1]. This patch
> currently monitor collectible objects, and we may add other interesting
> stuff later if needed.
>
> To enable the monitor, you must set:
>
> [devel]
> health_monitor_enable = true
>
> In vdsm.conf, and restart vdsm.
>
> Here is an example logs - this was logged 60 seconds after starting vdsm,
> when running as spm:
>
> Thread-13::DEBUG::2016-01-14
> 18:53:43,239::health::84::health::(_check) Checking health
> Thread-13::DEBUG::2016-01-14
> 18:53:43,272::health::86::health::(_check) Collected 460 objects
> Thread-13::WARNING::2016-01-14
> 18:53:43,277::health::100::health::(_check) Found 5873 uncollectible
> objects, 10 suspects implementing __del__: ['<pthreading.Lock object
> at 0x7f012c115320>', '<pthreading.Lock object at 0x7f012c115440>',
> '<pthreading.Lock object at 0x7f012c1153b0>', '<pthreading.Lock object
> at 0x7f012c1154d0>', '<pthread.Cond object at 0x7f012c111350>',
> '<pthread.Cond object at 0x7f012c1112d0>', '<pthread.Cond object at
> 0x7f012c111250>', '<pthread.Cond object at 0x7f012c111850>',
> '<pthread.Cond object at 0x7f012c1117d0>', '<pthread.Cond object at
> 0x7f012c111750>']
>
> We found 460 uncollectible objects, 10 of them are objects implementing __del__.
> Having such object in a reference cycle make the entire cycle uncollectible.
>
> The pthreading objects in this log grow as vdsm continue to run, even without
> any interaction with engine. The pthreading objects leak is caused by a cycle
> in storage.misc.RWLock, fixed in patch [2].
>
> After solving the pthreading leak, we still have many uncollectible
> objects in the logs,
> listing them, it seems that the suspect is hiding inside a collection
> (dict, list, tuple)
> and does not appear in the gc.garbage list. I will address this in later patch,
> searching in the entire object graph instead of the top level of gc.garbage.
>
> See end of this mail for list of these objects. I printed them like this:
>
> Enable manhole debugging shell:
>
> [devel]
> manhole_enable = true
>
> Connect to vdsm using manhole:
>
> nc -U /run/vdsm/vdsmd.manhole
>
> Print garbage:
>
>>>> import pprint
>>>> import gc
>>>> with open('/tmp/garbage.txt', 'w') as f:
> pprint.pprint(gc.garbage, stream=f, indent=4)
>
> These objects looks related to libvirt - Francesco, can you take a look?
Since you’re addressing Francesco directly, I am adding him explicitly - To point that out - Someone might skip that little part here ;)
>
> [1] https://gerrit.ovirt.org/51708
> [2] https://gerrit.ovirt.org/51868
>
> Nir
>
> --- garbage after patch [2] ---
>
> [ (<type '_ctypes.Array'>,),
> { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>,
> '__doc__': None,
> '__module__': 'vdsm.netlink',
> '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>,
> '_length_': 60,
> '_type_': <class 'ctypes.c_char'>,
> 'raw': <attribute 'raw' of 'c_char_Array_60' objects>,
> 'value': <attribute 'value' of 'c_char_Array_60' objects>},
> <class 'vdsm.netlink.c_char_Array_60'>,
> <attribute '__dict__' of 'c_char_Array_60' objects>,
> <attribute '__weakref__' of 'c_char_Array_60' objects>,
> ( <class 'vdsm.netlink.c_char_Array_60'>,
> <type '_ctypes.Array'>,
> <type '_ctypes._CData'>,
> <type 'object'>),
> <attribute 'raw' of 'c_char_Array_60' objects>,
> <attribute 'value' of 'c_char_Array_60' objects>,
> <cell at 0x23662f0: bool object at 0x7f1e086c8c40>,
> <cell at 0x2366328: NoneType object at 0x7f1e086dcf00>,
> <cell at 0x2366360: function object at 0x233f488>,
> ( <cell at 0x23662f0: bool object at 0x7f1e086c8c40>,
> <cell at 0x2366328: NoneType object at 0x7f1e086dcf00>,
> <cell at 0x2366360: function object at 0x233f488>),
> <function wrapMethod at 0x233f488>,
> [ [ <Recursion on list with id=34033464>,
> <Recursion on list with id=34033464>,
> 'class'],
> [ <Recursion on list with id=34033464>,
> <Recursion on list with id=34033464>,
> 'class'],
> None],
> [ [ <Recursion on list with id=33979064>,
> <Recursion on list with id=33979064>,
> None],
> [ <Recursion on list with id=33979064>,
> <Recursion on list with id=33979064>,
> None],
> 'class'],
> [ [ <Recursion on list with id=35535256>,
> <Recursion on list with id=35535256>,
> 'class'],
> [ <Recursion on list with id=35535256>,
> <Recursion on list with id=35535256>,
> 'class'],
> None],
> [ [ <Recursion on list with id=35535832>,
> <Recursion on list with id=35535832>,
> None],
> [ <Recursion on list with id=35535832>,
> <Recursion on list with id=35535832>,
> None],
> 'class'],
> [ [ <Recursion on list with id=35556312>,
> <Recursion on list with id=35556312>,
> 'class'],
> [ <Recursion on list with id=35556312>,
> <Recursion on list with id=35556312>,
> 'class'],
> None],
> [ [ <Recursion on list with id=35556600>,
> <Recursion on list with id=35556600>,
> None],
> [ <Recursion on list with id=35556600>,
> <Recursion on list with id=35556600>,
> None],
> 'class'],
> [ [ <Recursion on list with id=35556960>,
> <Recursion on list with id=35556960>,
> 'class'],
> [ <Recursion on list with id=35556960>,
> <Recursion on list with id=35556960>,
> 'class'],
> None],
> [ [ <Recursion on list with id=35561912>,
> <Recursion on list with id=35561912>,
> None],
> [ <Recursion on list with id=35561912>,
> <Recursion on list with id=35561912>,
> None],
> 'class'],
> [ [ <Recursion on list with id=35583904>,
> <Recursion on list with id=35583904>,
> 'class'],
> [ <Recursion on list with id=35583904>,
> <Recursion on list with id=35583904>,
> 'class'],
> None],
> [ [ <Recursion on list with id=35584912>,
> <Recursion on list with id=35584912>,
> None],
> [ <Recursion on list with id=35584912>,
> <Recursion on list with id=35584912>,
> None],
> 'class'],
> [ [ <Recursion on list with id=36340368>,
> <Recursion on list with id=36340368>,
> 'class'],
> [ <Recursion on list with id=36340368>,
> <Recursion on list with id=36340368>,
> 'class'],
> None],
> [ [ <Recursion on list with id=36340872>,
> <Recursion on list with id=36340872>,
> None],
> [ <Recursion on list with id=36340872>,
> <Recursion on list with id=36340872>,
> None],
> 'class'],
> [ [ <Recursion on list with id=36364872>,
> <Recursion on list with id=36364872>,
> 'class'],
> [ <Recursion on list with id=36364872>,
> <Recursion on list with id=36364872>,
> 'class'],
> None],
> [ [ <Recursion on list with id=36365520>,
> <Recursion on list with id=36365520>,
> None],
> [ <Recursion on list with id=36365520>,
> <Recursion on list with id=36365520>,
> None],
> 'class'],
> [ [ <Recursion on list with id=36457216>,
> <Recursion on list with id=36457216>,
> 'class'],
> [ <Recursion on list with id=36457216>,
> <Recursion on list with id=36457216>,
> 'class'],
> None],
> [ [ <Recursion on list with id=36458008>,
> <Recursion on list with id=36458008>,
> None],
> [ <Recursion on list with id=36458008>,
> <Recursion on list with id=36458008>,
> None],
> 'class'],
> [ [ <Recursion on list with id=36460520>,
> <Recursion on list with id=36460520>,
> 'class'],
> [ <Recursion on list with id=36460520>,
> <Recursion on list with id=36460520>,
> 'class'],
> None],
> [ [ <Recursion on list with id=36461168>,
> <Recursion on list with id=36461168>,
> None],
> [ <Recursion on list with id=36461168>,
> <Recursion on list with id=36461168>,
> None],
> 'class'],
> [ [ <Recursion on list with id=36562344>,
> <Recursion on list with id=36562344>,
> 'class'],
> [ <Recursion on list with id=36562344>,
> <Recursion on list with id=36562344>,
> 'class'],
> None],
> [ [ <Recursion on list with id=36563280>,
> <Recursion on list with id=36563280>,
> None],
> [ <Recursion on list with id=36563280>,
> <Recursion on list with id=36563280>,
> None],
> 'class'],
> <Element 'memory' at 0x2364710>,
> { '_children': [],
> 'attrib': { 'unit': 'KiB'},
> 'tag': 'memory',
> 'tail': '\n ',
> 'text': '4095476'},
> [],
> <Element 'pages' at 0x2364750>,
> { '_children': [],
> 'attrib': { 'size': '4', 'unit': 'KiB'},
> 'tag': 'pages',
> 'tail': '\n ',
> 'text': '1023869'},
> [],
> <Element 'pages' at 0x2364790>,
> { '_children': [],
> 'attrib': { 'size': '2048', 'unit': 'KiB'},
> 'tag': 'pages',
> 'tail': '\n ',
> 'text': '0'},
> [],
> <Element 'distances' at 0x23647d0>,
> { '_children': [<Element 'sibling' at 0x2364810>],
> 'attrib': { },
> 'tag': 'distances',
> 'tail': '\n ',
> 'text': '\n '},
> [<Element 'sibling' at 0x2364810>],
> <Element 'sibling' at 0x2364810>,
> { '_children': [],
> 'attrib': { 'id': '0', 'value': '10'},
> 'tag': 'sibling',
> 'tail': '\n '},
> [],
> <Element 'cpus' at 0x2364850>,
> { '_children': [ <Element 'cpu' at 0x2364890>,
> <Element 'cpu' at 0x23648d0>,
> <Element 'cpu' at 0x2364910>,
> <Element 'cpu' at 0x2364950>,
> <Element 'cpu' at 0x2364990>,
> <Element 'cpu' at 0x23649d0>,
> <Element 'cpu' at 0x2364a10>,
> <Element 'cpu' at 0x2364a50>],
> 'attrib': { 'num': '8'},
> 'tag': 'cpus',
> 'tail': '\n ',
> 'text': '\n '},
> [ <Element 'cpu' at 0x2364890>,
> <Element 'cpu' at 0x23648d0>,
> <Element 'cpu' at 0x2364910>,
> <Element 'cpu' at 0x2364950>,
> <Element 'cpu' at 0x2364990>,
> <Element 'cpu' at 0x23649d0>,
> <Element 'cpu' at 0x2364a10>,
> <Element 'cpu' at 0x2364a50>],
> <Element 'cpu' at 0x2364890>,
> { '_children': [],
> 'attrib': { 'core_id': '0',
> 'id': '0',
> 'siblings': '0',
> 'socket_id': '0'},
> 'tag': 'cpu',
> 'tail': '\n '},
> [],
> <Element 'cpu' at 0x23648d0>,
> { '_children': [],
> 'attrib': { 'core_id': '0',
> 'id': '1',
> 'siblings': '1',
> 'socket_id': '1'},
> 'tag': 'cpu',
> 'tail': '\n '},
> [],
> <Element 'cpu' at 0x2364910>,
> { '_children': [],
> 'attrib': { 'core_id': '0',
> 'id': '2',
> 'siblings': '2',
> 'socket_id': '2'},
> 'tag': 'cpu',
> 'tail': '\n '},
> [],
> <Element 'cpu' at 0x2364950>,
> { '_children': [],
> 'attrib': { 'core_id': '0',
> 'id': '3',
> 'siblings': '3',
> 'socket_id': '3'},
> 'tag': 'cpu',
> 'tail': '\n '},
> [],
> <Element 'cpu' at 0x2364990>,
> { '_children': [],
> 'attrib': { 'core_id': '0',
> 'id': '4',
> 'siblings': '4',
> 'socket_id': '4'},
> 'tag': 'cpu',
> 'tail': '\n '},
> [],
> <Element 'cpu' at 0x23649d0>,
> { '_children': [],
> 'attrib': { 'core_id': '0',
> 'id': '5',
> 'siblings': '5',
> 'socket_id': '5'},
> 'tag': 'cpu',
> 'tail': '\n '},
> [],
> <Element 'cpu' at 0x2364a10>,
> { '_children': [],
> 'attrib': { 'core_id': '0',
> 'id': '6',
> 'siblings': '6',
> 'socket_id': '6'},
> 'tag': 'cpu',
> 'tail': '\n '},
> [],
> <Element 'cpu' at 0x2364a50>,
> { '_children': [],
> 'attrib': { 'core_id': '0',
> 'id': '7',
> 'siblings': '7',
> 'socket_id': '7'},
> 'tag': 'cpu',
> 'tail': '\n '},
> [],
> <Element 'secmodel' at 0x2364a90>,
> { '_children': [ <Element 'model' at 0x2364ad0>,
> <Element 'doi' at 0x2364b10>,
> <Element 'baselabel' at 0x2364b50>,
> <Element 'baselabel' at 0x2364b90>],
> 'attrib': { },
> 'tag': 'secmodel',
> 'tail': '\n ',
> 'text': '\n '},
> [ <Element 'model' at 0x2364ad0>,
> <Element 'doi' at 0x2364b10>,
> <Element 'baselabel' at 0x2364b50>,
> <Element 'baselabel' at 0x2364b90>],
> <Element 'model' at 0x2364ad0>,
> { '_children': [],
> 'attrib': { },
> 'tag': 'model',
> 'tail': '\n ',
> 'text': 'selinux'},
> [],
> <Element 'doi' at 0x2364b10>,
> { '_children': [],
> 'attrib': { },
> 'tag': 'doi',
> 'tail': '\n ',
> 'text': '0'},
> [],
> <Element 'baselabel' at 0x2364b50>,
> { '_children': [],
> 'attrib': { 'type': 'kvm'},
> 'tag': 'baselabel',
> 'tail': '\n ',
> 'text': 'system_u:system_r:svirt_t:s0'},
> [],
> <Element 'baselabel' at 0x2364b90>,
> { '_children': [],
> 'attrib': { 'type': 'qemu'},
> 'tag': 'baselabel',
> 'tail': '\n ',
> 'text': 'system_u:system_r:svirt_tcg_t:s0'},
> [],
> <Element 'secmodel' at 0x2364bd0>,
> { '_children': [ <Element 'model' at 0x2364c10>,
> <Element 'doi' at 0x2364c50>,
> <Element 'baselabel' at 0x2364c90>,
> <Element 'baselabel' at 0x2364cd0>],
> 'attrib': { },
> 'tag': 'secmodel',
> 'tail': '\n ',
> 'text': '\n '},
> [ <Element 'model' at 0x2364c10>,
> <Element 'doi' at 0x2364c50>,
> <Element 'baselabel' at 0x2364c90>,
> <Element 'baselabel' at 0x2364cd0>],
> <Element 'model' at 0x2364c10>,
> { '_children': [],
> 'attrib': { },
> 'tag': 'model',
> 'tail': '\n ',
> 'text': 'dac'},
> [],
> <Element 'doi' at 0x2364c50>,
> { '_children': [],
> 'attrib': { },
> 'tag': 'doi',
> 'tail': '\n ',
> 'text': '0'},
> [],
> <Element 'baselabel' at 0x2364c90>,
> { '_children': [],
> 'attrib': { 'type': 'kvm'},
> 'tag': 'baselabel',
> 'tail': '\n ',
> 'text': '+107:+107'},
> [],
> <Element 'baselabel' at 0x2364cd0>,
> { '_children': [],
> 'attrib': { 'type': 'qemu'},
> 'tag': 'baselabel',
> 'tail': '\n ',
> 'text': '+107:+107'},
> [],
> <Element 'guest' at 0x2364d10>,
> { '_children': [ <Element 'os_type' at 0x2364d50>,
> <Element 'arch' at 0x2364d90>,
> <Element 'features' at 0x2369390>],
> 'attrib': { },
> 'tag': 'guest',
> 'tail': '\n\n ',
> 'text': '\n '},
> [ <Element 'os_type' at 0x2364d50>,
> <Element 'arch' at 0x2364d90>,
> <Element 'features' at 0x2369390>],
> <Element 'os_type' at 0x2364d50>,
> { '_children': [],
> 'attrib': { },
> 'tag': 'os_type',
> 'tail': '\n ',
> 'text': 'hvm'},
> [],
> <Element 'arch' at 0x2364d90>,
> { '_children': [ <Element 'wordsize' at 0x2364dd0>,
> <Element 'emulator' at 0x2364e10>,
> <Element 'machine' at 0x2364e90>,
> <Element 'machine' at 0x2364ed0>,
> <Element 'machine' at 0x2364f10>,
> <Element 'machine' at 0x2364f50>,
> <Element 'machine' at 0x2364f90>,
> <Element 'machine' at 0x2364fd0>,
> <Element 'machine' at 0x2369050>,
> <Element 'machine' at 0x2369090>,
> <Element 'machine' at 0x23690d0>,
> <Element 'machine' at 0x2369110>,
> <Element 'machine' at 0x2369150>,
> <Element 'machine' at 0x2369190>,
> <Element 'machine' at 0x23691d0>,
> <Element 'machine' at 0x2369210>,
> <Element 'machine' at 0x2369250>,
> <Element 'domain' at 0x2369290>,
> <Element 'domain' at 0x23692d0>],
> 'attrib': { 'name': 'i686'},
> 'tag': 'arch',
> 'tail': '\n ',
> 'text': '\n '},
> [ <Element 'wordsize' at 0x2364dd0>,
> <Element 'emulator' at 0x2364e10>,
> <Element 'machine' at 0x2364e90>,
> <Element 'machine' at 0x2364ed0>,
> <Element 'machine' at 0x2364f10>,
> <Element 'machine' at 0x2364f50>,
> <Element 'machine' at 0x2364f90>,
> <Element 'machine' at 0x2364fd0>,
> <Element 'machine' at 0x2369050>,
> <Element 'machine' at 0x2369090>,
> <Element 'machine' at 0x23690d0>,
> <Element 'machine' at 0x2369110>,
> <Element 'machine' at 0x2369150>,
> <Element 'machine' at 0x2369190>,
> <Element 'machine' at 0x23691d0>,
> <Element 'machine' at 0x2369210>,
> <Element 'machine' at 0x2369250>,
> <Element 'domain' at 0x2369290>,
> <Element 'domain' at 0x23692d0>],
> <Element 'wordsize' at 0x2364dd0>,
> { '_children': [],
> 'attrib': { },
> 'tag': 'wordsize',
> 'tail': '\n ',
> 'text': '32'},
> [],
> <Element 'emulator' at 0x2364e10>,
> { '_children': [],
> 'attrib': { },
> 'tag': 'emulator',
> 'tail': '\n ',
> 'text': '/usr/libexec/qemu-kvm'},
> [],
> <Element 'machine' at 0x2364e90>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'pc-i440fx-rhel7.2.0'},
> [],
> <Element 'machine' at 0x2364ed0>,
> { '_children': [],
> 'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'pc'},
> [],
> <Element 'machine' at 0x2364f10>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'pc-i440fx-rhel7.0.0'},
> [],
> <Element 'machine' at 0x2364f50>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'pc-q35-rhel7.1.0'},
> [],
> <Element 'machine' at 0x2364f90>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'rhel6.3.0'},
> [],
> <Element 'machine' at 0x2364fd0>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'pc-q35-rhel7.2.0'},
> [],
> <Element 'machine' at 0x2369050>,
> { '_children': [],
> 'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'q35'},
> [],
> <Element 'machine' at 0x2369090>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'rhel6.4.0'},
> [],
> <Element 'machine' at 0x23690d0>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'rhel6.0.0'},
> [],
> <Element 'machine' at 0x2369110>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'pc-i440fx-rhel7.1.0'},
> [],
> <Element 'machine' at 0x2369150>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'rhel6.5.0'},
> [],
> <Element 'machine' at 0x2369190>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'rhel6.6.0'},
> [],
> <Element 'machine' at 0x23691d0>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'rhel6.1.0'},
> [],
> <Element 'machine' at 0x2369210>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'pc-q35-rhel7.0.0'},
> [],
> <Element 'machine' at 0x2369250>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'rhel6.2.0'},
> [],
> <Element 'domain' at 0x2369290>,
> { '_children': [],
> 'attrib': { 'type': 'qemu'},
> 'tag': 'domain',
> 'tail': '\n '},
> [],
> <Element 'domain' at 0x23692d0>,
> { '_children': [<Element 'emulator' at 0x2369310>],
> 'attrib': { 'type': 'kvm'},
> 'tag': 'domain',
> 'tail': '\n ',
> 'text': '\n '},
> [<Element 'emulator' at 0x2369310>],
> <Element 'emulator' at 0x2369310>,
> { '_children': [],
> 'attrib': { },
> 'tag': 'emulator',
> 'tail': '\n ',
> 'text': '/usr/libexec/qemu-kvm'},
> [],
> <Element 'features' at 0x2369390>,
> { '_children': [ <Element 'cpuselection' at 0x23693d0>,
> <Element 'deviceboot' at 0x2369410>,
> <Element 'disksnapshot' at 0x2369450>,
> <Element 'acpi' at 0x2369490>,
> <Element 'apic' at 0x23694d0>,
> <Element 'pae' at 0x2369510>,
> <Element 'nonpae' at 0x2369550>],
> 'attrib': { },
> 'tag': 'features',
> 'tail': '\n ',
> 'text': '\n '},
> [ <Element 'cpuselection' at 0x23693d0>,
> <Element 'deviceboot' at 0x2369410>,
> <Element 'disksnapshot' at 0x2369450>,
> <Element 'acpi' at 0x2369490>,
> <Element 'apic' at 0x23694d0>,
> <Element 'pae' at 0x2369510>,
> <Element 'nonpae' at 0x2369550>],
> <Element 'cpuselection' at 0x23693d0>,
> { '_children': [], 'attrib': { }, 'tag': 'cpuselection',
> 'tail': '\n '},
> [],
> <Element 'deviceboot' at 0x2369410>,
> { '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail':
> '\n '},
> [],
> <Element 'disksnapshot' at 0x2369450>,
> { '_children': [],
> 'attrib': { 'default': 'on', 'toggle': 'no'},
> 'tag': 'disksnapshot',
> 'tail': '\n '},
> [],
> <Element 'acpi' at 0x2369490>,
> { '_children': [],
> 'attrib': { 'default': 'on', 'toggle': 'yes'},
> 'tag': 'acpi',
> 'tail': '\n '},
> [],
> <Element 'apic' at 0x23694d0>,
> { '_children': [],
> 'attrib': { 'default': 'on', 'toggle': 'no'},
> 'tag': 'apic',
> 'tail': '\n '},
> [],
> <Element 'pae' at 0x2369510>,
> { '_children': [], 'attrib': { }, 'tag': 'pae', 'tail': '\n '},
> [],
> <Element 'nonpae' at 0x2369550>,
> { '_children': [], 'attrib': { }, 'tag': 'nonpae', 'tail': '\n '},
> [],
> <Element 'guest' at 0x2369590>,
> { '_children': [ <Element 'os_type' at 0x23695d0>,
> <Element 'arch' at 0x2369610>,
> <Element 'features' at 0x2369bd0>],
> 'attrib': { },
> 'tag': 'guest',
> 'tail': '\n\n',
> 'text': '\n '},
> [ <Element 'os_type' at 0x23695d0>,
> <Element 'arch' at 0x2369610>,
> <Element 'features' at 0x2369bd0>],
> <Element 'os_type' at 0x23695d0>,
> { '_children': [],
> 'attrib': { },
> 'tag': 'os_type',
> 'tail': '\n ',
> 'text': 'hvm'},
> [],
> <Element 'arch' at 0x2369610>,
> { '_children': [ <Element 'wordsize' at 0x2369650>,
> <Element 'emulator' at 0x2369690>,
> <Element 'machine' at 0x2369710>,
> <Element 'machine' at 0x2369750>,
> <Element 'machine' at 0x2369790>,
> <Element 'machine' at 0x23697d0>,
> <Element 'machine' at 0x2369810>,
> <Element 'machine' at 0x2369850>,
> <Element 'machine' at 0x2369890>,
> <Element 'machine' at 0x23698d0>,
> <Element 'machine' at 0x2369910>,
> <Element 'machine' at 0x2369950>,
> <Element 'machine' at 0x2369990>,
> <Element 'machine' at 0x23699d0>,
> <Element 'machine' at 0x2369a10>,
> <Element 'machine' at 0x2369a50>,
> <Element 'machine' at 0x2369a90>,
> <Element 'domain' at 0x2369ad0>,
> <Element 'domain' at 0x2369b10>],
> 'attrib': { 'name': 'x86_64'},
> 'tag': 'arch',
> 'tail': '\n ',
> 'text': '\n '},
> [ <Element 'wordsize' at 0x2369650>,
> <Element 'emulator' at 0x2369690>,
> <Element 'machine' at 0x2369710>,
> <Element 'machine' at 0x2369750>,
> <Element 'machine' at 0x2369790>,
> <Element 'machine' at 0x23697d0>,
> <Element 'machine' at 0x2369810>,
> <Element 'machine' at 0x2369850>,
> <Element 'machine' at 0x2369890>,
> <Element 'machine' at 0x23698d0>,
> <Element 'machine' at 0x2369910>,
> <Element 'machine' at 0x2369950>,
> <Element 'machine' at 0x2369990>,
> <Element 'machine' at 0x23699d0>,
> <Element 'machine' at 0x2369a10>,
> <Element 'machine' at 0x2369a50>,
> <Element 'machine' at 0x2369a90>,
> <Element 'domain' at 0x2369ad0>,
> <Element 'domain' at 0x2369b10>],
> <Element 'wordsize' at 0x2369650>,
> { '_children': [],
> 'attrib': { },
> 'tag': 'wordsize',
> 'tail': '\n ',
> 'text': '64'},
> [],
> <Element 'emulator' at 0x2369690>,
> { '_children': [],
> 'attrib': { },
> 'tag': 'emulator',
> 'tail': '\n ',
> 'text': '/usr/libexec/qemu-kvm'},
> [],
> <Element 'machine' at 0x2369710>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'pc-i440fx-rhel7.2.0'},
> [],
> <Element 'machine' at 0x2369750>,
> { '_children': [],
> 'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'pc'},
> [],
> <Element 'machine' at 0x2369790>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'pc-i440fx-rhel7.0.0'},
> [],
> <Element 'machine' at 0x23697d0>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'pc-q35-rhel7.1.0'},
> [],
> <Element 'machine' at 0x2369810>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'rhel6.3.0'},
> [],
> <Element 'machine' at 0x2369850>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'pc-q35-rhel7.2.0'},
> [],
> <Element 'machine' at 0x2369890>,
> { '_children': [],
> 'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'q35'},
> [],
> <Element 'machine' at 0x23698d0>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'rhel6.4.0'},
> [],
> <Element 'machine' at 0x2369910>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'rhel6.0.0'},
> [],
> <Element 'machine' at 0x2369950>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'pc-i440fx-rhel7.1.0'},
> [],
> <Element 'machine' at 0x2369990>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'rhel6.5.0'},
> [],
> <Element 'machine' at 0x23699d0>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'rhel6.6.0'},
> [],
> <Element 'machine' at 0x2369a10>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'rhel6.1.0'},
> [],
> <Element 'machine' at 0x2369a50>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'pc-q35-rhel7.0.0'},
> [],
> <Element 'machine' at 0x2369a90>,
> { '_children': [],
> 'attrib': { 'maxCpus': '240'},
> 'tag': 'machine',
> 'tail': '\n ',
> 'text': 'rhel6.2.0'},
> [],
> <Element 'domain' at 0x2369ad0>,
> { '_children': [],
> 'attrib': { 'type': 'qemu'},
> 'tag': 'domain',
> 'tail': '\n '},
> [],
> <Element 'domain' at 0x2369b10>,
> { '_children': [<Element 'emulator' at 0x2369b50>],
> 'attrib': { 'type': 'kvm'},
> 'tag': 'domain',
> 'tail': '\n ',
> 'text': '\n '},
> [<Element 'emulator' at 0x2369b50>],
> <Element 'emulator' at 0x2369b50>,
> { '_children': [],
> 'attrib': { },
> 'tag': 'emulator',
> 'tail': '\n ',
> 'text': '/usr/libexec/qemu-kvm'},
> [],
> <Element 'features' at 0x2369bd0>,
> { '_children': [ <Element 'cpuselection' at 0x2369c10>,
> <Element 'deviceboot' at 0x2369c50>,
> <Element 'disksnapshot' at 0x2369c90>,
> <Element 'acpi' at 0x2369cd0>,
> <Element 'apic' at 0x2369d10>],
> 'attrib': { },
> 'tag': 'features',
> 'tail': '\n ',
> 'text': '\n '},
> [ <Element 'cpuselection' at 0x2369c10>,
> <Element 'deviceboot' at 0x2369c50>,
> <Element 'disksnapshot' at 0x2369c90>,
> <Element 'acpi' at 0x2369cd0>,
> <Element 'apic' at 0x2369d10>],
> <Element 'cpuselection' at 0x2369c10>,
> { '_children': [], 'attrib': { }, 'tag': 'cpuselection',
> 'tail': '\n '},
> [],
> <Element 'deviceboot' at 0x2369c50>,
> { '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail':
> '\n '},
> [],
> <Element 'disksnapshot' at 0x2369c90>,
> { '_children': [],
> 'attrib': { 'default': 'on', 'toggle': 'no'},
> 'tag': 'disksnapshot',
> 'tail': '\n '},
> [],
> <Element 'acpi' at 0x2369cd0>,
> { '_children': [],
> 'attrib': { 'default': 'on', 'toggle': 'yes'},
> 'tag': 'acpi',
> 'tail': '\n '},
> [],
> <Element 'apic' at 0x2369d10>,
> { '_children': [],
> 'attrib': { 'default': 'on', 'toggle': 'no'},
> 'tag': 'apic',
> 'tail': '\n '},
> [],
> [],
> (<type '_ctypes.Array'>,),
> { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>,
> '__doc__': None,
> '__module__': 'vdsm.netlink',
> '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>,
> '_length_': 60,
> '_type_': <class 'ctypes.c_char'>,
> 'raw': <attribute 'raw' of 'c_char_Array_60' objects>,
> 'value': <attribute 'value' of 'c_char_Array_60' objects>},
> <class 'vdsm.netlink.c_char_Array_60'>,
> <attribute '__dict__' of 'c_char_Array_60' objects>,
> <attribute '__weakref__' of 'c_char_Array_60' objects>,
> ( <class 'vdsm.netlink.c_char_Array_60'>,
> <type '_ctypes.Array'>,
> <type '_ctypes._CData'>,
> <type 'object'>),
> <attribute 'raw' of 'c_char_Array_60' objects>,
> <attribute 'value' of 'c_char_Array_60' objects>,
> <xml.etree.ElementTree.XMLParser object at 0x2357f50>,
> { '_doctype': None,
> '_error': <class 'xml.parsers.expat.ExpatError'>,
> '_names': { u'acpi': 'acpi',
> u'apic': 'apic',
> u'arch': 'arch',
> u'baselabel': 'baselabel',
> u'canonical': 'canonical',
> u'capabilities': 'capabilities',
> u'cell': 'cell',
> u'cells': 'cells',
> u'core_id': 'core_id',
> u'cores': 'cores',
> u'cpu': 'cpu',
> u'cpus': 'cpus',
> u'cpuselection': 'cpuselection',
> u'default': 'default',
> u'deviceboot': 'deviceboot',
> u'disksnapshot': 'disksnapshot',
> u'distances': 'distances',
> u'doi': 'doi',
> u'domain': 'domain',
> u'emulator': 'emulator',
> u'feature': 'feature',
> u'features': 'features',
> u'guest': 'guest',
> u'host': 'host',
> u'id': 'id',
> u'live': 'live',
> u'machine': 'machine',
> u'maxCpus': 'maxCpus',
> u'memory': 'memory',
> u'migration_features': 'migration_features',
> u'model': 'model',
> u'name': 'name',
> u'nonpae': 'nonpae',
> u'num': 'num',
> u'os_type': 'os_type',
> u'pae': 'pae',
> u'pages': 'pages',
> u'power_management': 'power_management',
> u'secmodel': 'secmodel',
> u'sibling': 'sibling',
> u'siblings': 'siblings',
> u'size': 'size',
> u'socket_id': 'socket_id',
> u'sockets': 'sockets',
> u'suspend_disk': 'suspend_disk',
> u'suspend_hybrid': 'suspend_hybrid',
> u'suspend_mem': 'suspend_mem',
> u'threads': 'threads',
> u'toggle': 'toggle',
> u'topology': 'topology',
> u'type': 'type',
> u'unit': 'unit',
> u'uri_transport': 'uri_transport',
> u'uri_transports': 'uri_transports',
> u'uuid': 'uuid',
> u'value': 'value',
> u'vendor': 'vendor',
> u'wordsize': 'wordsize'},
> '_target': <xml.etree.ElementTree.TreeBuilder object at 0x2357f10>,
> 'entity': { },
> 'parser': <pyexpat.xmlparser object at 0x22e5360>,
> 'version': 'Expat 2.1.0'},
> <pyexpat.xmlparser object at 0x22e5360>,
> <xml.etree.ElementTree.TreeBuilder object at 0x2357f10>,
> <bound method XMLParser._start_list of
> <xml.etree.ElementTree.XMLParser object at 0x2357f50>>,
> <bound method XMLParser._end of <xml.etree.ElementTree.XMLParser
> object at 0x2357f50>>,
> <bound method XMLParser._pi of <xml.etree.ElementTree.XMLParser
> object at 0x2357f50>>,
> <bound method XMLParser._data of <xml.etree.ElementTree.XMLParser
> object at 0x2357f50>>,
> <bound method XMLParser._comment of
> <xml.etree.ElementTree.XMLParser object at 0x2357f50>>,
> <bound method XMLParser._default of
> <xml.etree.ElementTree.XMLParser object at 0x2357f50>>,
> { '_data': [],
> '_elem': [],
> '_factory': <class 'xml.etree.ElementTree.Element'>,
> '_last': <Element 'capabilities' at 0x2357f90>,
> '_tail': 1},
> [],
> <Element 'cell' at 0x23646d0>,
> <Element 'cells' at 0x2364690>,
> <Element 'topology' at 0x2364650>,
> <Element 'host' at 0x2357fd0>,
> <Element 'capabilities' at 0x2357f90>,
> { '_children': [ <Element 'memory' at 0x2364710>,
> <Element 'pages' at 0x2364750>,
> <Element 'pages' at 0x2364790>,
> <Element 'distances' at 0x23647d0>,
> <Element 'cpus' at 0x2364850>],
> 'attrib': { 'id': '0'},
> 'tag': 'cell',
> 'tail': '\n ',
> 'text': '\n '},
> { '_children': [<Element 'cell' at 0x23646d0>],
> 'attrib': { 'num': '1'},
> 'tag': 'cells',
> 'tail': '\n ',
> 'text': '\n '},
> { '_children': [<Element 'cells' at 0x2364690>],
> 'attrib': { },
> 'tag': 'topology',
> 'tail': '\n ',
> 'text': '\n '},
> { '_children': [ <Element 'uuid' at 0x2364050>,
> <Element 'cpu' at 0x2364090>,
> <Element 'power_management' at 0x2364390>,
> <Element 'migration_features' at 0x23644d0>,
> <Element 'topology' at 0x2364650>,
> <Element 'secmodel' at 0x2364a90>,
> <Element 'secmodel' at 0x2364bd0>],
> 'attrib': { },
> 'tag': 'host',
> 'tail': '\n\n ',
> 'text': '\n '},
> { '_children': [ <Element 'host' at 0x2357fd0>,
> <Element 'guest' at 0x2364d10>,
> <Element 'guest' at 0x2369590>],
> 'attrib': { },
> 'tag': 'capabilities',
> 'text': '\n\n '},
> [ <Element 'memory' at 0x2364710>,
> <Element 'pages' at 0x2364750>,
> <Element 'pages' at 0x2364790>,
> <Element 'distances' at 0x23647d0>,
> <Element 'cpus' at 0x2364850>],
> [<Element 'cell' at 0x23646d0>],
> [<Element 'cells' at 0x2364690>],
> [ <Element 'uuid' at 0x2364050>,
> <Element 'cpu' at 0x2364090>,
> <Element 'power_management' at 0x2364390>,
> <Element 'migration_features' at 0x23644d0>,
> <Element 'topology' at 0x2364650>,
> <Element 'secmodel' at 0x2364a90>,
> <Element 'secmodel' at 0x2364bd0>],
> [ <Element 'host' at 0x2357fd0>,
> <Element 'guest' at 0x2364d10>,
> <Element 'guest' at 0x2369590>],
> <Element 'migration_features' at 0x23644d0>,
> <Element 'power_management' at 0x2364390>,
> <Element 'cpu' at 0x2364090>,
> <Element 'uuid' at 0x2364050>,
> { '_children': [ <Element 'live' at 0x2364510>,
> <Element 'uri_transports' at 0x2364590>],
> 'attrib': { },
> 'tag': 'migration_features',
> 'tail': '\n ',
> 'text': '\n '},
> { '_children': [ <Element 'suspend_mem' at 0x23643d0>,
> <Element 'suspend_disk' at 0x2364410>,
> <Element 'suspend_hybrid' at 0x2364490>],
> 'attrib': { },
> 'tag': 'power_management',
> 'tail': '\n ',
> 'text': '\n '},
> { '_children': [ <Element 'arch' at 0x23640d0>,
> <Element 'model' at 0x2364110>,
> <Element 'vendor' at 0x2364150>,
> <Element 'topology' at 0x2364190>,
> <Element 'feature' at 0x23641d0>,
> <Element 'feature' at 0x2364210>,
> <Element 'feature' at 0x2364250>,
> <Element 'feature' at 0x2364290>,
> <Element 'feature' at 0x23642d0>,
> <Element 'pages' at 0x2364310>,
> <Element 'pages' at 0x2364350>],
> 'attrib': { },
> 'tag': 'cpu',
> 'tail': '\n ',
> 'text': '\n '},
> { '_children': [],
> 'attrib': { },
> 'tag': 'uuid',
> 'tail': '\n ',
> 'text': 'a920d963-c18f-4fda-aa5a-deee9bbcde9d'},
> [<Element 'live' at 0x2364510>, <Element 'uri_transports' at 0x2364590>],
> [ <Element 'suspend_mem' at 0x23643d0>,
> <Element 'suspend_disk' at 0x2364410>,
> <Element 'suspend_hybrid' at 0x2364490>],
> [ <Element 'arch' at 0x23640d0>,
> <Element 'model' at 0x2364110>,
> <Element 'vendor' at 0x2364150>,
> <Element 'topology' at 0x2364190>,
> <Element 'feature' at 0x23641d0>,
> <Element 'feature' at 0x2364210>,
> <Element 'feature' at 0x2364250>,
> <Element 'feature' at 0x2364290>,
> <Element 'feature' at 0x23642d0>,
> <Element 'pages' at 0x2364310>,
> <Element 'pages' at 0x2364350>],
> [],
> <Element 'uri_transports' at 0x2364590>,
> <Element 'live' at 0x2364510>,
> <Element 'suspend_hybrid' at 0x2364490>,
> <Element 'suspend_disk' at 0x2364410>,
> <Element 'suspend_mem' at 0x23643d0>,
> <Element 'pages' at 0x2364350>,
> <Element 'pages' at 0x2364310>,
> <Element 'feature' at 0x23642d0>,
> <Element 'feature' at 0x2364290>,
> <Element 'feature' at 0x2364250>,
> <Element 'feature' at 0x2364210>,
> <Element 'feature' at 0x23641d0>,
> <Element 'topology' at 0x2364190>,
> <Element 'vendor' at 0x2364150>,
> <Element 'model' at 0x2364110>,
> <Element 'arch' at 0x23640d0>,
> { '_children': [ <Element 'uri_transport' at 0x23645d0>,
> <Element 'uri_transport' at 0x2364610>],
> 'attrib': { },
> 'tag': 'uri_transports',
> 'tail': '\n ',
> 'text': '\n '},
> { '_children': [], 'attrib': { }, 'tag': 'live', 'tail': '\n '},
> { '_children': [], 'attrib': { }, 'tag': 'suspend_hybrid',
> 'tail': '\n '},
> { '_children': [], 'attrib': { }, 'tag': 'suspend_disk',
> 'tail': '\n '},
> { '_children': [], 'attrib': { }, 'tag': 'suspend_mem',
> 'tail': '\n '},
> { '_children': [],
> 'attrib': { 'size': '2048', 'unit': 'KiB'},
> 'tag': 'pages',
> 'tail': '\n '},
> { '_children': [],
> 'attrib': { 'size': '4', 'unit': 'KiB'},
> 'tag': 'pages',
> 'tail': '\n '},
> { '_children': [],
> 'attrib': { 'name': 'ss'},
> 'tag': 'feature',
> 'tail': '\n '},
> { '_children': [],
> 'attrib': { 'name': 'vmx'},
> 'tag': 'feature',
> 'tail': '\n '},
> { '_children': [],
> 'attrib': { 'name': 'pcid'},
> 'tag': 'feature',
> 'tail': '\n '},
> { '_children': [],
> 'attrib': { 'name': 'osxsave'},
> 'tag': 'feature',
> 'tail': '\n '},
> { '_children': [],
> 'attrib': { 'name': 'hypervisor'},
> 'tag': 'feature',
> 'tail': '\n '},
> { '_children': [],
> 'attrib': { 'cores': '1', 'sockets': '8', 'threads': '1'},
> 'tag': 'topology',
> 'tail': '\n '},
> { '_children': [],
> 'attrib': { },
> 'tag': 'vendor',
> 'tail': '\n ',
> 'text': 'Intel'},
> { '_children': [],
> 'attrib': { },
> 'tag': 'model',
> 'tail': '\n ',
> 'text': 'IvyBridge'},
> { '_children': [],
> 'attrib': { },
> 'tag': 'arch',
> 'tail': '\n ',
> 'text': 'x86_64'},
> [ <Element 'uri_transport' at 0x23645d0>,
> <Element 'uri_transport' at 0x2364610>],
> [],
> [],
> [],
> [],
> [],
> [],
> [],
> [],
> [],
> [],
> [],
> [],
> [],
> [],
> [],
> <Element 'uri_transport' at 0x2364610>,
> <Element 'uri_transport' at 0x23645d0>,
> { '_children': [],
> 'attrib': { },
> 'tag': 'uri_transport',
> 'tail': '\n ',
> 'text': 'rdma'},
> { '_children': [],
> 'attrib': { },
> 'tag': 'uri_transport',
> 'tail': '\n ',
> 'text': 'tcp'},
> [],
> [],
> (<type '_ctypes.Array'>,),
> { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>,
> '__doc__': None,
> '__module__': 'vdsm.netlink',
> '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>,
> '_length_': 60,
> '_type_': <class 'ctypes.c_char'>,
> 'raw': <attribute 'raw' of 'c_char_Array_60' objects>,
> 'value': <attribute 'value' of 'c_char_Array_60' objects>},
> <class 'vdsm.netlink.c_char_Array_60'>,
> <attribute '__dict__' of 'c_char_Array_60' objects>,
> <attribute '__weakref__' of 'c_char_Array_60' objects>,
> ( <class 'vdsm.netlink.c_char_Array_60'>,
> <type '_ctypes.Array'>,
> <type '_ctypes._CData'>,
> <type 'object'>),
> <attribute 'raw' of 'c_char_Array_60' objects>,
> <attribute 'value' of 'c_char_Array_60' objects>]
> _______________________________________________
> Devel mailing list
> Devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
More information about the Devel
mailing list