Re: [ovirt-devel] [VDSM] health: Introduce Vdsm health monitoring

Here are example logs with running and stopping one vm: # tail -f /var/log/vdsm/vdsm.log | grep health Thread-13::DEBUG::2016-01-14 19:42:04,625::health::84::health::(_check) Checking health Thread-13::DEBUG::2016-01-14 19:42:04,651::health::86::health::(_check) Collected 25 objects Thread-13::WARNING::2016-01-14 19:42:04,656::health::100::health::(_check) Found 6085 uncollectible objects, 0 suspects implementing __del__: [] Vm was started here... Thread-13::DEBUG::2016-01-14 19:43:04,657::health::84::health::(_check) Checking health Thread-13::DEBUG::2016-01-14 19:43:04,688::health::86::health::(_check) Collected 1680 objects Thread-13::WARNING::2016-01-14 19:43:04,697::health::100::health::(_check) Found 11041 uncollectible objects, 0 suspects implementing __del__: [] Thread-13::DEBUG::2016-01-14 19:44:04,698::health::84::health::(_check) Checking health Thread-13::DEBUG::2016-01-14 19:44:04,729::health::86::health::(_check) Collected 57 objects Thread-13::WARNING::2016-01-14 19:44:04,737::health::100::health::(_check) Found 11096 uncollectible objects, 0 suspects implementing __del__: [] Thread-13::DEBUG::2016-01-14 19:45:04,738::health::84::health::(_check) Checking health Thread-13::DEBUG::2016-01-14 19:45:04,769::health::86::health::(_check) Collected 69 objects Thread-13::WARNING::2016-01-14 19:45:04,778::health::100::health::(_check) Found 11163 uncollectible objects, 0 suspects implementing __del__: [] Vm was shutdown here... Thread-13::DEBUG::2016-01-14 19:46:04,779::health::84::health::(_check) Checking health Thread-13::DEBUG::2016-01-14 19:46:04,809::health::86::health::(_check) Collected 2272 objects Thread-13::WARNING::2016-01-14 19:46:04,819::health::100::health::(_check) Found 13464 uncollectible objects, 28 suspects implementing __del__: ['<pthreading.Lock object at 0x7f1d8016f908>', '<pthreading.Lock object at 0x7f1d801804d0>', '<pthreading.Lock object at 0x7f1cf8010f38>', '<pthreading.Lock object at 0x7f1d8016f7a0>', '<pthreading.Lock object at 0x7f1d20003680>', '<pthreading.Lock object at 0x7f1da8112998>', '<pthreading.Lock object at 0x7f1d8016fa28>', '<pthreading.Lock object at 0x7f1da8112830>', '<pthread.Cond object at 0x7f1da811cf50>', '<pthreading.Lock object at 0x7f1d8016f200>', '<pthread.Cond object at 0x7f1da811ca50>', '<pthreading.Lock object at 0x7f1cf8010fc8>', '<pthread.Cond object at 0x7f1da8115c50>', '<pthreading.Lock object at 0x7f1da8112878>', '<pthread.Cond object at 0x7f1da811c550>', '<pthreading.Lock object at 0x7f1d8016f9e0>', '<pthread.Cond object at 0x7f1da811ca90>', '<pthreading.Lock object at 0x7f1d8016f680>', '<pthread.Cond object at 0x7f1da811cc50>', '<libvirt.virDomain object at 0x1ba8210>', '<pthreading.Lock object at 0x7f1d8016f758>', '<pthread.Cond object at 0x7f1da811cb50>', '<pthreading.Lock object at 0x7f1d8016fb90>', '<pthread.Cond object at 0x7f1da811cc10>', '<pthread.Cond object at 0x233ed50>', '<pthreading.Lock object at 0x7f1ce0227c20>', '<pthreading.Lock object at 0x7f1ce0227ab8>', '<pthread.Cond object at 0x233ed10>'] Need to test with should be fixed by https://gerrit.ovirt.org/51630, which should eliminate '<libvirt.virDomain object at 0x1ba8210>', and maybe also the pthreading objects. Nir On Thu, Jan 14, 2016 at 7:38 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > Hi all, > > Continuing the leak investigation Francesco and Milan are working on, > I posted this patch, adding leak health monitoring to vdsm [1]. This patch > currently monitor collectible objects, and we may add other interesting > stuff later if needed. > > To enable the monitor, you must set: > > [devel] > health_monitor_enable = true > > In vdsm.conf, and restart vdsm. > > Here is an example logs - this was logged 60 seconds after starting vdsm, > when running as spm: > > Thread-13::DEBUG::2016-01-14 > 18:53:43,239::health::84::health::(_check) Checking health > Thread-13::DEBUG::2016-01-14 > 18:53:43,272::health::86::health::(_check) Collected 460 objects > Thread-13::WARNING::2016-01-14 > 18:53:43,277::health::100::health::(_check) Found 5873 uncollectible > objects, 10 suspects implementing __del__: ['<pthreading.Lock object > at 0x7f012c115320>', '<pthreading.Lock object at 0x7f012c115440>', > '<pthreading.Lock object at 0x7f012c1153b0>', '<pthreading.Lock object > at 0x7f012c1154d0>', '<pthread.Cond object at 0x7f012c111350>', > '<pthread.Cond object at 0x7f012c1112d0>', '<pthread.Cond object at > 0x7f012c111250>', '<pthread.Cond object at 0x7f012c111850>', > '<pthread.Cond object at 0x7f012c1117d0>', '<pthread.Cond object at > 0x7f012c111750>'] > > We found 460 uncollectible objects, 10 of them are objects implementing __del__. > Having such object in a reference cycle make the entire cycle uncollectible. > > The pthreading objects in this log grow as vdsm continue to run, even without > any interaction with engine. The pthreading objects leak is caused by a cycle > in storage.misc.RWLock, fixed in patch [2]. > > After solving the pthreading leak, we still have many uncollectible > objects in the logs, > listing them, it seems that the suspect is hiding inside a collection > (dict, list, tuple) > and does not appear in the gc.garbage list. I will address this in later patch, > searching in the entire object graph instead of the top level of gc.garbage. > > See end of this mail for list of these objects. I printed them like this: > > Enable manhole debugging shell: > > [devel] > manhole_enable = true > > Connect to vdsm using manhole: > > nc -U /run/vdsm/vdsmd.manhole > > Print garbage: > >>>> import pprint >>>> import gc >>>> with open('/tmp/garbage.txt', 'w') as f: > pprint.pprint(gc.garbage, stream=f, indent=4) > > These objects looks related to libvirt - Francesco, can you take a look? > > [1] https://gerrit.ovirt.org/51708 > [2] https://gerrit.ovirt.org/51868 > > Nir > > --- garbage after patch [2] --- > > [ (<type '_ctypes.Array'>,), > { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>, > '__doc__': None, > '__module__': 'vdsm.netlink', > '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>, > '_length_': 60, > '_type_': <class 'ctypes.c_char'>, > 'raw': <attribute 'raw' of 'c_char_Array_60' objects>, > 'value': <attribute 'value' of 'c_char_Array_60' objects>}, > <class 'vdsm.netlink.c_char_Array_60'>, > <attribute '__dict__' of 'c_char_Array_60' objects>, > <attribute '__weakref__' of 'c_char_Array_60' objects>, > ( <class 'vdsm.netlink.c_char_Array_60'>, > <type '_ctypes.Array'>, > <type '_ctypes._CData'>, > <type 'object'>), > <attribute 'raw' of 'c_char_Array_60' objects>, > <attribute 'value' of 'c_char_Array_60' objects>, > <cell at 0x23662f0: bool object at 0x7f1e086c8c40>, > <cell at 0x2366328: NoneType object at 0x7f1e086dcf00>, > <cell at 0x2366360: function object at 0x233f488>, > ( <cell at 0x23662f0: bool object at 0x7f1e086c8c40>, > <cell at 0x2366328: NoneType object at 0x7f1e086dcf00>, > <cell at 0x2366360: function object at 0x233f488>), > <function wrapMethod at 0x233f488>, > [ [ <Recursion on list with id=34033464>, > <Recursion on list with id=34033464>, > 'class'], > [ <Recursion on list with id=34033464>, > <Recursion on list with id=34033464>, > 'class'], > None], > [ [ <Recursion on list with id=33979064>, > <Recursion on list with id=33979064>, > None], > [ <Recursion on list with id=33979064>, > <Recursion on list with id=33979064>, > None], > 'class'], > [ [ <Recursion on list with id=35535256>, > <Recursion on list with id=35535256>, > 'class'], > [ <Recursion on list with id=35535256>, > <Recursion on list with id=35535256>, > 'class'], > None], > [ [ <Recursion on list with id=35535832>, > <Recursion on list with id=35535832>, > None], > [ <Recursion on list with id=35535832>, > <Recursion on list with id=35535832>, > None], > 'class'], > [ [ <Recursion on list with id=35556312>, > <Recursion on list with id=35556312>, > 'class'], > [ <Recursion on list with id=35556312>, > <Recursion on list with id=35556312>, > 'class'], > None], > [ [ <Recursion on list with id=35556600>, > <Recursion on list with id=35556600>, > None], > [ <Recursion on list with id=35556600>, > <Recursion on list with id=35556600>, > None], > 'class'], > [ [ <Recursion on list with id=35556960>, > <Recursion on list with id=35556960>, > 'class'], > [ <Recursion on list with id=35556960>, > <Recursion on list with id=35556960>, > 'class'], > None], > [ [ <Recursion on list with id=35561912>, > <Recursion on list with id=35561912>, > None], > [ <Recursion on list with id=35561912>, > <Recursion on list with id=35561912>, > None], > 'class'], > [ [ <Recursion on list with id=35583904>, > <Recursion on list with id=35583904>, > 'class'], > [ <Recursion on list with id=35583904>, > <Recursion on list with id=35583904>, > 'class'], > None], > [ [ <Recursion on list with id=35584912>, > <Recursion on list with id=35584912>, > None], > [ <Recursion on list with id=35584912>, > <Recursion on list with id=35584912>, > None], > 'class'], > [ [ <Recursion on list with id=36340368>, > <Recursion on list with id=36340368>, > 'class'], > [ <Recursion on list with id=36340368>, > <Recursion on list with id=36340368>, > 'class'], > None], > [ [ <Recursion on list with id=36340872>, > <Recursion on list with id=36340872>, > None], > [ <Recursion on list with id=36340872>, > <Recursion on list with id=36340872>, > None], > 'class'], > [ [ <Recursion on list with id=36364872>, > <Recursion on list with id=36364872>, > 'class'], > [ <Recursion on list with id=36364872>, > <Recursion on list with id=36364872>, > 'class'], > None], > [ [ <Recursion on list with id=36365520>, > <Recursion on list with id=36365520>, > None], > [ <Recursion on list with id=36365520>, > <Recursion on list with id=36365520>, > None], > 'class'], > [ [ <Recursion on list with id=36457216>, > <Recursion on list with id=36457216>, > 'class'], > [ <Recursion on list with id=36457216>, > <Recursion on list with id=36457216>, > 'class'], > None], > [ [ <Recursion on list with id=36458008>, > <Recursion on list with id=36458008>, > None], > [ <Recursion on list with id=36458008>, > <Recursion on list with id=36458008>, > None], > 'class'], > [ [ <Recursion on list with id=36460520>, > <Recursion on list with id=36460520>, > 'class'], > [ <Recursion on list with id=36460520>, > <Recursion on list with id=36460520>, > 'class'], > None], > [ [ <Recursion on list with id=36461168>, > <Recursion on list with id=36461168>, > None], > [ <Recursion on list with id=36461168>, > <Recursion on list with id=36461168>, > None], > 'class'], > [ [ <Recursion on list with id=36562344>, > <Recursion on list with id=36562344>, > 'class'], > [ <Recursion on list with id=36562344>, > <Recursion on list with id=36562344>, > 'class'], > None], > [ [ <Recursion on list with id=36563280>, > <Recursion on list with id=36563280>, > None], > [ <Recursion on list with id=36563280>, > <Recursion on list with id=36563280>, > None], > 'class'], > <Element 'memory' at 0x2364710>, > { '_children': [], > 'attrib': { 'unit': 'KiB'}, > 'tag': 'memory', > 'tail': '\n ', > 'text': '4095476'}, > [], > <Element 'pages' at 0x2364750>, > { '_children': [], > 'attrib': { 'size': '4', 'unit': 'KiB'}, > 'tag': 'pages', > 'tail': '\n ', > 'text': '1023869'}, > [], > <Element 'pages' at 0x2364790>, > { '_children': [], > 'attrib': { 'size': '2048', 'unit': 'KiB'}, > 'tag': 'pages', > 'tail': '\n ', > 'text': '0'}, > [], > <Element 'distances' at 0x23647d0>, > { '_children': [<Element 'sibling' at 0x2364810>], > 'attrib': { }, > 'tag': 'distances', > 'tail': '\n ', > 'text': '\n '}, > [<Element 'sibling' at 0x2364810>], > <Element 'sibling' at 0x2364810>, > { '_children': [], > 'attrib': { 'id': '0', 'value': '10'}, > 'tag': 'sibling', > 'tail': '\n '}, > [], > <Element 'cpus' at 0x2364850>, > { '_children': [ <Element 'cpu' at 0x2364890>, > <Element 'cpu' at 0x23648d0>, > <Element 'cpu' at 0x2364910>, > <Element 'cpu' at 0x2364950>, > <Element 'cpu' at 0x2364990>, > <Element 'cpu' at 0x23649d0>, > <Element 'cpu' at 0x2364a10>, > <Element 'cpu' at 0x2364a50>], > 'attrib': { 'num': '8'}, > 'tag': 'cpus', > 'tail': '\n ', > 'text': '\n '}, > [ <Element 'cpu' at 0x2364890>, > <Element 'cpu' at 0x23648d0>, > <Element 'cpu' at 0x2364910>, > <Element 'cpu' at 0x2364950>, > <Element 'cpu' at 0x2364990>, > <Element 'cpu' at 0x23649d0>, > <Element 'cpu' at 0x2364a10>, > <Element 'cpu' at 0x2364a50>], > <Element 'cpu' at 0x2364890>, > { '_children': [], > 'attrib': { 'core_id': '0', > 'id': '0', > 'siblings': '0', > 'socket_id': '0'}, > 'tag': 'cpu', > 'tail': '\n '}, > [], > <Element 'cpu' at 0x23648d0>, > { '_children': [], > 'attrib': { 'core_id': '0', > 'id': '1', > 'siblings': '1', > 'socket_id': '1'}, > 'tag': 'cpu', > 'tail': '\n '}, > [], > <Element 'cpu' at 0x2364910>, > { '_children': [], > 'attrib': { 'core_id': '0', > 'id': '2', > 'siblings': '2', > 'socket_id': '2'}, > 'tag': 'cpu', > 'tail': '\n '}, > [], > <Element 'cpu' at 0x2364950>, > { '_children': [], > 'attrib': { 'core_id': '0', > 'id': '3', > 'siblings': '3', > 'socket_id': '3'}, > 'tag': 'cpu', > 'tail': '\n '}, > [], > <Element 'cpu' at 0x2364990>, > { '_children': [], > 'attrib': { 'core_id': '0', > 'id': '4', > 'siblings': '4', > 'socket_id': '4'}, > 'tag': 'cpu', > 'tail': '\n '}, > [], > <Element 'cpu' at 0x23649d0>, > { '_children': [], > 'attrib': { 'core_id': '0', > 'id': '5', > 'siblings': '5', > 'socket_id': '5'}, > 'tag': 'cpu', > 'tail': '\n '}, > [], > <Element 'cpu' at 0x2364a10>, > { '_children': [], > 'attrib': { 'core_id': '0', > 'id': '6', > 'siblings': '6', > 'socket_id': '6'}, > 'tag': 'cpu', > 'tail': '\n '}, > [], > <Element 'cpu' at 0x2364a50>, > { '_children': [], > 'attrib': { 'core_id': '0', > 'id': '7', > 'siblings': '7', > 'socket_id': '7'}, > 'tag': 'cpu', > 'tail': '\n '}, > [], > <Element 'secmodel' at 0x2364a90>, > { '_children': [ <Element 'model' at 0x2364ad0>, > <Element 'doi' at 0x2364b10>, > <Element 'baselabel' at 0x2364b50>, > <Element 'baselabel' at 0x2364b90>], > 'attrib': { }, > 'tag': 'secmodel', > 'tail': '\n ', > 'text': '\n '}, > [ <Element 'model' at 0x2364ad0>, > <Element 'doi' at 0x2364b10>, > <Element 'baselabel' at 0x2364b50>, > <Element 'baselabel' at 0x2364b90>], > <Element 'model' at 0x2364ad0>, > { '_children': [], > 'attrib': { }, > 'tag': 'model', > 'tail': '\n ', > 'text': 'selinux'}, > [], > <Element 'doi' at 0x2364b10>, > { '_children': [], > 'attrib': { }, > 'tag': 'doi', > 'tail': '\n ', > 'text': '0'}, > [], > <Element 'baselabel' at 0x2364b50>, > { '_children': [], > 'attrib': { 'type': 'kvm'}, > 'tag': 'baselabel', > 'tail': '\n ', > 'text': 'system_u:system_r:svirt_t:s0'}, > [], > <Element 'baselabel' at 0x2364b90>, > { '_children': [], > 'attrib': { 'type': 'qemu'}, > 'tag': 'baselabel', > 'tail': '\n ', > 'text': 'system_u:system_r:svirt_tcg_t:s0'}, > [], > <Element 'secmodel' at 0x2364bd0>, > { '_children': [ <Element 'model' at 0x2364c10>, > <Element 'doi' at 0x2364c50>, > <Element 'baselabel' at 0x2364c90>, > <Element 'baselabel' at 0x2364cd0>], > 'attrib': { }, > 'tag': 'secmodel', > 'tail': '\n ', > 'text': '\n '}, > [ <Element 'model' at 0x2364c10>, > <Element 'doi' at 0x2364c50>, > <Element 'baselabel' at 0x2364c90>, > <Element 'baselabel' at 0x2364cd0>], > <Element 'model' at 0x2364c10>, > { '_children': [], > 'attrib': { }, > 'tag': 'model', > 'tail': '\n ', > 'text': 'dac'}, > [], > <Element 'doi' at 0x2364c50>, > { '_children': [], > 'attrib': { }, > 'tag': 'doi', > 'tail': '\n ', > 'text': '0'}, > [], > <Element 'baselabel' at 0x2364c90>, > { '_children': [], > 'attrib': { 'type': 'kvm'}, > 'tag': 'baselabel', > 'tail': '\n ', > 'text': '+107:+107'}, > [], > <Element 'baselabel' at 0x2364cd0>, > { '_children': [], > 'attrib': { 'type': 'qemu'}, > 'tag': 'baselabel', > 'tail': '\n ', > 'text': '+107:+107'}, > [], > <Element 'guest' at 0x2364d10>, > { '_children': [ <Element 'os_type' at 0x2364d50>, > <Element 'arch' at 0x2364d90>, > <Element 'features' at 0x2369390>], > 'attrib': { }, > 'tag': 'guest', > 'tail': '\n\n ', > 'text': '\n '}, > [ <Element 'os_type' at 0x2364d50>, > <Element 'arch' at 0x2364d90>, > <Element 'features' at 0x2369390>], > <Element 'os_type' at 0x2364d50>, > { '_children': [], > 'attrib': { }, > 'tag': 'os_type', > 'tail': '\n ', > 'text': 'hvm'}, > [], > <Element 'arch' at 0x2364d90>, > { '_children': [ <Element 'wordsize' at 0x2364dd0>, > <Element 'emulator' at 0x2364e10>, > <Element 'machine' at 0x2364e90>, > <Element 'machine' at 0x2364ed0>, > <Element 'machine' at 0x2364f10>, > <Element 'machine' at 0x2364f50>, > <Element 'machine' at 0x2364f90>, > <Element 'machine' at 0x2364fd0>, > <Element 'machine' at 0x2369050>, > <Element 'machine' at 0x2369090>, > <Element 'machine' at 0x23690d0>, > <Element 'machine' at 0x2369110>, > <Element 'machine' at 0x2369150>, > <Element 'machine' at 0x2369190>, > <Element 'machine' at 0x23691d0>, > <Element 'machine' at 0x2369210>, > <Element 'machine' at 0x2369250>, > <Element 'domain' at 0x2369290>, > <Element 'domain' at 0x23692d0>], > 'attrib': { 'name': 'i686'}, > 'tag': 'arch', > 'tail': '\n ', > 'text': '\n '}, > [ <Element 'wordsize' at 0x2364dd0>, > <Element 'emulator' at 0x2364e10>, > <Element 'machine' at 0x2364e90>, > <Element 'machine' at 0x2364ed0>, > <Element 'machine' at 0x2364f10>, > <Element 'machine' at 0x2364f50>, > <Element 'machine' at 0x2364f90>, > <Element 'machine' at 0x2364fd0>, > <Element 'machine' at 0x2369050>, > <Element 'machine' at 0x2369090>, > <Element 'machine' at 0x23690d0>, > <Element 'machine' at 0x2369110>, > <Element 'machine' at 0x2369150>, > <Element 'machine' at 0x2369190>, > <Element 'machine' at 0x23691d0>, > <Element 'machine' at 0x2369210>, > <Element 'machine' at 0x2369250>, > <Element 'domain' at 0x2369290>, > <Element 'domain' at 0x23692d0>], > <Element 'wordsize' at 0x2364dd0>, > { '_children': [], > 'attrib': { }, > 'tag': 'wordsize', > 'tail': '\n ', > 'text': '32'}, > [], > <Element 'emulator' at 0x2364e10>, > { '_children': [], > 'attrib': { }, > 'tag': 'emulator', > 'tail': '\n ', > 'text': '/usr/libexec/qemu-kvm'}, > [], > <Element 'machine' at 0x2364e90>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'pc-i440fx-rhel7.2.0'}, > [], > <Element 'machine' at 0x2364ed0>, > { '_children': [], > 'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'pc'}, > [], > <Element 'machine' at 0x2364f10>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'pc-i440fx-rhel7.0.0'}, > [], > <Element 'machine' at 0x2364f50>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'pc-q35-rhel7.1.0'}, > [], > <Element 'machine' at 0x2364f90>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'rhel6.3.0'}, > [], > <Element 'machine' at 0x2364fd0>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'pc-q35-rhel7.2.0'}, > [], > <Element 'machine' at 0x2369050>, > { '_children': [], > 'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'q35'}, > [], > <Element 'machine' at 0x2369090>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'rhel6.4.0'}, > [], > <Element 'machine' at 0x23690d0>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'rhel6.0.0'}, > [], > <Element 'machine' at 0x2369110>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'pc-i440fx-rhel7.1.0'}, > [], > <Element 'machine' at 0x2369150>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'rhel6.5.0'}, > [], > <Element 'machine' at 0x2369190>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'rhel6.6.0'}, > [], > <Element 'machine' at 0x23691d0>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'rhel6.1.0'}, > [], > <Element 'machine' at 0x2369210>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'pc-q35-rhel7.0.0'}, > [], > <Element 'machine' at 0x2369250>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'rhel6.2.0'}, > [], > <Element 'domain' at 0x2369290>, > { '_children': [], > 'attrib': { 'type': 'qemu'}, > 'tag': 'domain', > 'tail': '\n '}, > [], > <Element 'domain' at 0x23692d0>, > { '_children': [<Element 'emulator' at 0x2369310>], > 'attrib': { 'type': 'kvm'}, > 'tag': 'domain', > 'tail': '\n ', > 'text': '\n '}, > [<Element 'emulator' at 0x2369310>], > <Element 'emulator' at 0x2369310>, > { '_children': [], > 'attrib': { }, > 'tag': 'emulator', > 'tail': '\n ', > 'text': '/usr/libexec/qemu-kvm'}, > [], > <Element 'features' at 0x2369390>, > { '_children': [ <Element 'cpuselection' at 0x23693d0>, > <Element 'deviceboot' at 0x2369410>, > <Element 'disksnapshot' at 0x2369450>, > <Element 'acpi' at 0x2369490>, > <Element 'apic' at 0x23694d0>, > <Element 'pae' at 0x2369510>, > <Element 'nonpae' at 0x2369550>], > 'attrib': { }, > 'tag': 'features', > 'tail': '\n ', > 'text': '\n '}, > [ <Element 'cpuselection' at 0x23693d0>, > <Element 'deviceboot' at 0x2369410>, > <Element 'disksnapshot' at 0x2369450>, > <Element 'acpi' at 0x2369490>, > <Element 'apic' at 0x23694d0>, > <Element 'pae' at 0x2369510>, > <Element 'nonpae' at 0x2369550>], > <Element 'cpuselection' at 0x23693d0>, > { '_children': [], 'attrib': { }, 'tag': 'cpuselection', > 'tail': '\n '}, > [], > <Element 'deviceboot' at 0x2369410>, > { '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail': > '\n '}, > [], > <Element 'disksnapshot' at 0x2369450>, > { '_children': [], > 'attrib': { 'default': 'on', 'toggle': 'no'}, > 'tag': 'disksnapshot', > 'tail': '\n '}, > [], > <Element 'acpi' at 0x2369490>, > { '_children': [], > 'attrib': { 'default': 'on', 'toggle': 'yes'}, > 'tag': 'acpi', > 'tail': '\n '}, > [], > <Element 'apic' at 0x23694d0>, > { '_children': [], > 'attrib': { 'default': 'on', 'toggle': 'no'}, > 'tag': 'apic', > 'tail': '\n '}, > [], > <Element 'pae' at 0x2369510>, > { '_children': [], 'attrib': { }, 'tag': 'pae', 'tail': '\n '}, > [], > <Element 'nonpae' at 0x2369550>, > { '_children': [], 'attrib': { }, 'tag': 'nonpae', 'tail': '\n '}, > [], > <Element 'guest' at 0x2369590>, > { '_children': [ <Element 'os_type' at 0x23695d0>, > <Element 'arch' at 0x2369610>, > <Element 'features' at 0x2369bd0>], > 'attrib': { }, > 'tag': 'guest', > 'tail': '\n\n', > 'text': '\n '}, > [ <Element 'os_type' at 0x23695d0>, > <Element 'arch' at 0x2369610>, > <Element 'features' at 0x2369bd0>], > <Element 'os_type' at 0x23695d0>, > { '_children': [], > 'attrib': { }, > 'tag': 'os_type', > 'tail': '\n ', > 'text': 'hvm'}, > [], > <Element 'arch' at 0x2369610>, > { '_children': [ <Element 'wordsize' at 0x2369650>, > <Element 'emulator' at 0x2369690>, > <Element 'machine' at 0x2369710>, > <Element 'machine' at 0x2369750>, > <Element 'machine' at 0x2369790>, > <Element 'machine' at 0x23697d0>, > <Element 'machine' at 0x2369810>, > <Element 'machine' at 0x2369850>, > <Element 'machine' at 0x2369890>, > <Element 'machine' at 0x23698d0>, > <Element 'machine' at 0x2369910>, > <Element 'machine' at 0x2369950>, > <Element 'machine' at 0x2369990>, > <Element 'machine' at 0x23699d0>, > <Element 'machine' at 0x2369a10>, > <Element 'machine' at 0x2369a50>, > <Element 'machine' at 0x2369a90>, > <Element 'domain' at 0x2369ad0>, > <Element 'domain' at 0x2369b10>], > 'attrib': { 'name': 'x86_64'}, > 'tag': 'arch', > 'tail': '\n ', > 'text': '\n '}, > [ <Element 'wordsize' at 0x2369650>, > <Element 'emulator' at 0x2369690>, > <Element 'machine' at 0x2369710>, > <Element 'machine' at 0x2369750>, > <Element 'machine' at 0x2369790>, > <Element 'machine' at 0x23697d0>, > <Element 'machine' at 0x2369810>, > <Element 'machine' at 0x2369850>, > <Element 'machine' at 0x2369890>, > <Element 'machine' at 0x23698d0>, > <Element 'machine' at 0x2369910>, > <Element 'machine' at 0x2369950>, > <Element 'machine' at 0x2369990>, > <Element 'machine' at 0x23699d0>, > <Element 'machine' at 0x2369a10>, > <Element 'machine' at 0x2369a50>, > <Element 'machine' at 0x2369a90>, > <Element 'domain' at 0x2369ad0>, > <Element 'domain' at 0x2369b10>], > <Element 'wordsize' at 0x2369650>, > { '_children': [], > 'attrib': { }, > 'tag': 'wordsize', > 'tail': '\n ', > 'text': '64'}, > [], > <Element 'emulator' at 0x2369690>, > { '_children': [], > 'attrib': { }, > 'tag': 'emulator', > 'tail': '\n ', > 'text': '/usr/libexec/qemu-kvm'}, > [], > <Element 'machine' at 0x2369710>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'pc-i440fx-rhel7.2.0'}, > [], > <Element 'machine' at 0x2369750>, > { '_children': [], > 'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'pc'}, > [], > <Element 'machine' at 0x2369790>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'pc-i440fx-rhel7.0.0'}, > [], > <Element 'machine' at 0x23697d0>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'pc-q35-rhel7.1.0'}, > [], > <Element 'machine' at 0x2369810>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'rhel6.3.0'}, > [], > <Element 'machine' at 0x2369850>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'pc-q35-rhel7.2.0'}, > [], > <Element 'machine' at 0x2369890>, > { '_children': [], > 'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'q35'}, > [], > <Element 'machine' at 0x23698d0>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'rhel6.4.0'}, > [], > <Element 'machine' at 0x2369910>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'rhel6.0.0'}, > [], > <Element 'machine' at 0x2369950>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'pc-i440fx-rhel7.1.0'}, > [], > <Element 'machine' at 0x2369990>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'rhel6.5.0'}, > [], > <Element 'machine' at 0x23699d0>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'rhel6.6.0'}, > [], > <Element 'machine' at 0x2369a10>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'rhel6.1.0'}, > [], > <Element 'machine' at 0x2369a50>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'pc-q35-rhel7.0.0'}, > [], > <Element 'machine' at 0x2369a90>, > { '_children': [], > 'attrib': { 'maxCpus': '240'}, > 'tag': 'machine', > 'tail': '\n ', > 'text': 'rhel6.2.0'}, > [], > <Element 'domain' at 0x2369ad0>, > { '_children': [], > 'attrib': { 'type': 'qemu'}, > 'tag': 'domain', > 'tail': '\n '}, > [], > <Element 'domain' at 0x2369b10>, > { '_children': [<Element 'emulator' at 0x2369b50>], > 'attrib': { 'type': 'kvm'}, > 'tag': 'domain', > 'tail': '\n ', > 'text': '\n '}, > [<Element 'emulator' at 0x2369b50>], > <Element 'emulator' at 0x2369b50>, > { '_children': [], > 'attrib': { }, > 'tag': 'emulator', > 'tail': '\n ', > 'text': '/usr/libexec/qemu-kvm'}, > [], > <Element 'features' at 0x2369bd0>, > { '_children': [ <Element 'cpuselection' at 0x2369c10>, > <Element 'deviceboot' at 0x2369c50>, > <Element 'disksnapshot' at 0x2369c90>, > <Element 'acpi' at 0x2369cd0>, > <Element 'apic' at 0x2369d10>], > 'attrib': { }, > 'tag': 'features', > 'tail': '\n ', > 'text': '\n '}, > [ <Element 'cpuselection' at 0x2369c10>, > <Element 'deviceboot' at 0x2369c50>, > <Element 'disksnapshot' at 0x2369c90>, > <Element 'acpi' at 0x2369cd0>, > <Element 'apic' at 0x2369d10>], > <Element 'cpuselection' at 0x2369c10>, > { '_children': [], 'attrib': { }, 'tag': 'cpuselection', > 'tail': '\n '}, > [], > <Element 'deviceboot' at 0x2369c50>, > { '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail': > '\n '}, > [], > <Element 'disksnapshot' at 0x2369c90>, > { '_children': [], > 'attrib': { 'default': 'on', 'toggle': 'no'}, > 'tag': 'disksnapshot', > 'tail': '\n '}, > [], > <Element 'acpi' at 0x2369cd0>, > { '_children': [], > 'attrib': { 'default': 'on', 'toggle': 'yes'}, > 'tag': 'acpi', > 'tail': '\n '}, > [], > <Element 'apic' at 0x2369d10>, > { '_children': [], > 'attrib': { 'default': 'on', 'toggle': 'no'}, > 'tag': 'apic', > 'tail': '\n '}, > [], > [], > (<type '_ctypes.Array'>,), > { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>, > '__doc__': None, > '__module__': 'vdsm.netlink', > '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>, > '_length_': 60, > '_type_': <class 'ctypes.c_char'>, > 'raw': <attribute 'raw' of 'c_char_Array_60' objects>, > 'value': <attribute 'value' of 'c_char_Array_60' objects>}, > <class 'vdsm.netlink.c_char_Array_60'>, > <attribute '__dict__' of 'c_char_Array_60' objects>, > <attribute '__weakref__' of 'c_char_Array_60' objects>, > ( <class 'vdsm.netlink.c_char_Array_60'>, > <type '_ctypes.Array'>, > <type '_ctypes._CData'>, > <type 'object'>), > <attribute 'raw' of 'c_char_Array_60' objects>, > <attribute 'value' of 'c_char_Array_60' objects>, > <xml.etree.ElementTree.XMLParser object at 0x2357f50>, > { '_doctype': None, > '_error': <class 'xml.parsers.expat.ExpatError'>, > '_names': { u'acpi': 'acpi', > u'apic': 'apic', > u'arch': 'arch', > u'baselabel': 'baselabel', > u'canonical': 'canonical', > u'capabilities': 'capabilities', > u'cell': 'cell', > u'cells': 'cells', > u'core_id': 'core_id', > u'cores': 'cores', > u'cpu': 'cpu', > u'cpus': 'cpus', > u'cpuselection': 'cpuselection', > u'default': 'default', > u'deviceboot': 'deviceboot', > u'disksnapshot': 'disksnapshot', > u'distances': 'distances', > u'doi': 'doi', > u'domain': 'domain', > u'emulator': 'emulator', > u'feature': 'feature', > u'features': 'features', > u'guest': 'guest', > u'host': 'host', > u'id': 'id', > u'live': 'live', > u'machine': 'machine', > u'maxCpus': 'maxCpus', > u'memory': 'memory', > u'migration_features': 'migration_features', > u'model': 'model', > u'name': 'name', > u'nonpae': 'nonpae', > u'num': 'num', > u'os_type': 'os_type', > u'pae': 'pae', > u'pages': 'pages', > u'power_management': 'power_management', > u'secmodel': 'secmodel', > u'sibling': 'sibling', > u'siblings': 'siblings', > u'size': 'size', > u'socket_id': 'socket_id', > u'sockets': 'sockets', > u'suspend_disk': 'suspend_disk', > u'suspend_hybrid': 'suspend_hybrid', > u'suspend_mem': 'suspend_mem', > u'threads': 'threads', > u'toggle': 'toggle', > u'topology': 'topology', > u'type': 'type', > u'unit': 'unit', > u'uri_transport': 'uri_transport', > u'uri_transports': 'uri_transports', > u'uuid': 'uuid', > u'value': 'value', > u'vendor': 'vendor', > u'wordsize': 'wordsize'}, > '_target': <xml.etree.ElementTree.TreeBuilder object at 0x2357f10>, > 'entity': { }, > 'parser': <pyexpat.xmlparser object at 0x22e5360>, > 'version': 'Expat 2.1.0'}, > <pyexpat.xmlparser object at 0x22e5360>, > <xml.etree.ElementTree.TreeBuilder object at 0x2357f10>, > <bound method XMLParser._start_list of > <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, > <bound method XMLParser._end of <xml.etree.ElementTree.XMLParser > object at 0x2357f50>>, > <bound method XMLParser._pi of <xml.etree.ElementTree.XMLParser > object at 0x2357f50>>, > <bound method XMLParser._data of <xml.etree.ElementTree.XMLParser > object at 0x2357f50>>, > <bound method XMLParser._comment of > <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, > <bound method XMLParser._default of > <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, > { '_data': [], > '_elem': [], > '_factory': <class 'xml.etree.ElementTree.Element'>, > '_last': <Element 'capabilities' at 0x2357f90>, > '_tail': 1}, > [], > <Element 'cell' at 0x23646d0>, > <Element 'cells' at 0x2364690>, > <Element 'topology' at 0x2364650>, > <Element 'host' at 0x2357fd0>, > <Element 'capabilities' at 0x2357f90>, > { '_children': [ <Element 'memory' at 0x2364710>, > <Element 'pages' at 0x2364750>, > <Element 'pages' at 0x2364790>, > <Element 'distances' at 0x23647d0>, > <Element 'cpus' at 0x2364850>], > 'attrib': { 'id': '0'}, > 'tag': 'cell', > 'tail': '\n ', > 'text': '\n '}, > { '_children': [<Element 'cell' at 0x23646d0>], > 'attrib': { 'num': '1'}, > 'tag': 'cells', > 'tail': '\n ', > 'text': '\n '}, > { '_children': [<Element 'cells' at 0x2364690>], > 'attrib': { }, > 'tag': 'topology', > 'tail': '\n ', > 'text': '\n '}, > { '_children': [ <Element 'uuid' at 0x2364050>, > <Element 'cpu' at 0x2364090>, > <Element 'power_management' at 0x2364390>, > <Element 'migration_features' at 0x23644d0>, > <Element 'topology' at 0x2364650>, > <Element 'secmodel' at 0x2364a90>, > <Element 'secmodel' at 0x2364bd0>], > 'attrib': { }, > 'tag': 'host', > 'tail': '\n\n ', > 'text': '\n '}, > { '_children': [ <Element 'host' at 0x2357fd0>, > <Element 'guest' at 0x2364d10>, > <Element 'guest' at 0x2369590>], > 'attrib': { }, > 'tag': 'capabilities', > 'text': '\n\n '}, > [ <Element 'memory' at 0x2364710>, > <Element 'pages' at 0x2364750>, > <Element 'pages' at 0x2364790>, > <Element 'distances' at 0x23647d0>, > <Element 'cpus' at 0x2364850>], > [<Element 'cell' at 0x23646d0>], > [<Element 'cells' at 0x2364690>], > [ <Element 'uuid' at 0x2364050>, > <Element 'cpu' at 0x2364090>, > <Element 'power_management' at 0x2364390>, > <Element 'migration_features' at 0x23644d0>, > <Element 'topology' at 0x2364650>, > <Element 'secmodel' at 0x2364a90>, > <Element 'secmodel' at 0x2364bd0>], > [ <Element 'host' at 0x2357fd0>, > <Element 'guest' at 0x2364d10>, > <Element 'guest' at 0x2369590>], > <Element 'migration_features' at 0x23644d0>, > <Element 'power_management' at 0x2364390>, > <Element 'cpu' at 0x2364090>, > <Element 'uuid' at 0x2364050>, > { '_children': [ <Element 'live' at 0x2364510>, > <Element 'uri_transports' at 0x2364590>], > 'attrib': { }, > 'tag': 'migration_features', > 'tail': '\n ', > 'text': '\n '}, > { '_children': [ <Element 'suspend_mem' at 0x23643d0>, > <Element 'suspend_disk' at 0x2364410>, > <Element 'suspend_hybrid' at 0x2364490>], > 'attrib': { }, > 'tag': 'power_management', > 'tail': '\n ', > 'text': '\n '}, > { '_children': [ <Element 'arch' at 0x23640d0>, > <Element 'model' at 0x2364110>, > <Element 'vendor' at 0x2364150>, > <Element 'topology' at 0x2364190>, > <Element 'feature' at 0x23641d0>, > <Element 'feature' at 0x2364210>, > <Element 'feature' at 0x2364250>, > <Element 'feature' at 0x2364290>, > <Element 'feature' at 0x23642d0>, > <Element 'pages' at 0x2364310>, > <Element 'pages' at 0x2364350>], > 'attrib': { }, > 'tag': 'cpu', > 'tail': '\n ', > 'text': '\n '}, > { '_children': [], > 'attrib': { }, > 'tag': 'uuid', > 'tail': '\n ', > 'text': 'a920d963-c18f-4fda-aa5a-deee9bbcde9d'}, > [<Element 'live' at 0x2364510>, <Element 'uri_transports' at 0x2364590>], > [ <Element 'suspend_mem' at 0x23643d0>, > <Element 'suspend_disk' at 0x2364410>, > <Element 'suspend_hybrid' at 0x2364490>], > [ <Element 'arch' at 0x23640d0>, > <Element 'model' at 0x2364110>, > <Element 'vendor' at 0x2364150>, > <Element 'topology' at 0x2364190>, > <Element 'feature' at 0x23641d0>, > <Element 'feature' at 0x2364210>, > <Element 'feature' at 0x2364250>, > <Element 'feature' at 0x2364290>, > <Element 'feature' at 0x23642d0>, > <Element 'pages' at 0x2364310>, > <Element 'pages' at 0x2364350>], > [], > <Element 'uri_transports' at 0x2364590>, > <Element 'live' at 0x2364510>, > <Element 'suspend_hybrid' at 0x2364490>, > <Element 'suspend_disk' at 0x2364410>, > <Element 'suspend_mem' at 0x23643d0>, > <Element 'pages' at 0x2364350>, > <Element 'pages' at 0x2364310>, > <Element 'feature' at 0x23642d0>, > <Element 'feature' at 0x2364290>, > <Element 'feature' at 0x2364250>, > <Element 'feature' at 0x2364210>, > <Element 'feature' at 0x23641d0>, > <Element 'topology' at 0x2364190>, > <Element 'vendor' at 0x2364150>, > <Element 'model' at 0x2364110>, > <Element 'arch' at 0x23640d0>, > { '_children': [ <Element 'uri_transport' at 0x23645d0>, > <Element 'uri_transport' at 0x2364610>], > 'attrib': { }, > 'tag': 'uri_transports', > 'tail': '\n ', > 'text': '\n '}, > { '_children': [], 'attrib': { }, 'tag': 'live', 'tail': '\n '}, > { '_children': [], 'attrib': { }, 'tag': 'suspend_hybrid', > 'tail': '\n '}, > { '_children': [], 'attrib': { }, 'tag': 'suspend_disk', > 'tail': '\n '}, > { '_children': [], 'attrib': { }, 'tag': 'suspend_mem', > 'tail': '\n '}, > { '_children': [], > 'attrib': { 'size': '2048', 'unit': 'KiB'}, > 'tag': 'pages', > 'tail': '\n '}, > { '_children': [], > 'attrib': { 'size': '4', 'unit': 'KiB'}, > 'tag': 'pages', > 'tail': '\n '}, > { '_children': [], > 'attrib': { 'name': 'ss'}, > 'tag': 'feature', > 'tail': '\n '}, > { '_children': [], > 'attrib': { 'name': 'vmx'}, > 'tag': 'feature', > 'tail': '\n '}, > { '_children': [], > 'attrib': { 'name': 'pcid'}, > 'tag': 'feature', > 'tail': '\n '}, > { '_children': [], > 'attrib': { 'name': 'osxsave'}, > 'tag': 'feature', > 'tail': '\n '}, > { '_children': [], > 'attrib': { 'name': 'hypervisor'}, > 'tag': 'feature', > 'tail': '\n '}, > { '_children': [], > 'attrib': { 'cores': '1', 'sockets': '8', 'threads': '1'}, > 'tag': 'topology', > 'tail': '\n '}, > { '_children': [], > 'attrib': { }, > 'tag': 'vendor', > 'tail': '\n ', > 'text': 'Intel'}, > { '_children': [], > 'attrib': { }, > 'tag': 'model', > 'tail': '\n ', > 'text': 'IvyBridge'}, > { '_children': [], > 'attrib': { }, > 'tag': 'arch', > 'tail': '\n ', > 'text': 'x86_64'}, > [ <Element 'uri_transport' at 0x23645d0>, > <Element 'uri_transport' at 0x2364610>], > [], > [], > [], > [], > [], > [], > [], > [], > [], > [], > [], > [], > [], > [], > [], > <Element 'uri_transport' at 0x2364610>, > <Element 'uri_transport' at 0x23645d0>, > { '_children': [], > 'attrib': { }, > 'tag': 'uri_transport', > 'tail': '\n ', > 'text': 'rdma'}, > { '_children': [], > 'attrib': { }, > 'tag': 'uri_transport', > 'tail': '\n ', > 'text': 'tcp'}, > [], > [], > (<type '_ctypes.Array'>,), > { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>, > '__doc__': None, > '__module__': 'vdsm.netlink', > '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>, > '_length_': 60, > '_type_': <class 'ctypes.c_char'>, > 'raw': <attribute 'raw' of 'c_char_Array_60' objects>, > 'value': <attribute 'value' of 'c_char_Array_60' objects>}, > <class 'vdsm.netlink.c_char_Array_60'>, > <attribute '__dict__' of 'c_char_Array_60' objects>, > <attribute '__weakref__' of 'c_char_Array_60' objects>, > ( <class 'vdsm.netlink.c_char_Array_60'>, > <type '_ctypes.Array'>, > <type '_ctypes._CData'>, > <type 'object'>), > <attribute 'raw' of 'c_char_Array_60' objects>, > <attribute 'value' of 'c_char_Array_60' objects>]