I tested with https://gerrit.ovirt.org/#/c/51630 libvirt.virDomain object does not leak any more, but pthreading object are still leaking, and the number of uncollectible objects is still very high (12,915). Each time we start and stop a vm, we leak about 6000 objects which are part of a reference cycle, and each time we create and leak 27 pthreading locks/conditions which are probably the reason for this leak (because they implement __del__ or reference object implementing __del__. When vdsm is idle, we leak 18 objects per minute, we don't know yet why. # tail -f /var/log/vdsm/vdsm.log | grep health Vdsm was connected to storage pool here, running as the spm... Thread-15::DEBUG::2016-01-15 10:00:00,159::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:00:00,178::health::86::health::(_check) Collected 468 objects Thread-15::WARNING::2016-01-15 10:00:00,182::health::100::health::(_check) Found 5953 uncollectible objects, 0 suspects implementing __del__: [] Thread-15::DEBUG::2016-01-15 10:01:00,182::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:01:00,207::health::86::health::(_check) Collected 202 objects Thread-15::WARNING::2016-01-15 10:01:00,211::health::100::health::(_check) Found 6169 uncollectible objects, 23 suspects implementing __del__: ['<pthreading.Lock object at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>', '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>', '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>', '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>', '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>', '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>', '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>', '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>'] vm was started here... Thread-15::DEBUG::2016-01-15 10:02:00,212::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:02:00,243::health::86::health::(_check) Collected 1689 objects Thread-15::WARNING::2016-01-15 10:02:00,252::health::100::health::(_check) Found 11107 uncollectible objects, 23 suspects implementing __del__: ['<pthreading.Lock object at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>', '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>', '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>', '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>', '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>', '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>', '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>', '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>'] Thread-15::DEBUG::2016-01-15 10:03:00,254::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:03:00,285::health::86::health::(_check) Collected 57 objects Thread-15::WARNING::2016-01-15 10:03:00,294::health::100::health::(_check) Found 11162 uncollectible objects, 23 suspects implementing __del__: ['<pthreading.Lock object at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>', '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>', '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>', '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>', '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>', '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>', '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>', '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>'] vm was stopped here... Thread-15::DEBUG::2016-01-15 10:04:00,294::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:04:00,324::health::86::health::(_check) Collected 1755 objects Thread-15::WARNING::2016-01-15 10:04:00,335::health::100::health::(_check) Found 12915 uncollectible objects, 50 suspects implementing __del__: ['<pthreading.Lock object at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>', '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>', '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>', '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>', '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>', '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>', '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>', '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>', '<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>', '<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>', '<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>', '<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at 0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>', '<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at 0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>', '<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at 0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>', '<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at 0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>', '<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at 0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>', '<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>'] Skipping some logs, no activity in engine... Thread-15::DEBUG::2016-01-15 10:23:01,082::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:23:01,113::health::86::health::(_check) Collected 18 objects Thread-15::WARNING::2016-01-15 10:23:01,124::health::100::health::(_check) Found 13267 uncollectible objects, 50 suspects implementing __del__: ['<pthreading.Lock object at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>', '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>', '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>', '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>', '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>', '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>', '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>', '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>', '<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>', '<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>', '<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>', '<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at 0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>', '<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at 0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>', '<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at 0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>', '<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at 0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>', '<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at 0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>', '<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>'] Thread-15::DEBUG::2016-01-15 10:24:01,125::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:24:01,156::health::86::health::(_check) Collected 18 objects Thread-15::WARNING::2016-01-15 10:24:01,167::health::100::health::(_check) Found 13283 uncollectible objects, 50 suspects implementing __del__: ['<pthreading.Lock object at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>', '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>', '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>', '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>', '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>', '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>', '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>', '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>', '<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>', '<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>', '<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>', '<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at 0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>', '<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at 0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>', '<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at 0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>', '<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at 0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>', '<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at 0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>', '<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>'] vm was started here... Thread-15::DEBUG::2016-01-15 10:25:01,167::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:25:01,204::health::86::health::(_check) Collected 1537 objects Thread-15::WARNING::2016-01-15 10:25:01,219::health::100::health::(_check) Found 18264 uncollectible objects, 50 suspects implementing __del__: ['<pthreading.Lock object at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>', '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>', '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>', '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>', '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>', '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>', '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>', '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>', '<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>', '<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>', '<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>', '<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at 0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>', '<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at 0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>', '<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at 0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>', '<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at 0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>', '<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at 0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>', '<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>'] vm was ask to stop here... Thread-15::DEBUG::2016-01-15 10:26:01,219::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:26:01,255::health::86::health::(_check) Collected 57 objects Thread-15::WARNING::2016-01-15 10:26:01,270::health::100::health::(_check) Found 18319 uncollectible objects, 50 suspects implementing __del__: ['<pthreading.Lock object at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>', '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>', '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>', '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>', '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>', '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>', '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>', '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>', '<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>', '<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>', '<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>', '<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at 0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>', '<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at 0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>', '<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at 0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>', '<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at 0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>', '<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at 0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>', '<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>'] vm was shutdown here... Thread-15::DEBUG::2016-01-15 10:27:01,270::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 10:27:01,306::health::86::health::(_check) Collected 1752 objects Thread-15::WARNING::2016-01-15 10:27:01,322::health::100::health::(_check) Found 20069 uncollectible objects, 77 suspects implementing __del__: ['<pthreading.Lock object at 0x7fa9b4031200>', '<pthreading.Lock object at 0x7fa9b4031170>', '<pthreading.Lock object at 0x7fa9b40535a8>', '<pthreading.Lock object at 0x7fa9b40313b0>', '<pthreading.Lock object at 0x7fa9b40311b8>', '<pthreading.Lock object at 0x7fa9b403d7a0>', '<pthreading.Lock object at 0x7fa9b40313f8>', '<pthreading.Lock object at 0x7fa9b4031488>', '<pthread.Cond object at 0x7fa9b4030790>', '<pthreading.Lock object at 0x7fa9b4031320>', '<pthread.Cond object at 0x7fa9b4030550>', '<pthreading.Lock object at 0x7fa9b4053560>', '<pthread.Cond object at 0x7fa9b4048f50>', '<pthreading.Lock object at 0x7fa9b4031290>', '<pthread.Cond object at 0x7fa9b4030390>', '<pthreading.Lock object at 0x7fa9b4031440>', '<pthread.Cond object at 0x7fa9b40306d0>', '<pthreading.Lock object at 0x7fa9b4031368>', '<pthread.Cond object at 0x7fa9b4030610>', '<pthreading.Lock object at 0x7fa9b40312d8>', '<pthread.Cond object at 0x7fa9b4030490>', '<pthreading.Lock object at 0x7fa9b4031248>', '<pthread.Cond object at 0x7fa9b4030290>', '<pthreading.Lock object at 0x7fa9a407e050>', '<pthreading.Lock object at 0x7fa9a408c830>', '<pthreading.Lock object at 0x7fa904052f80>', '<pthreading.Lock object at 0x7fa9a407ecf8>', '<pthreading.Lock object at 0x7fa904043d88>', '<pthreading.Lock object at 0x7fa904052098>', '<pthreading.Lock object at 0x7fa9a407ecb0>', '<pthreading.Lock object at 0x7fa904052710>', '<pthread.Cond object at 0x7fa9a4148c10>', '<pthreading.Lock object at 0x7fa9a407e5a8>', '<pthread.Cond object at 0x7fa9a4148090>', '<pthreading.Lock object at 0x7fa904052f38>', '<pthread.Cond object at 0x7fa904045f50>', '<pthreading.Lock object at 0x7fa9a407e098>', '<pthread.Cond object at 0x7fa9b40da750>', '<pthreading.Lock object at 0x7fa9a407e248>', '<pthread.Cond object at 0x7fa9a4148990>', '<pthreading.Lock object at 0x7fa9a407e3b0>', '<pthread.Cond object at 0x7fa9a4148dd0>', '<pthreading.Lock object at 0x7fa9a407e440>', '<pthread.Cond object at 0x7fa9b40da410>', '<pthreading.Lock object at 0x7fa9a407e1b8>', '<pthread.Cond object at 0x7fa9b40da9d0>', '<pthread.Cond object at 0x7fa9a413b210>', '<pthreading.Lock object at 0x7fa9b40d6908>', '<pthreading.Lock object at 0x7fa8e80f30e0>', '<pthread.Cond object at 0x7fa9a413b510>', '<pthreading.Lock object at 0x7fa904078710>', '<pthreading.Lock object at 0x7fa9b40af440>', '<pthreading.Lock object at 0x7fa90405b200>', '<pthreading.Lock object at 0x7fa904078998>', '<pthreading.Lock object at 0x7fa9a407cfc8>', '<pthreading.Lock object at 0x7fa904078638>', '<pthreading.Lock object at 0x7fa904078560>', '<pthreading.Lock object at 0x7fa904078200>', '<pthread.Cond object at 0x7fa9a4142d10>', '<pthreading.Lock object at 0x7fa9040785f0>', '<pthread.Cond object at 0x7fa9b40d1a50>', '<pthreading.Lock object at 0x7fa90405b9e0>', '<pthread.Cond object at 0x7fa9a4142610>', '<pthreading.Lock object at 0x7fa9040787a0>', '<pthread.Cond object at 0x7fa9b40d1f10>', '<pthreading.Lock object at 0x7fa9040786c8>', '<pthread.Cond object at 0x7fa9a4142b10>', '<pthreading.Lock object at 0x7fa904078368>', '<pthread.Cond object at 0x7fa9b40d1790>', '<pthreading.Lock object at 0x7fa904078290>', '<pthread.Cond object at 0x7fa9b40d14d0>', '<pthreading.Lock object at 0x7fa904078248>', '<pthread.Cond object at 0x7fa9b40d1150>', '<pthread.Cond object at 0x7fa9a41360d0>', '<pthreading.Lock object at 0x7fa9441a8d40>', '<pthreading.Lock object at 0x7fa944049b00>', '<pthread.Cond object at 0x7fa9a4136410>'] On Thu, Jan 14, 2016 at 8:00 PM, Nir Soffer <nsoffer@redhat.com> wrote:

Here are example logs with running and stopping one vm:

# tail -f /var/log/vdsm/vdsm.log | grep health Thread-13::DEBUG::2016-01-14 19:42:04,625::health::84::health::(_check) Checking health Thread-13::DEBUG::2016-01-14 19:42:04,651::health::86::health::(_check) Collected 25 objects Thread-13::WARNING::2016-01-14 19:42:04,656::health::100::health::(_check) Found 6085 uncollectible objects, 0 suspects implementing __del__: []

Vm was started here...

Thread-13::DEBUG::2016-01-14 19:43:04,657::health::84::health::(_check) Checking health Thread-13::DEBUG::2016-01-14 19:43:04,688::health::86::health::(_check) Collected 1680 objects Thread-13::WARNING::2016-01-14 19:43:04,697::health::100::health::(_check) Found 11041 uncollectible objects, 0 suspects implementing __del__: [] Thread-13::DEBUG::2016-01-14 19:44:04,698::health::84::health::(_check) Checking health Thread-13::DEBUG::2016-01-14 19:44:04,729::health::86::health::(_check) Collected 57 objects Thread-13::WARNING::2016-01-14 19:44:04,737::health::100::health::(_check) Found 11096 uncollectible objects, 0 suspects implementing __del__: [] Thread-13::DEBUG::2016-01-14 19:45:04,738::health::84::health::(_check) Checking health Thread-13::DEBUG::2016-01-14 19:45:04,769::health::86::health::(_check) Collected 69 objects Thread-13::WARNING::2016-01-14 19:45:04,778::health::100::health::(_check) Found 11163 uncollectible objects, 0 suspects implementing __del__: []

Vm was shutdown here...

Thread-13::DEBUG::2016-01-14 19:46:04,779::health::84::health::(_check) Checking health Thread-13::DEBUG::2016-01-14 19:46:04,809::health::86::health::(_check) Collected 2272 objects Thread-13::WARNING::2016-01-14 19:46:04,819::health::100::health::(_check) Found 13464 uncollectible objects, 28 suspects implementing __del__: ['<pthreading.Lock object at 0x7f1d8016f908>', '<pthreading.Lock object at 0x7f1d801804d0>', '<pthreading.Lock object at 0x7f1cf8010f38>', '<pthreading.Lock object at 0x7f1d8016f7a0>', '<pthreading.Lock object at 0x7f1d20003680>', '<pthreading.Lock object at 0x7f1da8112998>', '<pthreading.Lock object at 0x7f1d8016fa28>', '<pthreading.Lock object at 0x7f1da8112830>', '<pthread.Cond object at 0x7f1da811cf50>', '<pthreading.Lock object at 0x7f1d8016f200>', '<pthread.Cond object at 0x7f1da811ca50>', '<pthreading.Lock object at 0x7f1cf8010fc8>', '<pthread.Cond object at 0x7f1da8115c50>', '<pthreading.Lock object at 0x7f1da8112878>', '<pthread.Cond object at 0x7f1da811c550>', '<pthreading.Lock object at 0x7f1d8016f9e0>', '<pthread.Cond object at 0x7f1da811ca90>', '<pthreading.Lock object at 0x7f1d8016f680>', '<pthread.Cond object at 0x7f1da811cc50>', '<libvirt.virDomain object at 0x1ba8210>', '<pthreading.Lock object at 0x7f1d8016f758>', '<pthread.Cond object at 0x7f1da811cb50>', '<pthreading.Lock object at 0x7f1d8016fb90>', '<pthread.Cond object at 0x7f1da811cc10>', '<pthread.Cond object at 0x233ed50>', '<pthreading.Lock object at 0x7f1ce0227c20>', '<pthreading.Lock object at 0x7f1ce0227ab8>', '<pthread.Cond object at 0x233ed10>']

Need to test with should be fixed by https://gerrit.ovirt.org/51630, which should eliminate '<libvirt.virDomain object at 0x1ba8210>', and maybe also the pthreading objects.

Nir

On Thu, Jan 14, 2016 at 7:38 PM, Nir Soffer <nsoffer@redhat.com> wrote:

...

Hi all,

Continuing the leak investigation Francesco and Milan are working on, I posted this patch, adding leak health monitoring to vdsm [1]. This patch currently monitor collectible objects, and we may add other interesting stuff later if needed.

To enable the monitor, you must set:

[devel] health_monitor_enable = true

In vdsm.conf, and restart vdsm.

Here is an example logs - this was logged 60 seconds after starting vdsm, when running as spm:

Thread-13::DEBUG::2016-01-14 18:53:43,239::health::84::health::(_check) Checking health Thread-13::DEBUG::2016-01-14 18:53:43,272::health::86::health::(_check) Collected 460 objects Thread-13::WARNING::2016-01-14 18:53:43,277::health::100::health::(_check) Found 5873 uncollectible objects, 10 suspects implementing __del__: ['<pthreading.Lock object at 0x7f012c115320>', '<pthreading.Lock object at 0x7f012c115440>', '<pthreading.Lock object at 0x7f012c1153b0>', '<pthreading.Lock object at 0x7f012c1154d0>', '<pthread.Cond object at 0x7f012c111350>', '<pthread.Cond object at 0x7f012c1112d0>', '<pthread.Cond object at 0x7f012c111250>', '<pthread.Cond object at 0x7f012c111850>', '<pthread.Cond object at 0x7f012c1117d0>', '<pthread.Cond object at 0x7f012c111750>']

We found 460 uncollectible objects, 10 of them are objects implementing __del__. Having such object in a reference cycle make the entire cycle uncollectible.

The pthreading objects in this log grow as vdsm continue to run, even without any interaction with engine. The pthreading objects leak is caused by a cycle in storage.misc.RWLock, fixed in patch [2].

After solving the pthreading leak, we still have many uncollectible objects in the logs, listing them, it seems that the suspect is hiding inside a collection (dict, list, tuple) and does not appear in the gc.garbage list. I will address this in later patch, searching in the entire object graph instead of the top level of gc.garbage.

See end of this mail for list of these objects. I printed them like this:

Enable manhole debugging shell:

[devel] manhole_enable = true

Connect to vdsm using manhole:

nc -U /run/vdsm/vdsmd.manhole

Print garbage:

...

...

...

import pprint import gc with open('/tmp/garbage.txt', 'w') as f: pprint.pprint(gc.garbage, stream=f, indent=4)

These objects looks related to libvirt - Francesco, can you take a look?

[1] https://gerrit.ovirt.org/51708 [2] https://gerrit.ovirt.org/51868

Nir

--- garbage after patch [2] ---

[ (<type '_ctypes.Array'>,), { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>, '__doc__': None, '__module__': 'vdsm.netlink', '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>, '_length_': 60, '_type_': <class 'ctypes.c_char'>, 'raw': <attribute 'raw' of 'c_char_Array_60' objects>, 'value': <attribute 'value' of 'c_char_Array_60' objects>}, <class 'vdsm.netlink.c_char_Array_60'>, <attribute '__dict__' of 'c_char_Array_60' objects>, <attribute '__weakref__' of 'c_char_Array_60' objects>, ( <class 'vdsm.netlink.c_char_Array_60'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_60' objects>, <attribute 'value' of 'c_char_Array_60' objects>, <cell at 0x23662f0: bool object at 0x7f1e086c8c40>, <cell at 0x2366328: NoneType object at 0x7f1e086dcf00>, <cell at 0x2366360: function object at 0x233f488>, ( <cell at 0x23662f0: bool object at 0x7f1e086c8c40>, <cell at 0x2366328: NoneType object at 0x7f1e086dcf00>, <cell at 0x2366360: function object at 0x233f488>), <function wrapMethod at 0x233f488>, [ [ <Recursion on list with id=34033464>, <Recursion on list with id=34033464>, 'class'], [ <Recursion on list with id=34033464>, <Recursion on list with id=34033464>, 'class'], None], [ [ <Recursion on list with id=33979064>, <Recursion on list with id=33979064>, None], [ <Recursion on list with id=33979064>, <Recursion on list with id=33979064>, None], 'class'], [ [ <Recursion on list with id=35535256>, <Recursion on list with id=35535256>, 'class'], [ <Recursion on list with id=35535256>, <Recursion on list with id=35535256>, 'class'], None], [ [ <Recursion on list with id=35535832>, <Recursion on list with id=35535832>, None], [ <Recursion on list with id=35535832>, <Recursion on list with id=35535832>, None], 'class'], [ [ <Recursion on list with id=35556312>, <Recursion on list with id=35556312>, 'class'], [ <Recursion on list with id=35556312>, <Recursion on list with id=35556312>, 'class'], None], [ [ <Recursion on list with id=35556600>, <Recursion on list with id=35556600>, None], [ <Recursion on list with id=35556600>, <Recursion on list with id=35556600>, None], 'class'], [ [ <Recursion on list with id=35556960>, <Recursion on list with id=35556960>, 'class'], [ <Recursion on list with id=35556960>, <Recursion on list with id=35556960>, 'class'], None], [ [ <Recursion on list with id=35561912>, <Recursion on list with id=35561912>, None], [ <Recursion on list with id=35561912>, <Recursion on list with id=35561912>, None], 'class'], [ [ <Recursion on list with id=35583904>, <Recursion on list with id=35583904>, 'class'], [ <Recursion on list with id=35583904>, <Recursion on list with id=35583904>, 'class'], None], [ [ <Recursion on list with id=35584912>, <Recursion on list with id=35584912>, None], [ <Recursion on list with id=35584912>, <Recursion on list with id=35584912>, None], 'class'], [ [ <Recursion on list with id=36340368>, <Recursion on list with id=36340368>, 'class'], [ <Recursion on list with id=36340368>, <Recursion on list with id=36340368>, 'class'], None], [ [ <Recursion on list with id=36340872>, <Recursion on list with id=36340872>, None], [ <Recursion on list with id=36340872>, <Recursion on list with id=36340872>, None], 'class'], [ [ <Recursion on list with id=36364872>, <Recursion on list with id=36364872>, 'class'], [ <Recursion on list with id=36364872>, <Recursion on list with id=36364872>, 'class'], None], [ [ <Recursion on list with id=36365520>, <Recursion on list with id=36365520>, None], [ <Recursion on list with id=36365520>, <Recursion on list with id=36365520>, None], 'class'], [ [ <Recursion on list with id=36457216>, <Recursion on list with id=36457216>, 'class'], [ <Recursion on list with id=36457216>, <Recursion on list with id=36457216>, 'class'], None], [ [ <Recursion on list with id=36458008>, <Recursion on list with id=36458008>, None], [ <Recursion on list with id=36458008>, <Recursion on list with id=36458008>, None], 'class'], [ [ <Recursion on list with id=36460520>, <Recursion on list with id=36460520>, 'class'], [ <Recursion on list with id=36460520>, <Recursion on list with id=36460520>, 'class'], None], [ [ <Recursion on list with id=36461168>, <Recursion on list with id=36461168>, None], [ <Recursion on list with id=36461168>, <Recursion on list with id=36461168>, None], 'class'], [ [ <Recursion on list with id=36562344>, <Recursion on list with id=36562344>, 'class'], [ <Recursion on list with id=36562344>, <Recursion on list with id=36562344>, 'class'], None], [ [ <Recursion on list with id=36563280>, <Recursion on list with id=36563280>, None], [ <Recursion on list with id=36563280>, <Recursion on list with id=36563280>, None], 'class'], <Element 'memory' at 0x2364710>, { '_children': [], 'attrib': { 'unit': 'KiB'}, 'tag': 'memory', 'tail': '\n ', 'text': '4095476'}, [], <Element 'pages' at 0x2364750>, { '_children': [], 'attrib': { 'size': '4', 'unit': 'KiB'}, 'tag': 'pages', 'tail': '\n ', 'text': '1023869'}, [], <Element 'pages' at 0x2364790>, { '_children': [], 'attrib': { 'size': '2048', 'unit': 'KiB'}, 'tag': 'pages', 'tail': '\n ', 'text': '0'}, [], <Element 'distances' at 0x23647d0>, { '_children': [<Element 'sibling' at 0x2364810>], 'attrib': { }, 'tag': 'distances', 'tail': '\n ', 'text': '\n '}, [<Element 'sibling' at 0x2364810>], <Element 'sibling' at 0x2364810>, { '_children': [], 'attrib': { 'id': '0', 'value': '10'}, 'tag': 'sibling', 'tail': '\n '}, [], <Element 'cpus' at 0x2364850>, { '_children': [ <Element 'cpu' at 0x2364890>, <Element 'cpu' at 0x23648d0>, <Element 'cpu' at 0x2364910>, <Element 'cpu' at 0x2364950>, <Element 'cpu' at 0x2364990>, <Element 'cpu' at 0x23649d0>, <Element 'cpu' at 0x2364a10>, <Element 'cpu' at 0x2364a50>], 'attrib': { 'num': '8'}, 'tag': 'cpus', 'tail': '\n ', 'text': '\n '}, [ <Element 'cpu' at 0x2364890>, <Element 'cpu' at 0x23648d0>, <Element 'cpu' at 0x2364910>, <Element 'cpu' at 0x2364950>, <Element 'cpu' at 0x2364990>, <Element 'cpu' at 0x23649d0>, <Element 'cpu' at 0x2364a10>, <Element 'cpu' at 0x2364a50>], <Element 'cpu' at 0x2364890>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '0', 'siblings': '0', 'socket_id': '0'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x23648d0>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '1', 'siblings': '1', 'socket_id': '1'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364910>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '2', 'siblings': '2', 'socket_id': '2'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364950>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '3', 'siblings': '3', 'socket_id': '3'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364990>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '4', 'siblings': '4', 'socket_id': '4'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x23649d0>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '5', 'siblings': '5', 'socket_id': '5'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364a10>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '6', 'siblings': '6', 'socket_id': '6'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'cpu' at 0x2364a50>, { '_children': [], 'attrib': { 'core_id': '0', 'id': '7', 'siblings': '7', 'socket_id': '7'}, 'tag': 'cpu', 'tail': '\n '}, [], <Element 'secmodel' at 0x2364a90>, { '_children': [ <Element 'model' at 0x2364ad0>, <Element 'doi' at 0x2364b10>, <Element 'baselabel' at 0x2364b50>, <Element 'baselabel' at 0x2364b90>], 'attrib': { }, 'tag': 'secmodel', 'tail': '\n ', 'text': '\n '}, [ <Element 'model' at 0x2364ad0>, <Element 'doi' at 0x2364b10>, <Element 'baselabel' at 0x2364b50>, <Element 'baselabel' at 0x2364b90>], <Element 'model' at 0x2364ad0>, { '_children': [], 'attrib': { }, 'tag': 'model', 'tail': '\n ', 'text': 'selinux'}, [], <Element 'doi' at 0x2364b10>, { '_children': [], 'attrib': { }, 'tag': 'doi', 'tail': '\n ', 'text': '0'}, [], <Element 'baselabel' at 0x2364b50>, { '_children': [], 'attrib': { 'type': 'kvm'}, 'tag': 'baselabel', 'tail': '\n ', 'text': 'system_u:system_r:svirt_t:s0'}, [], <Element 'baselabel' at 0x2364b90>, { '_children': [], 'attrib': { 'type': 'qemu'}, 'tag': 'baselabel', 'tail': '\n ', 'text': 'system_u:system_r:svirt_tcg_t:s0'}, [], <Element 'secmodel' at 0x2364bd0>, { '_children': [ <Element 'model' at 0x2364c10>, <Element 'doi' at 0x2364c50>, <Element 'baselabel' at 0x2364c90>, <Element 'baselabel' at 0x2364cd0>], 'attrib': { }, 'tag': 'secmodel', 'tail': '\n ', 'text': '\n '}, [ <Element 'model' at 0x2364c10>, <Element 'doi' at 0x2364c50>, <Element 'baselabel' at 0x2364c90>, <Element 'baselabel' at 0x2364cd0>], <Element 'model' at 0x2364c10>, { '_children': [], 'attrib': { }, 'tag': 'model', 'tail': '\n ', 'text': 'dac'}, [], <Element 'doi' at 0x2364c50>, { '_children': [], 'attrib': { }, 'tag': 'doi', 'tail': '\n ', 'text': '0'}, [], <Element 'baselabel' at 0x2364c90>, { '_children': [], 'attrib': { 'type': 'kvm'}, 'tag': 'baselabel', 'tail': '\n ', 'text': '+107:+107'}, [], <Element 'baselabel' at 0x2364cd0>, { '_children': [], 'attrib': { 'type': 'qemu'}, 'tag': 'baselabel', 'tail': '\n ', 'text': '+107:+107'}, [], <Element 'guest' at 0x2364d10>, { '_children': [ <Element 'os_type' at 0x2364d50>, <Element 'arch' at 0x2364d90>, <Element 'features' at 0x2369390>], 'attrib': { }, 'tag': 'guest', 'tail': '\n\n ', 'text': '\n '}, [ <Element 'os_type' at 0x2364d50>, <Element 'arch' at 0x2364d90>, <Element 'features' at 0x2369390>], <Element 'os_type' at 0x2364d50>, { '_children': [], 'attrib': { }, 'tag': 'os_type', 'tail': '\n ', 'text': 'hvm'}, [], <Element 'arch' at 0x2364d90>, { '_children': [ <Element 'wordsize' at 0x2364dd0>, <Element 'emulator' at 0x2364e10>, <Element 'machine' at 0x2364e90>, <Element 'machine' at 0x2364ed0>, <Element 'machine' at 0x2364f10>, <Element 'machine' at 0x2364f50>, <Element 'machine' at 0x2364f90>, <Element 'machine' at 0x2364fd0>, <Element 'machine' at 0x2369050>, <Element 'machine' at 0x2369090>, <Element 'machine' at 0x23690d0>, <Element 'machine' at 0x2369110>, <Element 'machine' at 0x2369150>, <Element 'machine' at 0x2369190>, <Element 'machine' at 0x23691d0>, <Element 'machine' at 0x2369210>, <Element 'machine' at 0x2369250>, <Element 'domain' at 0x2369290>, <Element 'domain' at 0x23692d0>], 'attrib': { 'name': 'i686'}, 'tag': 'arch', 'tail': '\n ', 'text': '\n '}, [ <Element 'wordsize' at 0x2364dd0>, <Element 'emulator' at 0x2364e10>, <Element 'machine' at 0x2364e90>, <Element 'machine' at 0x2364ed0>, <Element 'machine' at 0x2364f10>, <Element 'machine' at 0x2364f50>, <Element 'machine' at 0x2364f90>, <Element 'machine' at 0x2364fd0>, <Element 'machine' at 0x2369050>, <Element 'machine' at 0x2369090>, <Element 'machine' at 0x23690d0>, <Element 'machine' at 0x2369110>, <Element 'machine' at 0x2369150>, <Element 'machine' at 0x2369190>, <Element 'machine' at 0x23691d0>, <Element 'machine' at 0x2369210>, <Element 'machine' at 0x2369250>, <Element 'domain' at 0x2369290>, <Element 'domain' at 0x23692d0>], <Element 'wordsize' at 0x2364dd0>, { '_children': [], 'attrib': { }, 'tag': 'wordsize', 'tail': '\n ', 'text': '32'}, [], <Element 'emulator' at 0x2364e10>, { '_children': [], 'attrib': { }, 'tag': 'emulator', 'tail': '\n ', 'text': '/usr/libexec/qemu-kvm'}, [], <Element 'machine' at 0x2364e90>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.2.0'}, [], <Element 'machine' at 0x2364ed0>, { '_children': [], 'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc'}, [], <Element 'machine' at 0x2364f10>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.0.0'}, [], <Element 'machine' at 0x2364f50>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.1.0'}, [], <Element 'machine' at 0x2364f90>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.3.0'}, [], <Element 'machine' at 0x2364fd0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.2.0'}, [], <Element 'machine' at 0x2369050>, { '_children': [], 'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'q35'}, [], <Element 'machine' at 0x2369090>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.4.0'}, [], <Element 'machine' at 0x23690d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.0.0'}, [], <Element 'machine' at 0x2369110>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.1.0'}, [], <Element 'machine' at 0x2369150>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.5.0'}, [], <Element 'machine' at 0x2369190>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.6.0'}, [], <Element 'machine' at 0x23691d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.1.0'}, [], <Element 'machine' at 0x2369210>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.0.0'}, [], <Element 'machine' at 0x2369250>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.2.0'}, [], <Element 'domain' at 0x2369290>, { '_children': [], 'attrib': { 'type': 'qemu'}, 'tag': 'domain', 'tail': '\n '}, [], <Element 'domain' at 0x23692d0>, { '_children': [<Element 'emulator' at 0x2369310>], 'attrib': { 'type': 'kvm'}, 'tag': 'domain', 'tail': '\n ', 'text': '\n '}, [<Element 'emulator' at 0x2369310>], <Element 'emulator' at 0x2369310>, { '_children': [], 'attrib': { }, 'tag': 'emulator', 'tail': '\n ', 'text': '/usr/libexec/qemu-kvm'}, [], <Element 'features' at 0x2369390>, { '_children': [ <Element 'cpuselection' at 0x23693d0>, <Element 'deviceboot' at 0x2369410>, <Element 'disksnapshot' at 0x2369450>, <Element 'acpi' at 0x2369490>, <Element 'apic' at 0x23694d0>, <Element 'pae' at 0x2369510>, <Element 'nonpae' at 0x2369550>], 'attrib': { }, 'tag': 'features', 'tail': '\n ', 'text': '\n '}, [ <Element 'cpuselection' at 0x23693d0>, <Element 'deviceboot' at 0x2369410>, <Element 'disksnapshot' at 0x2369450>, <Element 'acpi' at 0x2369490>, <Element 'apic' at 0x23694d0>, <Element 'pae' at 0x2369510>, <Element 'nonpae' at 0x2369550>], <Element 'cpuselection' at 0x23693d0>, { '_children': [], 'attrib': { }, 'tag': 'cpuselection', 'tail': '\n '}, [], <Element 'deviceboot' at 0x2369410>, { '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail': '\n '}, [], <Element 'disksnapshot' at 0x2369450>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'no'}, 'tag': 'disksnapshot', 'tail': '\n '}, [], <Element 'acpi' at 0x2369490>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'yes'}, 'tag': 'acpi', 'tail': '\n '}, [], <Element 'apic' at 0x23694d0>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'no'}, 'tag': 'apic', 'tail': '\n '}, [], <Element 'pae' at 0x2369510>, { '_children': [], 'attrib': { }, 'tag': 'pae', 'tail': '\n '}, [], <Element 'nonpae' at 0x2369550>, { '_children': [], 'attrib': { }, 'tag': 'nonpae', 'tail': '\n '}, [], <Element 'guest' at 0x2369590>, { '_children': [ <Element 'os_type' at 0x23695d0>, <Element 'arch' at 0x2369610>, <Element 'features' at 0x2369bd0>], 'attrib': { }, 'tag': 'guest', 'tail': '\n\n', 'text': '\n '}, [ <Element 'os_type' at 0x23695d0>, <Element 'arch' at 0x2369610>, <Element 'features' at 0x2369bd0>], <Element 'os_type' at 0x23695d0>, { '_children': [], 'attrib': { }, 'tag': 'os_type', 'tail': '\n ', 'text': 'hvm'}, [], <Element 'arch' at 0x2369610>, { '_children': [ <Element 'wordsize' at 0x2369650>, <Element 'emulator' at 0x2369690>, <Element 'machine' at 0x2369710>, <Element 'machine' at 0x2369750>, <Element 'machine' at 0x2369790>, <Element 'machine' at 0x23697d0>, <Element 'machine' at 0x2369810>, <Element 'machine' at 0x2369850>, <Element 'machine' at 0x2369890>, <Element 'machine' at 0x23698d0>, <Element 'machine' at 0x2369910>, <Element 'machine' at 0x2369950>, <Element 'machine' at 0x2369990>, <Element 'machine' at 0x23699d0>, <Element 'machine' at 0x2369a10>, <Element 'machine' at 0x2369a50>, <Element 'machine' at 0x2369a90>, <Element 'domain' at 0x2369ad0>, <Element 'domain' at 0x2369b10>], 'attrib': { 'name': 'x86_64'}, 'tag': 'arch', 'tail': '\n ', 'text': '\n '}, [ <Element 'wordsize' at 0x2369650>, <Element 'emulator' at 0x2369690>, <Element 'machine' at 0x2369710>, <Element 'machine' at 0x2369750>, <Element 'machine' at 0x2369790>, <Element 'machine' at 0x23697d0>, <Element 'machine' at 0x2369810>, <Element 'machine' at 0x2369850>, <Element 'machine' at 0x2369890>, <Element 'machine' at 0x23698d0>, <Element 'machine' at 0x2369910>, <Element 'machine' at 0x2369950>, <Element 'machine' at 0x2369990>, <Element 'machine' at 0x23699d0>, <Element 'machine' at 0x2369a10>, <Element 'machine' at 0x2369a50>, <Element 'machine' at 0x2369a90>, <Element 'domain' at 0x2369ad0>, <Element 'domain' at 0x2369b10>], <Element 'wordsize' at 0x2369650>, { '_children': [], 'attrib': { }, 'tag': 'wordsize', 'tail': '\n ', 'text': '64'}, [], <Element 'emulator' at 0x2369690>, { '_children': [], 'attrib': { }, 'tag': 'emulator', 'tail': '\n ', 'text': '/usr/libexec/qemu-kvm'}, [], <Element 'machine' at 0x2369710>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.2.0'}, [], <Element 'machine' at 0x2369750>, { '_children': [], 'attrib': { 'canonical': 'pc-i440fx-rhel7.2.0', 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc'}, [], <Element 'machine' at 0x2369790>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.0.0'}, [], <Element 'machine' at 0x23697d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.1.0'}, [], <Element 'machine' at 0x2369810>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.3.0'}, [], <Element 'machine' at 0x2369850>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.2.0'}, [], <Element 'machine' at 0x2369890>, { '_children': [], 'attrib': { 'canonical': 'pc-q35-rhel7.2.0', 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'q35'}, [], <Element 'machine' at 0x23698d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.4.0'}, [], <Element 'machine' at 0x2369910>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.0.0'}, [], <Element 'machine' at 0x2369950>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-i440fx-rhel7.1.0'}, [], <Element 'machine' at 0x2369990>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.5.0'}, [], <Element 'machine' at 0x23699d0>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.6.0'}, [], <Element 'machine' at 0x2369a10>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.1.0'}, [], <Element 'machine' at 0x2369a50>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'pc-q35-rhel7.0.0'}, [], <Element 'machine' at 0x2369a90>, { '_children': [], 'attrib': { 'maxCpus': '240'}, 'tag': 'machine', 'tail': '\n ', 'text': 'rhel6.2.0'}, [], <Element 'domain' at 0x2369ad0>, { '_children': [], 'attrib': { 'type': 'qemu'}, 'tag': 'domain', 'tail': '\n '}, [], <Element 'domain' at 0x2369b10>, { '_children': [<Element 'emulator' at 0x2369b50>], 'attrib': { 'type': 'kvm'}, 'tag': 'domain', 'tail': '\n ', 'text': '\n '}, [<Element 'emulator' at 0x2369b50>], <Element 'emulator' at 0x2369b50>, { '_children': [], 'attrib': { }, 'tag': 'emulator', 'tail': '\n ', 'text': '/usr/libexec/qemu-kvm'}, [], <Element 'features' at 0x2369bd0>, { '_children': [ <Element 'cpuselection' at 0x2369c10>, <Element 'deviceboot' at 0x2369c50>, <Element 'disksnapshot' at 0x2369c90>, <Element 'acpi' at 0x2369cd0>, <Element 'apic' at 0x2369d10>], 'attrib': { }, 'tag': 'features', 'tail': '\n ', 'text': '\n '}, [ <Element 'cpuselection' at 0x2369c10>, <Element 'deviceboot' at 0x2369c50>, <Element 'disksnapshot' at 0x2369c90>, <Element 'acpi' at 0x2369cd0>, <Element 'apic' at 0x2369d10>], <Element 'cpuselection' at 0x2369c10>, { '_children': [], 'attrib': { }, 'tag': 'cpuselection', 'tail': '\n '}, [], <Element 'deviceboot' at 0x2369c50>, { '_children': [], 'attrib': { }, 'tag': 'deviceboot', 'tail': '\n '}, [], <Element 'disksnapshot' at 0x2369c90>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'no'}, 'tag': 'disksnapshot', 'tail': '\n '}, [], <Element 'acpi' at 0x2369cd0>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'yes'}, 'tag': 'acpi', 'tail': '\n '}, [], <Element 'apic' at 0x2369d10>, { '_children': [], 'attrib': { 'default': 'on', 'toggle': 'no'}, 'tag': 'apic', 'tail': '\n '}, [], [], (<type '_ctypes.Array'>,), { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>, '__doc__': None, '__module__': 'vdsm.netlink', '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>, '_length_': 60, '_type_': <class 'ctypes.c_char'>, 'raw': <attribute 'raw' of 'c_char_Array_60' objects>, 'value': <attribute 'value' of 'c_char_Array_60' objects>}, <class 'vdsm.netlink.c_char_Array_60'>, <attribute '__dict__' of 'c_char_Array_60' objects>, <attribute '__weakref__' of 'c_char_Array_60' objects>, ( <class 'vdsm.netlink.c_char_Array_60'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_60' objects>, <attribute 'value' of 'c_char_Array_60' objects>, <xml.etree.ElementTree.XMLParser object at 0x2357f50>, { '_doctype': None, '_error': <class 'xml.parsers.expat.ExpatError'>, '_names': { u'acpi': 'acpi', u'apic': 'apic', u'arch': 'arch', u'baselabel': 'baselabel', u'canonical': 'canonical', u'capabilities': 'capabilities', u'cell': 'cell', u'cells': 'cells', u'core_id': 'core_id', u'cores': 'cores', u'cpu': 'cpu', u'cpus': 'cpus', u'cpuselection': 'cpuselection', u'default': 'default', u'deviceboot': 'deviceboot', u'disksnapshot': 'disksnapshot', u'distances': 'distances', u'doi': 'doi', u'domain': 'domain', u'emulator': 'emulator', u'feature': 'feature', u'features': 'features', u'guest': 'guest', u'host': 'host', u'id': 'id', u'live': 'live', u'machine': 'machine', u'maxCpus': 'maxCpus', u'memory': 'memory', u'migration_features': 'migration_features', u'model': 'model', u'name': 'name', u'nonpae': 'nonpae', u'num': 'num', u'os_type': 'os_type', u'pae': 'pae', u'pages': 'pages', u'power_management': 'power_management', u'secmodel': 'secmodel', u'sibling': 'sibling', u'siblings': 'siblings', u'size': 'size', u'socket_id': 'socket_id', u'sockets': 'sockets', u'suspend_disk': 'suspend_disk', u'suspend_hybrid': 'suspend_hybrid', u'suspend_mem': 'suspend_mem', u'threads': 'threads', u'toggle': 'toggle', u'topology': 'topology', u'type': 'type', u'unit': 'unit', u'uri_transport': 'uri_transport', u'uri_transports': 'uri_transports', u'uuid': 'uuid', u'value': 'value', u'vendor': 'vendor', u'wordsize': 'wordsize'}, '_target': <xml.etree.ElementTree.TreeBuilder object at 0x2357f10>, 'entity': { }, 'parser': <pyexpat.xmlparser object at 0x22e5360>, 'version': 'Expat 2.1.0'}, <pyexpat.xmlparser object at 0x22e5360>, <xml.etree.ElementTree.TreeBuilder object at 0x2357f10>, <bound method XMLParser._start_list of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._end of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._pi of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._data of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._comment of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, <bound method XMLParser._default of <xml.etree.ElementTree.XMLParser object at 0x2357f50>>, { '_data': [], '_elem': [], '_factory': <class 'xml.etree.ElementTree.Element'>, '_last': <Element 'capabilities' at 0x2357f90>, '_tail': 1}, [], <Element 'cell' at 0x23646d0>, <Element 'cells' at 0x2364690>, <Element 'topology' at 0x2364650>, <Element 'host' at 0x2357fd0>, <Element 'capabilities' at 0x2357f90>, { '_children': [ <Element 'memory' at 0x2364710>, <Element 'pages' at 0x2364750>, <Element 'pages' at 0x2364790>, <Element 'distances' at 0x23647d0>, <Element 'cpus' at 0x2364850>], 'attrib': { 'id': '0'}, 'tag': 'cell', 'tail': '\n ', 'text': '\n '}, { '_children': [<Element 'cell' at 0x23646d0>], 'attrib': { 'num': '1'}, 'tag': 'cells', 'tail': '\n ', 'text': '\n '}, { '_children': [<Element 'cells' at 0x2364690>], 'attrib': { }, 'tag': 'topology', 'tail': '\n ', 'text': '\n '}, { '_children': [ <Element 'uuid' at 0x2364050>, <Element 'cpu' at 0x2364090>, <Element 'power_management' at 0x2364390>, <Element 'migration_features' at 0x23644d0>, <Element 'topology' at 0x2364650>, <Element 'secmodel' at 0x2364a90>, <Element 'secmodel' at 0x2364bd0>], 'attrib': { }, 'tag': 'host', 'tail': '\n\n ', 'text': '\n '}, { '_children': [ <Element 'host' at 0x2357fd0>, <Element 'guest' at 0x2364d10>, <Element 'guest' at 0x2369590>], 'attrib': { }, 'tag': 'capabilities', 'text': '\n\n '}, [ <Element 'memory' at 0x2364710>, <Element 'pages' at 0x2364750>, <Element 'pages' at 0x2364790>, <Element 'distances' at 0x23647d0>, <Element 'cpus' at 0x2364850>], [<Element 'cell' at 0x23646d0>], [<Element 'cells' at 0x2364690>], [ <Element 'uuid' at 0x2364050>, <Element 'cpu' at 0x2364090>, <Element 'power_management' at 0x2364390>, <Element 'migration_features' at 0x23644d0>, <Element 'topology' at 0x2364650>, <Element 'secmodel' at 0x2364a90>, <Element 'secmodel' at 0x2364bd0>], [ <Element 'host' at 0x2357fd0>, <Element 'guest' at 0x2364d10>, <Element 'guest' at 0x2369590>], <Element 'migration_features' at 0x23644d0>, <Element 'power_management' at 0x2364390>, <Element 'cpu' at 0x2364090>, <Element 'uuid' at 0x2364050>, { '_children': [ <Element 'live' at 0x2364510>, <Element 'uri_transports' at 0x2364590>], 'attrib': { }, 'tag': 'migration_features', 'tail': '\n ', 'text': '\n '}, { '_children': [ <Element 'suspend_mem' at 0x23643d0>, <Element 'suspend_disk' at 0x2364410>, <Element 'suspend_hybrid' at 0x2364490>], 'attrib': { }, 'tag': 'power_management', 'tail': '\n ', 'text': '\n '}, { '_children': [ <Element 'arch' at 0x23640d0>, <Element 'model' at 0x2364110>, <Element 'vendor' at 0x2364150>, <Element 'topology' at 0x2364190>, <Element 'feature' at 0x23641d0>, <Element 'feature' at 0x2364210>, <Element 'feature' at 0x2364250>, <Element 'feature' at 0x2364290>, <Element 'feature' at 0x23642d0>, <Element 'pages' at 0x2364310>, <Element 'pages' at 0x2364350>], 'attrib': { }, 'tag': 'cpu', 'tail': '\n ', 'text': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'uuid', 'tail': '\n ', 'text': 'a920d963-c18f-4fda-aa5a-deee9bbcde9d'}, [<Element 'live' at 0x2364510>, <Element 'uri_transports' at 0x2364590>], [ <Element 'suspend_mem' at 0x23643d0>, <Element 'suspend_disk' at 0x2364410>, <Element 'suspend_hybrid' at 0x2364490>], [ <Element 'arch' at 0x23640d0>, <Element 'model' at 0x2364110>, <Element 'vendor' at 0x2364150>, <Element 'topology' at 0x2364190>, <Element 'feature' at 0x23641d0>, <Element 'feature' at 0x2364210>, <Element 'feature' at 0x2364250>, <Element 'feature' at 0x2364290>, <Element 'feature' at 0x23642d0>, <Element 'pages' at 0x2364310>, <Element 'pages' at 0x2364350>], [], <Element 'uri_transports' at 0x2364590>, <Element 'live' at 0x2364510>, <Element 'suspend_hybrid' at 0x2364490>, <Element 'suspend_disk' at 0x2364410>, <Element 'suspend_mem' at 0x23643d0>, <Element 'pages' at 0x2364350>, <Element 'pages' at 0x2364310>, <Element 'feature' at 0x23642d0>, <Element 'feature' at 0x2364290>, <Element 'feature' at 0x2364250>, <Element 'feature' at 0x2364210>, <Element 'feature' at 0x23641d0>, <Element 'topology' at 0x2364190>, <Element 'vendor' at 0x2364150>, <Element 'model' at 0x2364110>, <Element 'arch' at 0x23640d0>, { '_children': [ <Element 'uri_transport' at 0x23645d0>, <Element 'uri_transport' at 0x2364610>], 'attrib': { }, 'tag': 'uri_transports', 'tail': '\n ', 'text': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'live', 'tail': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'suspend_hybrid', 'tail': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'suspend_disk', 'tail': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'suspend_mem', 'tail': '\n '}, { '_children': [], 'attrib': { 'size': '2048', 'unit': 'KiB'}, 'tag': 'pages', 'tail': '\n '}, { '_children': [], 'attrib': { 'size': '4', 'unit': 'KiB'}, 'tag': 'pages', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'ss'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'vmx'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'pcid'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'osxsave'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'name': 'hypervisor'}, 'tag': 'feature', 'tail': '\n '}, { '_children': [], 'attrib': { 'cores': '1', 'sockets': '8', 'threads': '1'}, 'tag': 'topology', 'tail': '\n '}, { '_children': [], 'attrib': { }, 'tag': 'vendor', 'tail': '\n ', 'text': 'Intel'}, { '_children': [], 'attrib': { }, 'tag': 'model', 'tail': '\n ', 'text': 'IvyBridge'}, { '_children': [], 'attrib': { }, 'tag': 'arch', 'tail': '\n ', 'text': 'x86_64'}, [ <Element 'uri_transport' at 0x23645d0>, <Element 'uri_transport' at 0x2364610>], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], <Element 'uri_transport' at 0x2364610>, <Element 'uri_transport' at 0x23645d0>, { '_children': [], 'attrib': { }, 'tag': 'uri_transport', 'tail': '\n ', 'text': 'rdma'}, { '_children': [], 'attrib': { }, 'tag': 'uri_transport', 'tail': '\n ', 'text': 'tcp'}, [], [], (<type '_ctypes.Array'>,), { '__dict__': <attribute '__dict__' of 'c_char_Array_60' objects>, '__doc__': None, '__module__': 'vdsm.netlink', '__weakref__': <attribute '__weakref__' of 'c_char_Array_60' objects>, '_length_': 60, '_type_': <class 'ctypes.c_char'>, 'raw': <attribute 'raw' of 'c_char_Array_60' objects>, 'value': <attribute 'value' of 'c_char_Array_60' objects>}, <class 'vdsm.netlink.c_char_Array_60'>, <attribute '__dict__' of 'c_char_Array_60' objects>, <attribute '__weakref__' of 'c_char_Array_60' objects>, ( <class 'vdsm.netlink.c_char_Array_60'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_60' objects>, <attribute 'value' of 'c_char_Array_60' objects>]

On Fri, Jan 15, 2016 at 11:22 AM, Francesco Romani <fromani@redhat.com> wrote:

----- Original Message -----

...

From: "Nir Soffer" <nsoffer@redhat.com> To: "devel" <devel@ovirt.org>, "Francesco Romani" <fromani@redhat.com> Sent: Friday, January 15, 2016 9:52:13 AM Subject: Re: [VDSM] health: Introduce Vdsm health monitoring

I tested with https://gerrit.ovirt.org/#/c/51630

libvirt.virDomain object does not leak any more, but pthreading object are still leaking, and the number of uncollectible objects is still very high (12,915).

Each time we start and stop a vm, we leak about 6000 objects which are part of a reference cycle, and each time we create and leak 27 pthreading locks/conditions which are probably the reason for this leak (because they implement __del__ or reference object implementing __del__.

*Surely* this deserve more investigation, *most likley* there is more to be fixed, and *surely* I'll invest more energy in research, but I think we should also apply some grains of salt to the raw results gc gives us. I've been running a test overnight like

- start 20 vms - sleep 2m - stop 20 vms - sleep 30s

repeated 100 times, monitoring VmRSS (/proc/$VDSM_PID/status) and I didn't observed any _significant_ leak. The VmRSS growth was pretty low, almost flat. So something doesn't click here.

I mean, 6000 objects (at least) per vm each time *should* be noticeable after all!

Once 51630 is in good shape I'll broad the net and go hunt for more virt leaks.

After removing the gc.DEBUG_SAVEALL, we see now only the libvirt.virDomain leak. Turns out that the pthreading objects leak was caused by saving all the unreachable objects when using gc.DEBUG_SAVEALL. Here is a run with https://gerrit.ovirt.org/51708, but without https://gerrit.ovirt.org/51630 # tail -f /var/log/vdsm/vdsm.log | grep health MainThread::INFO::2016-01-15 16:14:14,789::health::58::health::(start) Starting health monitor (interval=60) Thread-15::DEBUG::2016-01-15 16:14:14,793::health::70::health::(_run) Health monitor started Thread-15::DEBUG::2016-01-15 16:15:14,794::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 16:15:14,815::health::86::health::(_check) Collected 54 objects activating host... Thread-15::DEBUG::2016-01-15 16:16:14,815::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 16:16:14,837::health::86::health::(_check) Collected 299 objects Thread-15::DEBUG::2016-01-15 16:17:14,838::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 16:17:14,858::health::86::health::(_check) Collected 128 objects starting vm... Thread-15::DEBUG::2016-01-15 16:18:14,858::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 16:18:14,883::health::86::health::(_check) Collected 1771 objects Thread-15::DEBUG::2016-01-15 16:19:14,884::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 16:19:14,909::health::86::health::(_check) Collected 167 objects stopping vm... Thread-15::DEBUG::2016-01-15 16:20:14,909::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 16:20:14,934::health::86::health::(_check) Collected 179 objects vm is down... Thread-15::DEBUG::2016-01-15 16:21:14,934::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 16:21:14,960::health::86::health::(_check) Collected 2284 objects Thread-15::WARNING::2016-01-15 16:21:14,960::health::94::health::(_check) Found 1 uncollectable objects: ['<libvirt.virDomain object at 0x1ef6550>'] starting vm... Thread-15::DEBUG::2016-01-15 16:22:14,961::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 16:22:14,987::health::86::health::(_check) Collected 1721 objects Thread-15::WARNING::2016-01-15 16:22:14,987::health::94::health::(_check) Found 1 uncollectable objects: ['<libvirt.virDomain object at 0x1ef6550>'] stopping vm... Thread-15::DEBUG::2016-01-15 16:23:14,988::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 16:23:15,014::health::86::health::(_check) Collected 189 objects Thread-15::WARNING::2016-01-15 16:23:15,014::health::94::health::(_check) Found 1 uncollectable objects: ['<libvirt.virDomain object at 0x1ef6550>'] Thread-15::DEBUG::2016-01-15 16:24:15,014::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 16:24:15,040::health::86::health::(_check) Collected 2240 objects Thread-15::WARNING::2016-01-15 16:24:15,041::health::94::health::(_check) Found 2 uncollectable objects: ['<libvirt.virDomain object at 0x1ef6550>', '<libvirt.virDomain object at 0x7f79ec10cf90>'] vm is down here (probbaly was done before the log) Thread-15::DEBUG::2016-01-15 16:25:15,041::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 16:25:15,065::health::86::health::(_check) Collected 128 objects Thread-15::WARNING::2016-01-15 16:25:15,065::health::94::health::(_check) Found 2 uncollectable objects: ['<libvirt.virDomain object at 0x1ef6550>', '<libvirt.virDomain object at 0x7f79ec10cf90>'] starting vm... Thread-15::DEBUG::2016-01-15 16:26:15,067::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 16:26:15,094::health::86::health::(_check) Collected 1749 objects Thread-15::WARNING::2016-01-15 16:26:15,094::health::94::health::(_check) Found 2 uncollectable objects: ['<libvirt.virDomain object at 0x1ef6550>', '<libvirt.virDomain object at 0x7f79ec10cf90>'] stopping vm... Thread-15::DEBUG::2016-01-15 16:27:15,096::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 16:27:15,119::health::86::health::(_check) Collected 167 objects Thread-15::WARNING::2016-01-15 16:27:15,120::health::94::health::(_check) Found 2 uncollectable objects: ['<libvirt.virDomain object at 0x1ef6550>', '<libvirt.virDomain object at 0x7f79ec10cf90>'] vm is shutting down... Thread-15::DEBUG::2016-01-15 16:28:15,120::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 16:28:15,144::health::86::health::(_check) Collected 2262 objects Thread-15::WARNING::2016-01-15 16:28:15,144::health::94::health::(_check) Found 3 uncollectable objects: ['<libvirt.virDomain object at 0x1ef6550>', '<libvirt.virDomain object at 0x7f79ec10cf90>', '<libvirt.virDomain object at 0x1f0a2d0>'] vm is down... Thread-15::DEBUG::2016-01-15 16:29:15,145::health::84::health::(_check) Checking health Thread-15::DEBUG::2016-01-15 16:29:15,167::health::86::health::(_check) Collected 128 objects Thread-15::WARNING::2016-01-15 16:29:15,167::health::94::health::(_check) Found 3 uncollectable objects: ['<libvirt.virDomain object at 0x1ef6550>', '<libvirt.virDomain object at 0x7f79ec10cf90>', '<libvirt.virDomain object at 0x1f0a2d0>'] When running with both https://gerrit.ovirt.org/51708 and https://gerrit.ovirt.org/51630, no object leak because of reference cycles and __del__. I'll abandon https://gerrit.ovirt.org/51868 since it is not needed. Nir