2019-04-13 21:16:30,190+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:16:33,222+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=b18b1a12-b498-46d2-ae3c-840d50a01ae8 (api:48) 2019-04-13 21:16:33,226+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=b18b1a12-b498-46d2-ae3c-840d50a01ae8 (api:54) 2019-04-13 21:16:33,226+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:16:33,240+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=f81f9e3f-c382-49a9-bc7f-f8a08d2747fc (api:48) 2019-04-13 21:16:33,250+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707244642304', 'isoprefix': '', 'alerts': [], 'disktotal': '3898342375424', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707244642304', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719993229312', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707244642304', 'isoprefix': '', 'alerts': [], 'disktotal': '4364633636864', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=f81f9e3f-c382-49a9-bc7f-f8a08d2747fc (api:54) 2019-04-13 21:16:33,252+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:16:35,144+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:16:35,176+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:16:35,177+0200 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:16:35,178+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:16:37,099+0200 WARN (qgapoller/9) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:16:41,380+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=d30471d0-e53d-4d45-ac56-99752544bbb3 (api:48) 2019-04-13 21:16:41,381+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000727359', 'lastCheck': '7.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00084636', 'lastCheck': '7.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000523396', 'lastCheck': '7.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000374187', 'lastCheck': '4.5', 'valid': True}} from=internal, task_id=d30471d0-e53d-4d45-ac56-99752544bbb3 (api:54) 2019-04-13 21:16:42,788+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:16:42,859+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=cb7efdaf-412d-4258-b4bb-94d28376e664 (api:48) 2019-04-13 21:16:42,859+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000727359', 'lastCheck': '8.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00084636', 'lastCheck': '8.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000523396', 'lastCheck': '8.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000374187', 'lastCheck': '6.0', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=cb7efdaf-412d-4258-b4bb-94d28376e664 (api:54) 2019-04-13 21:16:42,860+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=00a1917e-34b9-4456-9060-0932fea223c8 (api:48) 2019-04-13 21:16:42,860+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=00a1917e-34b9-4456-9060-0932fea223c8 (api:54) 2019-04-13 21:16:42,872+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '29.29', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '68.26'}, '21': {'cpuUser': '27.30', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '69.92'}, '22': {'cpuUser': '32.27', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '65.28'}, '23': {'cpuUser': '19.15', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '78.00'}, '1': {'cpuUser': '46.85', 'nodeIndex': 0, 'cpuSys': '4.24', 'cpuIdle': '48.91'}, '0': {'cpuUser': '54.94', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '43.34'}, '3': {'cpuUser': '56.46', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '41.55'}, '2': {'cpuUser': '54.94', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '43.14'}, '5': {'cpuUser': '52.02', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '46.19'}, '4': {'cpuUser': '56.33', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '41.75'}, '7': {'cpuUser': '41.29', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '55.86'}, '6': {'cpuUser': '46.45', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '51.23'}, '9': {'cpuUser': '39.17', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '57.85'}, '8': {'cpuUser': '41.62', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '55.73'}, '11': {'cpuUser': '42.74', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '54.48'}, '10': {'cpuUser': '41.68', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '55.60'}, '13': {'cpuUser': '27.24', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '70.57'}, '12': {'cpuUser': '29.09', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '68.86'}, '15': {'cpuUser': '26.24', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '71.51'}, '14': {'cpuUser': '24.85', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '72.96'}, '17': {'cpuUser': '29.56', 'nodeIndex': 0, 'cpuSys': '2.65', 'cpuIdle': '67.79'}, '16': {'cpuUser': '25.25', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '72.89'}, '19': {'cpuUser': '28.89', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '68.79'}, '18': {'cpuUser': '25.65', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '71.77'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '71317'}, '0': {'memPercent': 23, 'memFree': '76404'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000727359', 'lastCheck': '8.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000523396', 'lastCheck': '8.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000374187', 'lastCheck': '6.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00084636', 'lastCheck': '8.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168834101', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '5091484318', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590425755827', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '2705546980034', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433722851837', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '258974056381', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239848593', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '366089574', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122779615', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121301112', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423572422214', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '253553588643', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215258911', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680229955', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '29545896351', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6651302556', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '29289188566', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255742384434', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '7013859955', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121675275', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '567173', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75240556109', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '61732344691', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122234595', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418911769', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '1012771265', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121487042', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183002.858235, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183002.858235, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183002.858235, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183002.858235, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121257143', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '221638', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458720809437', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183002.858235, 'rx': '2576908854332', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1321892580044', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183002.858235, 'rx': '1321892580044', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183002.858235, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940794702', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '2700596128523', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571484961125', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '4950851511', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329349286934', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183002.858235, 'rx': '979816456106', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183002.858235, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183002.858235, 'rx': '45344334', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183002.858235, 'rx': '39825238', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40570', 'ksmPages': 100, 'elapsedTime': '1727750.95', 'cpuLoad': '8.41', 'cpuSys': '2.43', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.18', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150379, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:16:42 GMT', 'cpuUser': '37.47', 'memFree': 150123, 'cpuIdle': '60.10', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.26'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:16:42,878+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:16:43,300+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=d6ef234e-ffe7-4507-abdc-af50f9864e9b (api:48) 2019-04-13 21:16:43,305+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=d6ef234e-ffe7-4507-abdc-af50f9864e9b (api:54) 2019-04-13 21:16:43,305+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:16:43,320+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=964c198a-b684-4522-9a1b-25298b1104fa (api:48) 2019-04-13 21:16:43,328+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707238350848', 'isoprefix': '', 'alerts': [], 'disktotal': '3898336083968', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707238350848', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719986937856', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707238350848', 'isoprefix': '', 'alerts': [], 'disktotal': '4364626296832', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=964c198a-b684-4522-9a1b-25298b1104fa (api:54) 2019-04-13 21:16:43,330+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:16:45,227+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:16:45,271+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:16:45,271+0200 ERROR (jsonrpc/0) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:16:45,273+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:16:47,106+0200 WARN (qgapoller/6) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:16:50,186+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:16:50,225+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:16:50,226+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:16:50,227+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:16:53,382+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=1bfc4936-b031-4a59-ae8c-337b9825baf6 (api:48) 2019-04-13 21:16:53,387+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=1bfc4936-b031-4a59-ae8c-337b9825baf6 (api:54) 2019-04-13 21:16:53,387+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:16:53,506+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=b836ca28-43e5-44d0-bd4f-553afc319662 (api:48) 2019-04-13 21:16:53,513+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707238350848', 'isoprefix': '', 'alerts': [], 'disktotal': '3898336083968', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707238350848', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719986937856', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707238350848', 'isoprefix': '', 'alerts': [], 'disktotal': '4364626296832', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=b836ca28-43e5-44d0-bd4f-553afc319662 (api:54) 2019-04-13 21:16:53,514+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:16:56,479+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=18869962-4a0e-4018-8471-c47e86a27310 (api:48) 2019-04-13 21:16:56,480+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000734907', 'lastCheck': '2.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000706679', 'lastCheck': '2.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000377165', 'lastCheck': '2.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000419556', 'lastCheck': '2.3', 'valid': True}} from=internal, task_id=18869962-4a0e-4018-8471-c47e86a27310 (api:54) 2019-04-13 21:16:57,857+0200 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:16:57,919+0200 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=753a61f2-65cf-4d3a-920a-f90a71fa8e87 (api:48) 2019-04-13 21:16:57,920+0200 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000734907', 'lastCheck': '3.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000706679', 'lastCheck': '3.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000377165', 'lastCheck': '3.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000419556', 'lastCheck': '1.1', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=753a61f2-65cf-4d3a-920a-f90a71fa8e87 (api:54) 2019-04-13 21:16:57,920+0200 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=1cc79582-920d-4a99-9479-ac58f099ad5c (api:48) 2019-04-13 21:16:57,921+0200 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=1cc79582-920d-4a99-9479-ac58f099ad5c (api:54) 2019-04-13 21:16:57,934+0200 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '20.56', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '76.92'}, '21': {'cpuUser': '21.22', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '75.73'}, '22': {'cpuUser': '17.57', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '79.71'}, '23': {'cpuUser': '25.33', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '71.62'}, '1': {'cpuUser': '40.91', 'nodeIndex': 0, 'cpuSys': '4.51', 'cpuIdle': '54.58'}, '0': {'cpuUser': '42.24', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '55.51'}, '3': {'cpuUser': '51.32', 'nodeIndex': 0, 'cpuSys': '2.06', 'cpuIdle': '46.62'}, '2': {'cpuUser': '45.88', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '52.26'}, '5': {'cpuUser': '46.08', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '52.13'}, '4': {'cpuUser': '44.56', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '53.58'}, '7': {'cpuUser': '44.43', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '52.79'}, '6': {'cpuUser': '37.99', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '59.29'}, '9': {'cpuUser': '40.32', 'nodeIndex': 1, 'cpuSys': '3.25', 'cpuIdle': '56.43'}, '8': {'cpuUser': '39.32', 'nodeIndex': 1, 'cpuSys': '3.32', 'cpuIdle': '57.36'}, '11': {'cpuUser': '40.78', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '56.57'}, '10': {'cpuUser': '45.75', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '51.47'}, '13': {'cpuUser': '21.48', 'nodeIndex': 0, 'cpuSys': '2.52', 'cpuIdle': '76.00'}, '12': {'cpuUser': '30.57', 'nodeIndex': 0, 'cpuSys': '2.06', 'cpuIdle': '67.37'}, '15': {'cpuUser': '18.23', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '79.58'}, '14': {'cpuUser': '20.69', 'nodeIndex': 0, 'cpuSys': '2.52', 'cpuIdle': '76.79'}, '17': {'cpuUser': '24.27', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '73.81'}, '16': {'cpuUser': '26.92', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '71.49'}, '19': {'cpuUser': '34.41', 'nodeIndex': 1, 'cpuSys': '1.79', 'cpuIdle': '63.80'}, '18': {'cpuUser': '34.02', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '63.73'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '70975'}, '0': {'memPercent': 22, 'memFree': '76747'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000734907', 'lastCheck': '3.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000377165', 'lastCheck': '3.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000419556', 'lastCheck': '1.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000706679', 'lastCheck': '3.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168834871', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '5091484318', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590426448990', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '2705548537199', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433726308303', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '258976722584', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239849363', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '366089574', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122780385', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121301882', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423576150862', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '253556559262', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215259681', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680235884', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '29545901098', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6651304011', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '29289189163', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255745037386', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '7013912459', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121676045', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '567173', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75242103963', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '61732356298', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122235365', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418912189', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '1012771265', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121487812', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183017.918082, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183017.918082, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183017.918082, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183017.918082, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121257913', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '221638', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458721468331', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183017.918082, 'rx': '2576910336204', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1321905036675', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183017.918082, 'rx': '1321905036675', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183017.918082, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940799527', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '2700597685560', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571485649463', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '4950851639', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329352366536', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183017.918082, 'rx': '979825571434', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183017.918082, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183017.918082, 'rx': '45344724', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183017.918082, 'rx': '39825560', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40570', 'ksmPages': 100, 'elapsedTime': '1727766.02', 'cpuLoad': '8.33', 'cpuSys': '2.50', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '2.72', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150379, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:16:57 GMT', 'cpuUser': '33.96', 'memFree': 150123, 'cpuIdle': '63.54', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.39'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:16:57,940+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.08 seconds (__init__:312) 2019-04-13 21:17:00,306+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:17:00,336+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:17:00,337+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:17:00,337+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:17:03,562+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=2fbeccc8-0611-4c53-95e1-29147a433556 (api:48) 2019-04-13 21:17:03,567+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=2fbeccc8-0611-4c53-95e1-29147a433556 (api:54) 2019-04-13 21:17:03,567+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:17:03,581+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=e46616a6-10c6-4f4c-ae76-c4a5f6725ddd (api:48) 2019-04-13 21:17:03,591+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707238350848', 'isoprefix': '', 'alerts': [], 'disktotal': '3898336083968', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707238350848', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719986937856', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707238350848', 'isoprefix': '', 'alerts': [], 'disktotal': '4364626296832', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=e46616a6-10c6-4f4c-ae76-c4a5f6725ddd (api:54) 2019-04-13 21:17:03,593+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:17:04,113+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID=u'ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', options=None) from=internal, task_id=72c36bc9-4fe2-49d4-b4b0-31a4575e084b (api:48) 2019-04-13 21:17:04,115+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', options=None) from=internal, task_id=a38ce7fc-61ac-4d0e-91e9-0cb79ca2fc8e (api:48) 2019-04-13 21:17:04,117+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', options=None) from=internal, task_id=7bc9dcdf-081f-411a-9421-7ac167d455ca (api:48) 2019-04-13 21:17:04,120+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '4579696128', 'apparentsize': '16106127360'} from=internal, task_id=a38ce7fc-61ac-4d0e-91e9-0cb79ca2fc8e (api:54) 2019-04-13 21:17:04,121+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID='ef6741d1-380a-4736-a924-f88ed68e1270', options=None) from=internal, task_id=cdedbf3f-27a7-4934-9eed-e50752fdef4b (api:48) 2019-04-13 21:17:04,122+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '53760980480', 'apparentsize': '139586437120'} from=internal, task_id=72c36bc9-4fe2-49d4-b4b0-31a4575e084b (api:54) 2019-04-13 21:17:04,122+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID=u'67a4dd71-69ff-49a8-b8a4-c7101c75e105', options=None) from=internal, task_id=8b6875fc-277d-440e-ab35-7ab373dd16cb (api:48) 2019-04-13 21:17:04,124+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '5697679360', 'apparentsize': '16106127360'} from=internal, task_id=7bc9dcdf-081f-411a-9421-7ac167d455ca (api:54) 2019-04-13 21:17:04,125+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', options=None) from=internal, task_id=80ccda56-ca6c-412c-b240-b9e6dc595545 (api:48) 2019-04-13 21:17:04,128+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '6477467136', 'apparentsize': '21474836480'} from=internal, task_id=cdedbf3f-27a7-4934-9eed-e50752fdef4b (api:54) 2019-04-13 21:17:04,129+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID=u'2b136ed7-8b9a-45c1-9edc-64ee96c87c43', options=None) from=internal, task_id=b1019e0e-f52f-47ff-8758-5ecc611ed7c6 (api:48) 2019-04-13 21:17:04,130+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '55059142656', 'apparentsize': '171798691840'} from=internal, task_id=8b6875fc-277d-440e-ab35-7ab373dd16cb (api:54) 2019-04-13 21:17:04,131+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID=u'44b3ceca-f0db-43da-af29-aa681df146ac', options=None) from=internal, task_id=16bf9615-c3de-4ae7-a0d8-fbfe56d3237c (api:48) 2019-04-13 21:17:04,132+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=e90c2962-5f4e-427b-ad80-6bb591be98c4 (api:48) 2019-04-13 21:17:04,134+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2524568064', 'apparentsize': '12884901888'} from=internal, task_id=80ccda56-ca6c-412c-b240-b9e6dc595545 (api:54) 2019-04-13 21:17:04,135+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID='123e2cc8-8f24-4ddd-b401-68393535f555', options=None) from=internal, task_id=9c94422b-5645-47f4-bb1c-ef16d0cebee7 (api:48) 2019-04-13 21:17:04,138+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '11196705280', 'apparentsize': '21474836480'} from=internal, task_id=16bf9615-c3de-4ae7-a0d8-fbfe56d3237c (api:54) 2019-04-13 21:17:04,138+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID='3f6c31c6-ec38-4979-a617-55537123ee61', options=None) from=internal, task_id=bdfbf52d-1138-43af-a2b1-eb926990b834 (api:48) 2019-04-13 21:17:04,139+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2024405504', 'apparentsize': '10737418240'} from=internal, task_id=b1019e0e-f52f-47ff-8758-5ecc611ed7c6 (api:54) 2019-04-13 21:17:04,140+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', options=None) from=internal, task_id=c5887753-aaa5-47e0-a7da-db05cbd3a460 (api:48) 2019-04-13 21:17:04,142+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2116492288', 'apparentsize': '3948150784'} from=internal, task_id=e90c2962-5f4e-427b-ad80-6bb591be98c4 (api:54) 2019-04-13 21:17:04,143+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID=u'a1d583c1-e135-4e3f-b6fa-0f29a5261366', options=None) from=internal, task_id=29019b4c-8392-4d7a-b554-89a3c1717d72 (api:48) 2019-04-13 21:17:04,143+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2282998272', 'apparentsize': '16106127360'} from=internal, task_id=9c94422b-5645-47f4-bb1c-ef16d0cebee7 (api:54) 2019-04-13 21:17:04,144+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', options=None) from=internal, task_id=cdfeebcf-e6a0-4e5f-af7b-6e5e7cedd9aa (api:48) 2019-04-13 21:17:04,147+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '4998727680', 'apparentsize': '16106127360'} from=internal, task_id=bdfbf52d-1138-43af-a2b1-eb926990b834 (api:54) 2019-04-13 21:17:04,148+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID='a884b4ce-381f-4653-be14-a31d15181f48', options=None) from=internal, task_id=331d1afb-8ae8-458f-b838-d508b46e3805 (api:48) 2019-04-13 21:17:04,150+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '1294900224', 'apparentsize': '12884901888'} from=internal, task_id=29019b4c-8392-4d7a-b554-89a3c1717d72 (api:54) 2019-04-13 21:17:04,151+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', options=None) from=internal, task_id=7bb9a807-2823-47c8-bbac-b43ab4b01441 (api:48) 2019-04-13 21:17:04,152+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '7263196672', 'apparentsize': '21474836480'} from=internal, task_id=cdfeebcf-e6a0-4e5f-af7b-6e5e7cedd9aa (api:54) 2019-04-13 21:17:04,152+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2109410304', 'apparentsize': '8589934592'} from=internal, task_id=c5887753-aaa5-47e0-a7da-db05cbd3a460 (api:54) 2019-04-13 21:17:04,155+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '3447297536', 'apparentsize': '16106127360'} from=internal, task_id=331d1afb-8ae8-458f-b838-d508b46e3805 (api:54) 2019-04-13 21:17:04,156+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2879759360', 'apparentsize': '8589934592'} from=internal, task_id=7bb9a807-2823-47c8-bbac-b43ab4b01441 (api:54) 2019-04-13 21:17:05,236+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:17:05,266+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:17:05,267+0200 ERROR (jsonrpc/0) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:17:05,268+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:17:11,609+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=6fe09577-3ae8-4bb3-9543-878e589be132 (api:48) 2019-04-13 21:17:11,612+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000439215', 'lastCheck': '7.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000672642', 'lastCheck': '7.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000649007', 'lastCheck': '7.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000433098', 'lastCheck': '4.7', 'valid': True}} from=internal, task_id=6fe09577-3ae8-4bb3-9543-878e589be132 (api:54) 2019-04-13 21:17:12,910+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:17:12,980+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=81b731e7-d8a1-4049-ae29-890add69ac8f (api:48) 2019-04-13 21:17:12,981+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000439215', 'lastCheck': '8.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000672642', 'lastCheck': '8.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000649007', 'lastCheck': '8.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000433098', 'lastCheck': '6.1', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=81b731e7-d8a1-4049-ae29-890add69ac8f (api:54) 2019-04-13 21:17:12,982+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=af00cf86-2448-47e8-8ecb-4ceb1e6fc3b5 (api:48) 2019-04-13 21:17:12,982+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=af00cf86-2448-47e8-8ecb-4ceb1e6fc3b5 (api:54) 2019-04-13 21:17:12,995+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '19.86', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '77.56'}, '21': {'cpuUser': '13.44', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '83.91'}, '22': {'cpuUser': '10.33', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '87.09'}, '23': {'cpuUser': '17.01', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '80.67'}, '1': {'cpuUser': '37.20', 'nodeIndex': 0, 'cpuSys': '3.91', 'cpuIdle': '58.89'}, '0': {'cpuUser': '44.02', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '53.99'}, '3': {'cpuUser': '42.23', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '55.92'}, '2': {'cpuUser': '46.87', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '51.08'}, '5': {'cpuUser': '42.30', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '55.65'}, '4': {'cpuUser': '44.29', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '53.66'}, '7': {'cpuUser': '27.07', 'nodeIndex': 1, 'cpuSys': '3.57', 'cpuIdle': '69.36'}, '6': {'cpuUser': '31.51', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '65.64'}, '9': {'cpuUser': '34.69', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '62.20'}, '8': {'cpuUser': '32.77', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '64.18'}, '11': {'cpuUser': '32.44', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '64.51'}, '10': {'cpuUser': '32.97', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '63.72'}, '13': {'cpuUser': '22.64', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '75.37'}, '12': {'cpuUser': '22.84', 'nodeIndex': 0, 'cpuSys': '1.52', 'cpuIdle': '75.64'}, '15': {'cpuUser': '32.17', 'nodeIndex': 0, 'cpuSys': '1.32', 'cpuIdle': '66.51'}, '14': {'cpuUser': '19.79', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '78.09'}, '17': {'cpuUser': '19.33', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '78.75'}, '16': {'cpuUser': '16.81', 'nodeIndex': 0, 'cpuSys': '2.52', 'cpuIdle': '80.67'}, '19': {'cpuUser': '23.70', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '74.05'}, '18': {'cpuUser': '24.76', 'nodeIndex': 1, 'cpuSys': '1.65', 'cpuIdle': '73.59'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '70946'}, '0': {'memPercent': 22, 'memFree': '76775'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000439215', 'lastCheck': '8.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000649007', 'lastCheck': '8.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000433098', 'lastCheck': '6.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000672642', 'lastCheck': '8.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168835174', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '5091484318', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590427708752', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '2705549694411', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433730257119', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '258978676006', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239850002', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '366089972', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122780688', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121302185', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423579808634', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '253558974611', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215259984', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680235944', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '29545901341', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6651306304', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '29289191132', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255746967201', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '7013963932', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121676348', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '567173', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75244327595', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '61739469475', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122235668', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418912897', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '1012771523', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121488115', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183032.979856, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183032.979856, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183032.979856, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183032.979856, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121258216', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '221638', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458722687055', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183032.979856, 'rx': '2576911438875', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1321915399562', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183032.979856, 'rx': '1321915399562', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183032.979856, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940800463', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '2700598839960', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571486908289', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '4950854451', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329362370202', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183032.979856, 'rx': '979835510878', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183032.979856, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183032.979856, 'rx': '45345114', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183032.979856, 'rx': '39825928', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40570', 'ksmPages': 100, 'elapsedTime': '1727781.07', 'cpuLoad': '8.25', 'cpuSys': '2.43', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.51', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150378, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:17:12 GMT', 'cpuUser': '28.79', 'memFree': 150122, 'cpuIdle': '68.78', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.13'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:17:13,001+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:17:13,635+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=1391c828-9748-40e9-a98b-8583fe823d39 (api:48) 2019-04-13 21:17:13,640+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=1391c828-9748-40e9-a98b-8583fe823d39 (api:54) 2019-04-13 21:17:13,640+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:17:13,659+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=806a55dd-f034-431e-9a15-85ef74e9aec3 (api:48) 2019-04-13 21:17:13,667+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707231010816', 'isoprefix': '', 'alerts': [], 'disktotal': '3898328743936', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707231010816', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719979597824', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707231010816', 'isoprefix': '', 'alerts': [], 'disktotal': '4364620005376', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=806a55dd-f034-431e-9a15-85ef74e9aec3 (api:54) 2019-04-13 21:17:13,668+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:17:15,372+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:17:15,415+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:17:15,415+0200 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:17:15,416+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.05 seconds (__init__:312) 2019-04-13 21:17:20,275+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:17:20,305+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:17:20,305+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:17:20,306+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:17:23,286+0200 INFO (monitor/42bb677) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_isos) Closing client (__init__:584) 2019-04-13 21:17:23,287+0200 INFO (monitor/42bb677) [IOProcessClient] (/glusterSD/onode-2.discovery.intern:vol-node-2) Closing client (__init__:584) 2019-04-13 21:17:23,287+0200 INFO (monitor/42bb677) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_vms) Closing client (__init__:584) 2019-04-13 21:17:23,287+0200 INFO (monitor/42bb677) [IOProcessClient] (/onode-1.discovery.intern:_mnt_store) Closing client (__init__:584) 2019-04-13 21:17:23,287+0200 INFO (monitor/42bb677) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Closing client (__init__:584) 2019-04-13 21:17:23,727+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=8e9554a5-5c82-4dff-8c87-7c0328d338b1 (api:48) 2019-04-13 21:17:23,732+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=8e9554a5-5c82-4dff-8c87-7c0328d338b1 (api:54) 2019-04-13 21:17:23,732+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:17:23,746+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=77e5a5bf-ee37-4b66-9955-46c5bb32ac0e (api:48) 2019-04-13 21:17:23,754+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707231010816', 'isoprefix': '', 'alerts': [], 'disktotal': '3898328743936', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707231010816', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719979597824', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707231010816', 'isoprefix': '', 'alerts': [], 'disktotal': '4364620005376', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=77e5a5bf-ee37-4b66-9955-46c5bb32ac0e (api:54) 2019-04-13 21:17:23,755+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:17:26,727+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=f79e95c7-fa1a-4ad6-86b4-63b9cdfa37c2 (api:48) 2019-04-13 21:17:26,728+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000700082', 'lastCheck': '2.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000798806', 'lastCheck': '2.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000737369', 'lastCheck': '2.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000431666', 'lastCheck': '2.5', 'valid': True}} from=internal, task_id=f79e95c7-fa1a-4ad6-86b4-63b9cdfa37c2 (api:54) 2019-04-13 21:17:27,954+0200 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:17:28,028+0200 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=625bccce-08cf-4457-9ff8-b69687dd8142 (api:48) 2019-04-13 21:17:28,030+0200 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000700082', 'lastCheck': '3.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000798806', 'lastCheck': '3.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000737369', 'lastCheck': '3.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000431666', 'lastCheck': '1.1', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=625bccce-08cf-4457-9ff8-b69687dd8142 (api:54) 2019-04-13 21:17:28,031+0200 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=0b983ba7-d529-4956-80ea-6672d8b079e3 (api:48) 2019-04-13 21:17:28,031+0200 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=0b983ba7-d529-4956-80ea-6672d8b079e3 (api:54) 2019-04-13 21:17:28,057+0200 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '19.68', 'nodeIndex': 1, 'cpuSys': '2.91', 'cpuIdle': '77.41'}, '21': {'cpuUser': '24.11', 'nodeIndex': 1, 'cpuSys': '2.38', 'cpuIdle': '73.51'}, '22': {'cpuUser': '18.82', 'nodeIndex': 1, 'cpuSys': '2.64', 'cpuIdle': '78.54'}, '23': {'cpuUser': '21.93', 'nodeIndex': 1, 'cpuSys': '2.77', 'cpuIdle': '75.30'}, '1': {'cpuUser': '45.64', 'nodeIndex': 0, 'cpuSys': '4.36', 'cpuIdle': '50.00'}, '0': {'cpuUser': '40.75', 'nodeIndex': 0, 'cpuSys': '1.98', 'cpuIdle': '57.27'}, '3': {'cpuUser': '50.00', 'nodeIndex': 0, 'cpuSys': '2.11', 'cpuIdle': '47.89'}, '2': {'cpuUser': '45.18', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '53.10'}, '5': {'cpuUser': '46.63', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '51.45'}, '4': {'cpuUser': '45.18', 'nodeIndex': 0, 'cpuSys': '1.78', 'cpuIdle': '53.04'}, '7': {'cpuUser': '32.30', 'nodeIndex': 1, 'cpuSys': '2.97', 'cpuIdle': '64.73'}, '6': {'cpuUser': '35.47', 'nodeIndex': 1, 'cpuSys': '3.24', 'cpuIdle': '61.29'}, '9': {'cpuUser': '34.35', 'nodeIndex': 1, 'cpuSys': '3.30', 'cpuIdle': '62.35'}, '8': {'cpuUser': '33.62', 'nodeIndex': 1, 'cpuSys': '3.50', 'cpuIdle': '62.88'}, '11': {'cpuUser': '39.56', 'nodeIndex': 1, 'cpuSys': '2.77', 'cpuIdle': '57.67'}, '10': {'cpuUser': '39.76', 'nodeIndex': 1, 'cpuSys': '2.97', 'cpuIdle': '57.27'}, '13': {'cpuUser': '26.49', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '71.79'}, '12': {'cpuUser': '33.62', 'nodeIndex': 0, 'cpuSys': '1.45', 'cpuIdle': '64.93'}, '15': {'cpuUser': '19.48', 'nodeIndex': 0, 'cpuSys': '1.98', 'cpuIdle': '78.54'}, '14': {'cpuUser': '22.26', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '76.02'}, '17': {'cpuUser': '32.30', 'nodeIndex': 0, 'cpuSys': '1.45', 'cpuIdle': '66.25'}, '16': {'cpuUser': '32.23', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '66.18'}, '19': {'cpuUser': '25.36', 'nodeIndex': 1, 'cpuSys': '2.18', 'cpuIdle': '72.46'}, '18': {'cpuUser': '18.76', 'nodeIndex': 1, 'cpuSys': '2.91', 'cpuIdle': '78.33'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '70847'}, '0': {'memPercent': 22, 'memFree': '76881'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000700082', 'lastCheck': '3.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000737369', 'lastCheck': '3.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000431666', 'lastCheck': '1.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000798806', 'lastCheck': '3.8', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168835390', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '5091484636', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590428134858', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '2705550830936', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433734454319', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '258981548234', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239850002', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '366089972', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122780688', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121302185', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423584206676', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '253561253073', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215259984', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680235944', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '29545901341', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6651314707', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '29289199250', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255749805882', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '7014015561', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121676348', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '567173', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75247231395', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '61745800783', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122235668', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418913317', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '1012771523', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121488115', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183048.02642, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183048.02642, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183048.02642, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183048.02642, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121258216', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '221638', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458723080499', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183048.02642, 'rx': '2576912515385', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1321927884341', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183048.02642, 'rx': '1321927884341', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183048.02642, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940809385', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '2700599967594', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571487325473', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '4950863342', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329371298126', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183048.02642, 'rx': '979847574868', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183048.02642, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183048.02642, 'rx': '45345504', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183048.02642, 'rx': '39826250', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1727796.12', 'cpuLoad': '8.14', 'cpuSys': '2.42', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.30', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150374, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:17:28 GMT', 'cpuUser': '32.64', 'memFree': 150118, 'cpuIdle': '64.93', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.45'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:17:28,079+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.12 seconds (__init__:312) 2019-04-13 21:17:30,448+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:17:30,475+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:17:30,475+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:17:30,476+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:17:33,797+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=33174105-3c2e-4f2f-97fc-05a75f1f7818 (api:48) 2019-04-13 21:17:33,802+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=33174105-3c2e-4f2f-97fc-05a75f1f7818 (api:54) 2019-04-13 21:17:33,802+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:17:33,814+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=f12bcf37-acf3-479b-b790-25d6eab14d84 (api:48) 2019-04-13 21:17:33,822+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707224719360', 'isoprefix': '', 'alerts': [], 'disktotal': '3898322452480', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707224719360', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719973306368', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707224719360', 'isoprefix': '', 'alerts': [], 'disktotal': '4364613713920', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=f12bcf37-acf3-479b-b790-25d6eab14d84 (api:54) 2019-04-13 21:17:33,824+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:17:35,313+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:17:35,343+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:17:35,343+0200 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:17:35,344+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:17:41,828+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=9756b576-af9a-4b77-afda-b3a6234c875f (api:48) 2019-04-13 21:17:41,828+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000801825', 'lastCheck': '7.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000658961', 'lastCheck': '7.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00069728', 'lastCheck': '7.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00033324', 'lastCheck': '4.9', 'valid': True}} from=internal, task_id=9756b576-af9a-4b77-afda-b3a6234c875f (api:54) 2019-04-13 21:17:42,997+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:17:43,064+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=f3f9e832-dce6-4a7e-bf0e-a3cc12b77e6c (api:48) 2019-04-13 21:17:43,065+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000801825', 'lastCheck': '8.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000658961', 'lastCheck': '8.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00069728', 'lastCheck': '8.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00033324', 'lastCheck': '6.2', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=f3f9e832-dce6-4a7e-bf0e-a3cc12b77e6c (api:54) 2019-04-13 21:17:43,066+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=0c9b01eb-d0ad-45bf-81f1-3caed6859165 (api:48) 2019-04-13 21:17:43,066+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=0c9b01eb-d0ad-45bf-81f1-3caed6859165 (api:54) 2019-04-13 21:17:43,079+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '22.44', 'nodeIndex': 1, 'cpuSys': '2.91', 'cpuIdle': '74.65'}, '21': {'cpuUser': '30.52', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '67.23'}, '22': {'cpuUser': '25.69', 'nodeIndex': 1, 'cpuSys': '1.72', 'cpuIdle': '72.59'}, '23': {'cpuUser': '19.53', 'nodeIndex': 1, 'cpuSys': '2.18', 'cpuIdle': '78.29'}, '1': {'cpuUser': '40.85', 'nodeIndex': 0, 'cpuSys': '4.83', 'cpuIdle': '54.32'}, '0': {'cpuUser': '44.09', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '53.66'}, '3': {'cpuUser': '49.46', 'nodeIndex': 0, 'cpuSys': '2.18', 'cpuIdle': '48.36'}, '2': {'cpuUser': '42.90', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '55.05'}, '5': {'cpuUser': '41.38', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '56.37'}, '4': {'cpuUser': '48.07', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '49.94'}, '7': {'cpuUser': '39.72', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '57.30'}, '6': {'cpuUser': '40.58', 'nodeIndex': 1, 'cpuSys': '2.38', 'cpuIdle': '57.04'}, '9': {'cpuUser': '34.16', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '63.26'}, '8': {'cpuUser': '39.99', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '57.43'}, '11': {'cpuUser': '40.32', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '56.97'}, '10': {'cpuUser': '38.53', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '58.89'}, '13': {'cpuUser': '28.40', 'nodeIndex': 0, 'cpuSys': '2.18', 'cpuIdle': '69.42'}, '12': {'cpuUser': '33.83', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '64.58'}, '15': {'cpuUser': '23.57', 'nodeIndex': 0, 'cpuSys': '2.52', 'cpuIdle': '73.91'}, '14': {'cpuUser': '33.43', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '64.98'}, '17': {'cpuUser': '26.75', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '70.80'}, '16': {'cpuUser': '23.44', 'nodeIndex': 0, 'cpuSys': '2.65', 'cpuIdle': '73.91'}, '19': {'cpuUser': '18.80', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '78.22'}, '18': {'cpuUser': '30.85', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '66.70'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '71484'}, '0': {'memPercent': 23, 'memFree': '76243'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000801825', 'lastCheck': '8.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00069728', 'lastCheck': '8.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00033324', 'lastCheck': '6.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000658961', 'lastCheck': '8.8', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168835784', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '5091484720', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590429168914', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '2705550882838', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433737663929', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '258983785383', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239850276', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '366089972', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122780962', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121302459', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423587749812', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '253564031790', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215260258', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680236218', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '29545901341', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6651324791', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '29289209000', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255752019678', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '7014068131', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121676622', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '567173', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75250203927', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '61745824057', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122235942', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418913797', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '1012771523', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121488389', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183063.063268, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183063.063268, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183063.063268, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183063.063268, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121258490', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '221638', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458724091193', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183063.063268, 'rx': '2576912555849', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1321938226808', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183063.063268, 'rx': '1321938226808', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183063.063268, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940819087', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '2700600008876', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571488349827', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '4950873962', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329374258937', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183063.063268, 'rx': '979857723090', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183063.063268, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183063.063268, 'rx': '45345894', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183063.063268, 'rx': '39826618', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40566', 'ksmPages': 100, 'elapsedTime': '1727811.16', 'cpuLoad': '8.26', 'cpuSys': '2.46', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '2.91', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150385, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:17:43 GMT', 'cpuUser': '34.06', 'memFree': 150129, 'cpuIdle': '63.48', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.26'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:17:43,086+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:17:43,866+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=2cfb4a86-be2c-419b-8f68-5f0a3e40058c (api:48) 2019-04-13 21:17:43,871+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=2cfb4a86-be2c-419b-8f68-5f0a3e40058c (api:54) 2019-04-13 21:17:43,872+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:17:43,883+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=09577e94-971e-47ba-8078-baeb594803de (api:48) 2019-04-13 21:17:43,893+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707222622208', 'isoprefix': '', 'alerts': [], 'disktotal': '3898320355328', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707222622208', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719971209216', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707222622208', 'isoprefix': '', 'alerts': [], 'disktotal': '4364611616768', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=09577e94-971e-47ba-8078-baeb594803de (api:54) 2019-04-13 21:17:43,895+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.02 seconds (__init__:312) 2019-04-13 21:17:45,528+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:17:45,556+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:17:45,556+0200 ERROR (jsonrpc/0) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:17:45,557+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:17:50,351+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:17:50,383+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:17:50,384+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:17:50,385+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:17:53,941+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=66ff6f49-b7ba-42f1-a920-1255a933009b (api:48) 2019-04-13 21:17:53,946+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=66ff6f49-b7ba-42f1-a920-1255a933009b (api:54) 2019-04-13 21:17:53,947+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:17:53,960+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=e5a88698-101e-402f-8d9f-34c4b415374a (api:48) 2019-04-13 21:17:53,970+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707217379328', 'isoprefix': '', 'alerts': [], 'disktotal': '3898315112448', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707217379328', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719965966336', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707217379328', 'isoprefix': '', 'alerts': [], 'disktotal': '4364606373888', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=e5a88698-101e-402f-8d9f-34c4b415374a (api:54) 2019-04-13 21:17:53,972+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:17:56,948+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=71965096-9e58-480f-99a1-3eabf571a295 (api:48) 2019-04-13 21:17:56,948+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000705054', 'lastCheck': '2.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000646622', 'lastCheck': '2.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000658765', 'lastCheck': '2.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000354103', 'lastCheck': '0.0', 'valid': True}} from=internal, task_id=71965096-9e58-480f-99a1-3eabf571a295 (api:54) 2019-04-13 21:17:58,075+0200 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:17:58,144+0200 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=9c4b61f8-bf52-417b-adad-bac43849d979 (api:48) 2019-04-13 21:17:58,144+0200 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000705054', 'lastCheck': '3.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000646622', 'lastCheck': '3.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000658765', 'lastCheck': '3.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000354103', 'lastCheck': '1.2', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=9c4b61f8-bf52-417b-adad-bac43849d979 (api:54) 2019-04-13 21:17:58,146+0200 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=7b4a0b96-1900-4773-9936-52405223ce3b (api:48) 2019-04-13 21:17:58,146+0200 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=7b4a0b96-1900-4773-9936-52405223ce3b (api:54) 2019-04-13 21:17:58,159+0200 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '27.41', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '70.27'}, '21': {'cpuUser': '20.13', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '77.22'}, '22': {'cpuUser': '24.63', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '72.66'}, '23': {'cpuUser': '22.64', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '74.84'}, '1': {'cpuUser': '43.89', 'nodeIndex': 0, 'cpuSys': '3.84', 'cpuIdle': '52.27'}, '0': {'cpuUser': '39.26', 'nodeIndex': 0, 'cpuSys': '2.58', 'cpuIdle': '58.16'}, '3': {'cpuUser': '50.25', 'nodeIndex': 0, 'cpuSys': '1.52', 'cpuIdle': '48.23'}, '2': {'cpuUser': '54.55', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '43.73'}, '5': {'cpuUser': '49.92', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '48.49'}, '4': {'cpuUser': '58.00', 'nodeIndex': 0, 'cpuSys': '1.39', 'cpuIdle': '40.61'}, '7': {'cpuUser': '29.59', 'nodeIndex': 1, 'cpuSys': '4.17', 'cpuIdle': '66.24'}, '6': {'cpuUser': '34.82', 'nodeIndex': 1, 'cpuSys': '3.64', 'cpuIdle': '61.54'}, '9': {'cpuUser': '31.84', 'nodeIndex': 1, 'cpuSys': '3.64', 'cpuIdle': '64.52'}, '8': {'cpuUser': '34.89', 'nodeIndex': 1, 'cpuSys': '3.58', 'cpuIdle': '61.53'}, '11': {'cpuUser': '35.02', 'nodeIndex': 1, 'cpuSys': '3.77', 'cpuIdle': '61.21'}, '10': {'cpuUser': '32.57', 'nodeIndex': 1, 'cpuSys': '3.58', 'cpuIdle': '63.85'}, '13': {'cpuUser': '39.79', 'nodeIndex': 0, 'cpuSys': '1.52', 'cpuIdle': '58.69'}, '12': {'cpuUser': '39.66', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '58.49'}, '15': {'cpuUser': '31.18', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '66.83'}, '14': {'cpuUser': '18.67', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '79.34'}, '17': {'cpuUser': '25.82', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '72.33'}, '16': {'cpuUser': '24.83', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '73.45'}, '19': {'cpuUser': '19.73', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '77.56'}, '18': {'cpuUser': '21.85', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '75.44'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '71084'}, '0': {'memPercent': 23, 'memFree': '76646'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000705054', 'lastCheck': '3.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000658765', 'lastCheck': '3.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000354103', 'lastCheck': '1.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000646622', 'lastCheck': '3.9', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168836155', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '5091484720', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590430032580', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '2705552575869', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433745216838', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '258985846610', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239850647', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '366089972', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122781333', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121302830', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423591029812', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '253566090728', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215260629', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680236769', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '29545901533', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6651330650', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '29289214383', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255753987585', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '7014117598', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121677383', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75252560083', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '61753202032', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122236313', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418914277', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '1012771523', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121488760', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183078.142624, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183078.142624, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183078.142624, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183078.142624, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121258861', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '221638', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458724891841', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183078.142624, 'rx': '2576914163242', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1321950707351', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183078.142624, 'rx': '1321950707351', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183078.142624, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940825574', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '2700601695785', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571489207006', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '4950880084', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329384148275', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183078.142624, 'rx': '979871096467', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183078.142624, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183078.142624, 'rx': '45346284', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183078.142624, 'rx': '39826986', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1727826.24', 'cpuLoad': '8.30', 'cpuSys': '2.56', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.11', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150387, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:17:58 GMT', 'cpuUser': '33.78', 'memFree': 150131, 'cpuIdle': '63.66', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.26'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:17:58,165+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:18:00,608+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:18:00,642+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:18:00,642+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:18:00,643+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:18:04,019+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=8170178d-0397-4a38-9149-3c841a9e3108 (api:48) 2019-04-13 21:18:04,023+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=8170178d-0397-4a38-9149-3c841a9e3108 (api:54) 2019-04-13 21:18:04,024+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:18:04,034+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=fa877a6d-5bb7-4b78-b562-205efe037a6b (api:48) 2019-04-13 21:18:04,042+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707217379328', 'isoprefix': '', 'alerts': [], 'disktotal': '3898315112448', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707217379328', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719965966336', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707217379328', 'isoprefix': '', 'alerts': [], 'disktotal': '4364606373888', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=fa877a6d-5bb7-4b78-b562-205efe037a6b (api:54) 2019-04-13 21:18:04,043+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:18:04,108+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID=u'ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', options=None) from=internal, task_id=80008edb-cab7-4d49-ab13-91badff18d48 (api:48) 2019-04-13 21:18:04,109+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', options=None) from=internal, task_id=01f7d49b-335f-4111-96b3-f0053ebb6461 (api:48) 2019-04-13 21:18:04,110+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', options=None) from=internal, task_id=82d3e9cf-1b19-4ac0-ae3b-39028c33b811 (api:48) 2019-04-13 21:18:04,114+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '5697695744', 'apparentsize': '16106127360'} from=internal, task_id=82d3e9cf-1b19-4ac0-ae3b-39028c33b811 (api:54) 2019-04-13 21:18:04,115+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID='ef6741d1-380a-4736-a924-f88ed68e1270', options=None) from=internal, task_id=b927e920-94c4-478b-919f-4fbde896ff8f (api:48) 2019-04-13 21:18:04,115+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '4579690496', 'apparentsize': '16106127360'} from=internal, task_id=01f7d49b-335f-4111-96b3-f0053ebb6461 (api:54) 2019-04-13 21:18:04,116+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID=u'67a4dd71-69ff-49a8-b8a4-c7101c75e105', options=None) from=internal, task_id=14fc7ead-5a57-45e4-8800-0e61ce2e7acb (api:48) 2019-04-13 21:18:04,117+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '53760974848', 'apparentsize': '139586437120'} from=internal, task_id=80008edb-cab7-4d49-ab13-91badff18d48 (api:54) 2019-04-13 21:18:04,117+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', options=None) from=internal, task_id=7ba80eb9-d05f-4077-8be8-1669e46454a4 (api:48) 2019-04-13 21:18:04,121+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID=u'2b136ed7-8b9a-45c1-9edc-64ee96c87c43', options=None) from=internal, task_id=bc91185b-fe5a-4778-8bf6-59e38ab47885 (api:48) 2019-04-13 21:18:04,122+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '6477467136', 'apparentsize': '21474836480'} from=internal, task_id=b927e920-94c4-478b-919f-4fbde896ff8f (api:54) 2019-04-13 21:18:04,123+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID=u'44b3ceca-f0db-43da-af29-aa681df146ac', options=None) from=internal, task_id=7f8e29db-4116-44fc-84a6-0af907da62d0 (api:48) 2019-04-13 21:18:04,125+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '55059131904', 'apparentsize': '171798691840'} from=internal, task_id=14fc7ead-5a57-45e4-8800-0e61ce2e7acb (api:54) 2019-04-13 21:18:04,126+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=189dafa2-2534-498c-b25e-98a66e88df4e (api:48) 2019-04-13 21:18:04,126+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2524573696', 'apparentsize': '12884901888'} from=internal, task_id=7ba80eb9-d05f-4077-8be8-1669e46454a4 (api:54) 2019-04-13 21:18:04,127+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID='123e2cc8-8f24-4ddd-b401-68393535f555', options=None) from=internal, task_id=9024def9-0c5c-4ca7-adb4-fb1096358529 (api:48) 2019-04-13 21:18:04,130+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2024405504', 'apparentsize': '10737418240'} from=internal, task_id=bc91185b-fe5a-4778-8bf6-59e38ab47885 (api:54) 2019-04-13 21:18:04,130+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID='3f6c31c6-ec38-4979-a617-55537123ee61', options=None) from=internal, task_id=6bc1eea9-bbb7-4b02-bab7-aaf0ba339f4b (api:48) 2019-04-13 21:18:04,133+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '11196700160', 'apparentsize': '21474836480'} from=internal, task_id=7f8e29db-4116-44fc-84a6-0af907da62d0 (api:54) 2019-04-13 21:18:04,133+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', options=None) from=internal, task_id=6389360d-4f7b-4383-b8d7-10a184259467 (api:48) 2019-04-13 21:18:04,134+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2116492288', 'apparentsize': '3948150784'} from=internal, task_id=189dafa2-2534-498c-b25e-98a66e88df4e (api:54) 2019-04-13 21:18:04,134+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID=u'a1d583c1-e135-4e3f-b6fa-0f29a5261366', options=None) from=internal, task_id=aea76a9d-50b1-4da3-ad99-cde2c71bda48 (api:48) 2019-04-13 21:18:04,135+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2283003904', 'apparentsize': '16106127360'} from=internal, task_id=9024def9-0c5c-4ca7-adb4-fb1096358529 (api:54) 2019-04-13 21:18:04,135+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', options=None) from=internal, task_id=45fd39c4-f6f6-4125-89c5-97c47ede8428 (api:48) 2019-04-13 21:18:04,139+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '4998722048', 'apparentsize': '16106127360'} from=internal, task_id=6bc1eea9-bbb7-4b02-bab7-aaf0ba339f4b (api:54) 2019-04-13 21:18:04,139+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID='a884b4ce-381f-4653-be14-a31d15181f48', options=None) from=internal, task_id=4b06293b-4f59-4f85-aec2-f878d255456d (api:48) 2019-04-13 21:18:04,140+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '1294894592', 'apparentsize': '12884901888'} from=internal, task_id=aea76a9d-50b1-4da3-ad99-cde2c71bda48 (api:54) 2019-04-13 21:18:04,141+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', options=None) from=internal, task_id=fc3d48a9-2aa5-4e1e-83b7-34516a8e1b25 (api:48) 2019-04-13 21:18:04,143+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2109410304', 'apparentsize': '8589934592'} from=internal, task_id=6389360d-4f7b-4383-b8d7-10a184259467 (api:54) 2019-04-13 21:18:04,143+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '7263196672', 'apparentsize': '21474836480'} from=internal, task_id=45fd39c4-f6f6-4125-89c5-97c47ede8428 (api:54) 2019-04-13 21:18:04,145+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '3447297536', 'apparentsize': '16106127360'} from=internal, task_id=4b06293b-4f59-4f85-aec2-f878d255456d (api:54) 2019-04-13 21:18:04,146+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2879764480', 'apparentsize': '8589934592'} from=internal, task_id=fc3d48a9-2aa5-4e1e-83b7-34516a8e1b25 (api:54) 2019-04-13 21:18:05,395+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:18:05,427+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:18:05,427+0200 ERROR (jsonrpc/0) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:18:05,428+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:18:12,057+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=aca18d3b-27c5-4da9-a1cc-96f3d8c9ab72 (api:48) 2019-04-13 21:18:12,057+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000659024', 'lastCheck': '7.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000732392', 'lastCheck': '7.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000463109', 'lastCheck': '7.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000356492', 'lastCheck': '5.1', 'valid': True}} from=internal, task_id=aca18d3b-27c5-4da9-a1cc-96f3d8c9ab72 (api:54) 2019-04-13 21:18:13,119+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:18:13,210+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=c80a2d57-09c8-4203-8815-c7acf9742589 (api:48) 2019-04-13 21:18:13,210+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000659024', 'lastCheck': '8.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000732392', 'lastCheck': '8.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000463109', 'lastCheck': '8.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000356492', 'lastCheck': '6.3', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=c80a2d57-09c8-4203-8815-c7acf9742589 (api:54) 2019-04-13 21:18:13,211+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=52893f3b-c752-40b6-b253-8844f79fe2f1 (api:48) 2019-04-13 21:18:13,211+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=52893f3b-c752-40b6-b253-8844f79fe2f1 (api:54) 2019-04-13 21:18:13,225+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '18.13', 'nodeIndex': 1, 'cpuSys': '2.18', 'cpuIdle': '79.69'}, '21': {'cpuUser': '22.63', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '74.92'}, '22': {'cpuUser': '19.12', 'nodeIndex': 1, 'cpuSys': '2.38', 'cpuIdle': '78.50'}, '23': {'cpuUser': '14.16', 'nodeIndex': 1, 'cpuSys': '2.51', 'cpuIdle': '83.33'}, '1': {'cpuUser': '38.44', 'nodeIndex': 0, 'cpuSys': '4.17', 'cpuIdle': '57.39'}, '0': {'cpuUser': '33.28', 'nodeIndex': 0, 'cpuSys': '2.38', 'cpuIdle': '64.34'}, '3': {'cpuUser': '48.24', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '49.64'}, '2': {'cpuUser': '31.89', 'nodeIndex': 0, 'cpuSys': '2.51', 'cpuIdle': '65.60'}, '5': {'cpuUser': '35.73', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '62.22'}, '4': {'cpuUser': '34.94', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '63.01'}, '7': {'cpuUser': '33.02', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '64.13'}, '6': {'cpuUser': '33.55', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '63.67'}, '9': {'cpuUser': '33.22', 'nodeIndex': 1, 'cpuSys': '2.91', 'cpuIdle': '63.87'}, '8': {'cpuUser': '35.66', 'nodeIndex': 1, 'cpuSys': '3.04', 'cpuIdle': '61.30'}, '11': {'cpuUser': '31.17', 'nodeIndex': 1, 'cpuSys': '3.51', 'cpuIdle': '65.32'}, '10': {'cpuUser': '31.43', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '65.59'}, '13': {'cpuUser': '11.84', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '85.91'}, '12': {'cpuUser': '18.26', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '79.69'}, '15': {'cpuUser': '8.14', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '89.81'}, '14': {'cpuUser': '22.56', 'nodeIndex': 0, 'cpuSys': '1.39', 'cpuIdle': '76.05'}, '17': {'cpuUser': '33.88', 'nodeIndex': 0, 'cpuSys': '1.32', 'cpuIdle': '64.80'}, '16': {'cpuUser': '31.83', 'nodeIndex': 0, 'cpuSys': '1.39', 'cpuIdle': '66.78'}, '19': {'cpuUser': '22.76', 'nodeIndex': 1, 'cpuSys': '2.05', 'cpuIdle': '75.19'}, '18': {'cpuUser': '12.44', 'nodeIndex': 1, 'cpuSys': '2.51', 'cpuIdle': '85.05'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '71159'}, '0': {'memPercent': 23, 'memFree': '76570'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000659024', 'lastCheck': '8.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000463109', 'lastCheck': '8.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000356492', 'lastCheck': '6.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000732392', 'lastCheck': '8.9', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168836347', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '5091484720', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590431472108', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '2705553879787', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433748925118', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '258988044958', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239851175', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '366090370', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122781525', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121303022', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423594749228', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '253568471990', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215260821', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680237069', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '29545901630', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6651330857', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '29289214573', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255756151340', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '7014167888', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121677575', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75253798503', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '61753211026', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122236505', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418914925', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '1012771781', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121488952', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183093.20876, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183093.20876, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183093.20876, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183093.20876, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121259053', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '221638', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458726317435', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183093.20876, 'rx': '2576915405986', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1321962105462', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183093.20876, 'rx': '1321962105462', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183093.20876, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940825574', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '2700602999223', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571490646534', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '4950880564', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329386771770', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183093.20876, 'rx': '979880856075', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183093.20876, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183093.20876, 'rx': '45346674', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183093.20876, 'rx': '39827308', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1727841.28', 'cpuLoad': '8.22', 'cpuSys': '2.42', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.04', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150385, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:18:13 GMT', 'cpuUser': '27.35', 'memFree': 150129, 'cpuIdle': '70.23', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.12'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:18:13,231+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.11 seconds (__init__:312) 2019-04-13 21:18:14,090+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=05530034-874c-4b6d-ba19-5951f837a11f (api:48) 2019-04-13 21:18:14,095+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=05530034-874c-4b6d-ba19-5951f837a11f (api:54) 2019-04-13 21:18:14,095+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:18:14,107+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=435d9305-e9a5-4629-814d-42c9a555100d (api:48) 2019-04-13 21:18:14,115+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707217379328', 'isoprefix': '', 'alerts': [], 'disktotal': '3898315112448', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707217379328', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719965966336', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707217379328', 'isoprefix': '', 'alerts': [], 'disktotal': '4364606373888', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=435d9305-e9a5-4629-814d-42c9a555100d (api:54) 2019-04-13 21:18:14,116+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:18:15,686+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:18:15,727+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:18:15,728+0200 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:18:15,729+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:18:20,436+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:18:20,479+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:18:20,480+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:18:20,481+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:18:24,164+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=0479df83-0e7d-4de5-9e86-a2b76f656a7d (api:48) 2019-04-13 21:18:24,169+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=0479df83-0e7d-4de5-9e86-a2b76f656a7d (api:54) 2019-04-13 21:18:24,169+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:18:24,190+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=9ea7a5b4-1777-4ff9-8871-29444ea15dd0 (api:48) 2019-04-13 21:18:24,198+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707208990720', 'isoprefix': '', 'alerts': [], 'disktotal': '3898306723840', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707208990720', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719957577728', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707208990720', 'isoprefix': '', 'alerts': [], 'disktotal': '4364597985280', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=9ea7a5b4-1777-4ff9-8871-29444ea15dd0 (api:54) 2019-04-13 21:18:24,199+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:18:27,161+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=e1f7749d-422f-4bec-81a2-c9bf8a8aac0d (api:48) 2019-04-13 21:18:27,162+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000761508', 'lastCheck': '2.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000620749', 'lastCheck': '2.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000583577', 'lastCheck': '2.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000424466', 'lastCheck': '0.2', 'valid': True}} from=internal, task_id=e1f7749d-422f-4bec-81a2-c9bf8a8aac0d (api:54) 2019-04-13 21:18:28,179+0200 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:18:28,249+0200 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=71b1a307-3539-4a6d-9e8c-0be14af01dad (api:48) 2019-04-13 21:18:28,249+0200 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000761508', 'lastCheck': '3.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000620749', 'lastCheck': '4.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000583577', 'lastCheck': '3.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000424466', 'lastCheck': '1.3', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=71b1a307-3539-4a6d-9e8c-0be14af01dad (api:54) 2019-04-13 21:18:28,250+0200 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=6435a05b-2a4b-426f-8f46-388877e0db72 (api:48) 2019-04-13 21:18:28,250+0200 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=6435a05b-2a4b-426f-8f46-388877e0db72 (api:54) 2019-04-13 21:18:28,264+0200 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '21.26', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '75.76'}, '21': {'cpuUser': '20.33', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '77.55'}, '22': {'cpuUser': '16.16', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '81.26'}, '23': {'cpuUser': '20.26', 'nodeIndex': 1, 'cpuSys': '2.38', 'cpuIdle': '77.36'}, '1': {'cpuUser': '42.18', 'nodeIndex': 0, 'cpuSys': '4.44', 'cpuIdle': '53.38'}, '0': {'cpuUser': '38.41', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '59.87'}, '3': {'cpuUser': '47.94', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '50.27'}, '2': {'cpuUser': '55.43', 'nodeIndex': 0, 'cpuSys': '1.39', 'cpuIdle': '43.18'}, '5': {'cpuUser': '43.51', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '54.30'}, '4': {'cpuUser': '50.39', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '47.56'}, '7': {'cpuUser': '32.45', 'nodeIndex': 1, 'cpuSys': '3.64', 'cpuIdle': '63.91'}, '6': {'cpuUser': '30.86', 'nodeIndex': 1, 'cpuSys': '3.51', 'cpuIdle': '65.63'}, '9': {'cpuUser': '37.28', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '59.54'}, '8': {'cpuUser': '32.51', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '64.18'}, '11': {'cpuUser': '30.92', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '66.10'}, '10': {'cpuUser': '37.81', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '59.08'}, '13': {'cpuUser': '30.86', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '67.29'}, '12': {'cpuUser': '49.00', 'nodeIndex': 0, 'cpuSys': '1.32', 'cpuIdle': '49.68'}, '15': {'cpuUser': '35.03', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '62.98'}, '14': {'cpuUser': '30.26', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '67.82'}, '17': {'cpuUser': '37.55', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '60.66'}, '16': {'cpuUser': '24.43', 'nodeIndex': 0, 'cpuSys': '2.38', 'cpuIdle': '73.19'}, '19': {'cpuUser': '16.55', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '80.93'}, '18': {'cpuUser': '17.48', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '79.41'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '71142'}, '0': {'memPercent': 23, 'memFree': '76588'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000761508', 'lastCheck': '3.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000583577', 'lastCheck': '3.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000424466', 'lastCheck': '1.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000620749', 'lastCheck': '4.0', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168836969', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '5091485344', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590432240492', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '2705567783435', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433752754226', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '258990187258', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239851257', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '366090370', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122781607', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121303104', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423598470540', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '253570566937', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215260903', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680349235', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '29546845994', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652275221', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '29289326739', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255758297349', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '7015795916', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121677657', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75256799953', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '61762465857', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122236587', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418915405', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '1012771781', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121489034', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183108.247731, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183108.247731, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183108.247731, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183108.247731, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121259135', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '221638', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458727019387', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183108.247731, 'rx': '2576928706168', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1321974590293', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183108.247731, 'rx': '1321974590293', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183108.247731, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940826428', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '2700616902105', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571491414064', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '4950881330', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329400153978', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183108.247731, 'rx': '979891827413', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183108.247731, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183108.247731, 'rx': '45347064', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183108.247731, 'rx': '39827676', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1727856.34', 'cpuLoad': '8.16', 'cpuSys': '2.50', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.38', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150387, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:18:28 GMT', 'cpuUser': '33.29', 'memFree': 150131, 'cpuIdle': '64.21', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.32'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:18:28,270+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:18:30,752+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:18:30,774+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:18:30,774+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:18:30,775+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:18:34,245+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=8efd5da7-f85e-4ed2-b4db-e4bd59aa246d (api:48) 2019-04-13 21:18:34,256+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=8efd5da7-f85e-4ed2-b4db-e4bd59aa246d (api:54) 2019-04-13 21:18:34,258+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:18:34,392+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=7cc95770-22ac-488d-8df0-23fba9a31798 (api:48) 2019-04-13 21:18:34,399+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707207942144', 'isoprefix': '', 'alerts': [], 'disktotal': '3898305675264', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707207942144', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719956529152', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707207942144', 'isoprefix': '', 'alerts': [], 'disktotal': '4364596936704', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=7cc95770-22ac-488d-8df0-23fba9a31798 (api:54) 2019-04-13 21:18:34,400+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:18:35,490+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:18:35,521+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:18:35,522+0200 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:18:35,522+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:18:42,267+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=ba037b41-5158-4705-a574-33030a2fd969 (api:48) 2019-04-13 21:18:42,268+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000641247', 'lastCheck': '7.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000598248', 'lastCheck': '8.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000619871', 'lastCheck': '7.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000391313', 'lastCheck': '5.3', 'valid': True}} from=internal, task_id=ba037b41-5158-4705-a574-33030a2fd969 (api:54) 2019-04-13 21:18:43,225+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:18:43,306+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=f005c20e-39e0-4e18-90c3-b1d1da16b16d (api:48) 2019-04-13 21:18:43,307+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000641247', 'lastCheck': '9.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000598248', 'lastCheck': '9.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000619871', 'lastCheck': '9.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000391313', 'lastCheck': '6.4', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=f005c20e-39e0-4e18-90c3-b1d1da16b16d (api:54) 2019-04-13 21:18:43,307+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=2e9f7e1e-646a-4500-9e58-b61338666579 (api:48) 2019-04-13 21:18:43,308+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=2e9f7e1e-646a-4500-9e58-b61338666579 (api:54) 2019-04-13 21:18:43,320+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '22.09', 'nodeIndex': 1, 'cpuSys': '2.51', 'cpuIdle': '75.40'}, '21': {'cpuUser': '21.96', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '75.26'}, '22': {'cpuUser': '13.96', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '83.06'}, '23': {'cpuUser': '31.95', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '65.93'}, '1': {'cpuUser': '47.62', 'nodeIndex': 0, 'cpuSys': '3.90', 'cpuIdle': '48.48'}, '0': {'cpuUser': '53.04', 'nodeIndex': 0, 'cpuSys': '1.98', 'cpuIdle': '44.98'}, '3': {'cpuUser': '49.47', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '48.48'}, '2': {'cpuUser': '43.06', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '55.09'}, '5': {'cpuUser': '56.95', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '41.33'}, '4': {'cpuUser': '44.51', 'nodeIndex': 0, 'cpuSys': '2.18', 'cpuIdle': '53.31'}, '7': {'cpuUser': '39.49', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '57.40'}, '6': {'cpuUser': '38.16', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '59.13'}, '9': {'cpuUser': '43.78', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '53.57'}, '8': {'cpuUser': '37.44', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '59.25'}, '11': {'cpuUser': '37.04', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '60.18'}, '10': {'cpuUser': '42.86', 'nodeIndex': 1, 'cpuSys': '2.84', 'cpuIdle': '54.30'}, '13': {'cpuUser': '25.86', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '71.89'}, '12': {'cpuUser': '17.33', 'nodeIndex': 0, 'cpuSys': '2.31', 'cpuIdle': '80.36'}, '15': {'cpuUser': '24.87', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '73.01'}, '14': {'cpuUser': '39.88', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '58.20'}, '17': {'cpuUser': '27.71', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '70.44'}, '16': {'cpuUser': '32.67', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '65.74'}, '19': {'cpuUser': '16.60', 'nodeIndex': 1, 'cpuSys': '3.04', 'cpuIdle': '80.36'}, '18': {'cpuUser': '25.79', 'nodeIndex': 1, 'cpuSys': '1.79', 'cpuIdle': '72.42'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '71163'}, '0': {'memPercent': 23, 'memFree': '76565'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000641247', 'lastCheck': '9.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000619871', 'lastCheck': '9.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000391313', 'lastCheck': '6.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000598248', 'lastCheck': '9.0', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168837029', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '5091485386', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590433387628', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '2705569686207', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433756524016', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '258992835623', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239851257', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '366090370', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122781607', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121303104', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423602317474', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '253572901930', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215260903', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680351226', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '29546850539', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652279766', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '29289328730', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255761157463', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '7026837426', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121677657', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75259685237', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '61767223257', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122236587', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418915825', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '1012771781', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121489034', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183123.305448, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183123.305448, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183123.305448, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183123.305448, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121259135', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '221638', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458728166877', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183123.305448, 'rx': '2576930524508', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1321983905736', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183123.305448, 'rx': '1321983905736', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183123.305448, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940826428', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '2700618804685', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571492561200', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '4950881522', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329418756320', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183123.305448, 'rx': '979902969678', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183123.305448, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183123.305448, 'rx': '45347454', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183123.305448, 'rx': '39827998', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40566', 'ksmPages': 100, 'elapsedTime': '1727871.39', 'cpuLoad': '8.20', 'cpuSys': '2.44', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '2.98', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150385, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:18:43 GMT', 'cpuUser': '34.75', 'memFree': 150129, 'cpuIdle': '62.81', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.19'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:18:43,326+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.10 seconds (__init__:312) 2019-04-13 21:18:44,451+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=de2483f2-af26-4ff3-8701-cbe7a2d133ab (api:48) 2019-04-13 21:18:44,455+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=de2483f2-af26-4ff3-8701-cbe7a2d133ab (api:54) 2019-04-13 21:18:44,456+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:18:44,469+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=53217b33-1d0b-4aa2-8168-292c8c82d8ce (api:48) 2019-04-13 21:18:44,478+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707207942144', 'isoprefix': '', 'alerts': [], 'disktotal': '3898305675264', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707207942144', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719956529152', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707207942144', 'isoprefix': '', 'alerts': [], 'disktotal': '4364596936704', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=53217b33-1d0b-4aa2-8168-292c8c82d8ce (api:54) 2019-04-13 21:18:44,479+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:18:45,799+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:18:45,826+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:18:45,826+0200 ERROR (jsonrpc/0) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:18:45,827+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:18:50,529+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:18:50,560+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:18:50,560+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:18:50,561+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:18:54,522+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=c2bf7cbf-7e71-410f-94f1-f882d2b7c0d7 (api:48) 2019-04-13 21:18:54,527+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=c2bf7cbf-7e71-410f-94f1-f882d2b7c0d7 (api:54) 2019-04-13 21:18:54,527+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:18:54,539+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=c2675d38-9f0d-4299-a75a-d69c3f5723ce (api:48) 2019-04-13 21:18:54,548+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707203747840', 'isoprefix': '', 'alerts': [], 'disktotal': '3898301480960', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707203747840', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719952334848', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707203747840', 'isoprefix': '', 'alerts': [], 'disktotal': '4364591693824', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=c2675d38-9f0d-4299-a75a-d69c3f5723ce (api:54) 2019-04-13 21:18:54,550+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:18:56,936+0200 INFO (itmap/0) [IOProcessClient] (/onode-1.discovery.intern:_mnt_store) Starting client (__init__:308) 2019-04-13 21:18:56,953+0200 INFO (itmap/1) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_vms) Starting client (__init__:308) 2019-04-13 21:18:56,959+0200 INFO (ioprocess/27714) [IOProcess] (/onode-1.discovery.intern:_mnt_store) Starting ioprocess (__init__:434) 2019-04-13 21:18:56,973+0200 INFO (itmap/2) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_isos) Starting client (__init__:308) 2019-04-13 21:18:56,978+0200 INFO (ioprocess/27722) [IOProcess] (/nas.discovery.intern:_mnt_d2600_vms) Starting ioprocess (__init__:434) 2019-04-13 21:18:56,990+0200 INFO (itmap/3) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting client (__init__:308) 2019-04-13 21:18:56,995+0200 INFO (ioprocess/27730) [IOProcess] (/nas.discovery.intern:_mnt_d2600_isos) Starting ioprocess (__init__:434) 2019-04-13 21:18:57,009+0200 INFO (itmap/4) [IOProcessClient] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting client (__init__:308) 2019-04-13 21:18:57,014+0200 INFO (ioprocess/27737) [IOProcess] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting ioprocess (__init__:434) 2019-04-13 21:18:57,027+0200 INFO (ioprocess/27744) [IOProcess] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting ioprocess (__init__:434) 2019-04-13 21:18:57,043+0200 INFO (monitor/6a4cde7) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:18:57,338+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=132a8613-648b-4220-a3e0-16bce49ebb5d (api:48) 2019-04-13 21:18:57,338+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000648963', 'lastCheck': '3.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000368754', 'lastCheck': '3.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00051778', 'lastCheck': '3.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000362598', 'lastCheck': '0.3', 'valid': True}} from=internal, task_id=132a8613-648b-4220-a3e0-16bce49ebb5d (api:54) 2019-04-13 21:18:58,274+0200 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:18:58,335+0200 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=ea6dc6f5-e97d-44e2-ac9b-d8594bd7a29a (api:48) 2019-04-13 21:18:58,336+0200 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000648963', 'lastCheck': '4.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000368754', 'lastCheck': '4.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00051778', 'lastCheck': '4.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000362598', 'lastCheck': '1.3', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=ea6dc6f5-e97d-44e2-ac9b-d8594bd7a29a (api:54) 2019-04-13 21:18:58,336+0200 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=7b028fef-a715-40c3-a839-bd421ead9c93 (api:48) 2019-04-13 21:18:58,337+0200 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=7b028fef-a715-40c3-a839-bd421ead9c93 (api:54) 2019-04-13 21:18:58,350+0200 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '18.29', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '78.86'}, '21': {'cpuUser': '19.75', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '77.80'}, '22': {'cpuUser': '20.41', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '77.01'}, '23': {'cpuUser': '22.86', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '74.62'}, '1': {'cpuUser': '46.78', 'nodeIndex': 0, 'cpuSys': '4.57', 'cpuIdle': '48.65'}, '0': {'cpuUser': '51.75', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '46.53'}, '3': {'cpuUser': '49.56', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '48.52'}, '2': {'cpuUser': '53.08', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '45.33'}, '5': {'cpuUser': '50.36', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '47.72'}, '4': {'cpuUser': '41.61', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '56.14'}, '7': {'cpuUser': '38.17', 'nodeIndex': 1, 'cpuSys': '3.45', 'cpuIdle': '58.38'}, '6': {'cpuUser': '40.22', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '56.93'}, '9': {'cpuUser': '41.41', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '55.74'}, '8': {'cpuUser': '46.45', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '50.97'}, '11': {'cpuUser': '39.82', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '57.60'}, '10': {'cpuUser': '38.43', 'nodeIndex': 1, 'cpuSys': '2.92', 'cpuIdle': '58.65'}, '13': {'cpuUser': '21.93', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '76.21'}, '12': {'cpuUser': '23.46', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '74.29'}, '15': {'cpuUser': '32.73', 'nodeIndex': 0, 'cpuSys': '1.46', 'cpuIdle': '65.81'}, '14': {'cpuUser': '16.63', 'nodeIndex': 0, 'cpuSys': '2.52', 'cpuIdle': '80.85'}, '17': {'cpuUser': '30.02', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '67.99'}, '16': {'cpuUser': '35.05', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '62.96'}, '19': {'cpuUser': '18.75', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '78.07'}, '18': {'cpuUser': '20.54', 'nodeIndex': 1, 'cpuSys': '1.92', 'cpuIdle': '77.54'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '71076'}, '0': {'memPercent': 23, 'memFree': '76642'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000648963', 'lastCheck': '4.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00051778', 'lastCheck': '4.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000362598', 'lastCheck': '1.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000368754', 'lastCheck': '4.1', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168842542', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '5091485658', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590434517182', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '2705571550684', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433760231244', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '258995341303', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239857042', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '366090370', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122787392', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121308889', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423605981780', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '253575579820', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215266688', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680357263', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '29546850845', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652285693', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '29289329146', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255763685179', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '7029020666', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121683442', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75261517543', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '61771445319', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122242372', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418916305', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '1012771781', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121494819', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183138.334089, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183138.334089, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183138.334089, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183138.334089, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121264920', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '221638', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458729265403', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183138.334089, 'rx': '2576932305543', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1321997424709', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183138.334089, 'rx': '1321997424709', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183138.334089, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940826642', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '2700620668029', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571493690540', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '4950882655', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329428272703', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183138.334089, 'rx': '979912563149', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183138.334089, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183138.334089, 'rx': '45347844', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183138.334089, 'rx': '39828366', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1727886.44', 'cpuLoad': '8.24', 'cpuSys': '2.45', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '2.98', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150375, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:18:58 GMT', 'cpuUser': '34.09', 'memFree': 150119, 'cpuIdle': '63.45', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.59'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:18:58,356+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:19:00,878+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:19:00,907+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:19:00,908+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:19:00,909+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:19:04,111+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID=u'ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', options=None) from=internal, task_id=1b5d182c-ad3e-445a-9300-aeb0096b0b5c (api:48) 2019-04-13 21:19:04,113+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', options=None) from=internal, task_id=f90667e4-73e5-420a-ae3d-ef2b17304e10 (api:48) 2019-04-13 21:19:04,114+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', options=None) from=internal, task_id=b74dd9c6-fb6f-4a7f-8d42-4837fb437ac7 (api:48) 2019-04-13 21:19:04,121+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '5697707008', 'apparentsize': '16106127360'} from=internal, task_id=b74dd9c6-fb6f-4a7f-8d42-4837fb437ac7 (api:54) 2019-04-13 21:19:04,122+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID='ef6741d1-380a-4736-a924-f88ed68e1270', options=None) from=internal, task_id=421fee22-8b69-4854-b208-41c26d6e3e45 (api:48) 2019-04-13 21:19:04,123+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '4579690496', 'apparentsize': '16106127360'} from=internal, task_id=f90667e4-73e5-420a-ae3d-ef2b17304e10 (api:54) 2019-04-13 21:19:04,124+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID=u'67a4dd71-69ff-49a8-b8a4-c7101c75e105', options=None) from=internal, task_id=099819f4-a59a-4222-b8ee-fc7aa5776c55 (api:48) 2019-04-13 21:19:04,126+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '53760952832', 'apparentsize': '139586437120'} from=internal, task_id=1b5d182c-ad3e-445a-9300-aeb0096b0b5c (api:54) 2019-04-13 21:19:04,126+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', options=None) from=internal, task_id=98620161-f82d-46fc-8896-d4ce45bb74ca (api:48) 2019-04-13 21:19:04,131+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID=u'2b136ed7-8b9a-45c1-9edc-64ee96c87c43', options=None) from=internal, task_id=3ec7807e-f294-4f7c-805f-15da5a187066 (api:48) 2019-04-13 21:19:04,135+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2524573696', 'apparentsize': '12884901888'} from=internal, task_id=98620161-f82d-46fc-8896-d4ce45bb74ca (api:54) 2019-04-13 21:19:04,136+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID=u'44b3ceca-f0db-43da-af29-aa681df146ac', options=None) from=internal, task_id=f487ad05-cd57-466d-912a-b3c61ec6fb88 (api:48) 2019-04-13 21:19:04,137+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '55059126272', 'apparentsize': '171798691840'} from=internal, task_id=099819f4-a59a-4222-b8ee-fc7aa5776c55 (api:54) 2019-04-13 21:19:04,138+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=1b88ca17-1266-4cf9-ba88-791dbee31a0c (api:48) 2019-04-13 21:19:04,140+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '6477467136', 'apparentsize': '21474836480'} from=internal, task_id=421fee22-8b69-4854-b208-41c26d6e3e45 (api:54) 2019-04-13 21:19:04,140+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID='123e2cc8-8f24-4ddd-b401-68393535f555', options=None) from=internal, task_id=6eb0bc06-8b8d-4276-9ae8-e4648c397a80 (api:48) 2019-04-13 21:19:04,144+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2024405504', 'apparentsize': '10737418240'} from=internal, task_id=3ec7807e-f294-4f7c-805f-15da5a187066 (api:54) 2019-04-13 21:19:04,145+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID='3f6c31c6-ec38-4979-a617-55537123ee61', options=None) from=internal, task_id=bab6f3d4-fc35-42b8-9ea4-aeb5a3736659 (api:48) 2019-04-13 21:19:04,149+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2116492288', 'apparentsize': '3948150784'} from=internal, task_id=1b88ca17-1266-4cf9-ba88-791dbee31a0c (api:54) 2019-04-13 21:19:04,150+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', options=None) from=internal, task_id=bc0bb649-abae-4960-8a0d-955e67fcdadf (api:48) 2019-04-13 21:19:04,151+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '11196700160', 'apparentsize': '21474836480'} from=internal, task_id=f487ad05-cd57-466d-912a-b3c61ec6fb88 (api:54) 2019-04-13 21:19:04,152+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID=u'a1d583c1-e135-4e3f-b6fa-0f29a5261366', options=None) from=internal, task_id=f5af6ec0-3925-47aa-8982-7aab9ccc441e (api:48) 2019-04-13 21:19:04,157+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2283003904', 'apparentsize': '16106127360'} from=internal, task_id=6eb0bc06-8b8d-4276-9ae8-e4648c397a80 (api:54) 2019-04-13 21:19:04,158+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', options=None) from=internal, task_id=06197170-e5a3-44a7-84b5-4f5fe5056233 (api:48) 2019-04-13 21:19:04,158+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '4998722048', 'apparentsize': '16106127360'} from=internal, task_id=bab6f3d4-fc35-42b8-9ea4-aeb5a3736659 (api:54) 2019-04-13 21:19:04,159+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID='a884b4ce-381f-4653-be14-a31d15181f48', options=None) from=internal, task_id=b70745f2-7b01-434a-a341-7aeff73b1816 (api:48) 2019-04-13 21:19:04,174+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '1294894592', 'apparentsize': '12884901888'} from=internal, task_id=f5af6ec0-3925-47aa-8982-7aab9ccc441e (api:54) 2019-04-13 21:19:04,178+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', options=None) from=internal, task_id=166684de-9741-432b-b7ad-52f19bd7a015 (api:48) 2019-04-13 21:19:04,179+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2109410304', 'apparentsize': '8589934592'} from=internal, task_id=bc0bb649-abae-4960-8a0d-955e67fcdadf (api:54) 2019-04-13 21:19:04,182+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '3447297536', 'apparentsize': '16106127360'} from=internal, task_id=b70745f2-7b01-434a-a341-7aeff73b1816 (api:54) 2019-04-13 21:19:04,183+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '7263196672', 'apparentsize': '21474836480'} from=internal, task_id=06197170-e5a3-44a7-84b5-4f5fe5056233 (api:54) 2019-04-13 21:19:04,186+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2879753728', 'apparentsize': '8589934592'} from=internal, task_id=166684de-9741-432b-b7ad-52f19bd7a015 (api:54) 2019-04-13 21:19:04,591+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=5a3f8f73-a783-4667-a374-376be752ab9b (api:48) 2019-04-13 21:19:04,595+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=5a3f8f73-a783-4667-a374-376be752ab9b (api:54) 2019-04-13 21:19:04,596+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:19:04,609+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=72f02aa7-d7ea-4997-b274-fe0c5eba6d54 (api:48) 2019-04-13 21:19:04,617+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707199553536', 'isoprefix': '', 'alerts': [], 'disktotal': '3898297286656', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707199553536', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719948140544', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707199553536', 'isoprefix': '', 'alerts': [], 'disktotal': '4364587499520', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=72f02aa7-d7ea-4997-b274-fe0c5eba6d54 (api:54) 2019-04-13 21:19:04,618+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:19:05,567+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:19:05,598+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:19:05,598+0200 ERROR (jsonrpc/0) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:19:05,599+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:19:12,420+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=75f7b763-985b-45b5-9fac-7112f760e0ed (api:48) 2019-04-13 21:19:12,421+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000460985', 'lastCheck': '8.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000753895', 'lastCheck': '8.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000675159', 'lastCheck': '8.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000415758', 'lastCheck': '5.4', 'valid': True}} from=internal, task_id=75f7b763-985b-45b5-9fac-7112f760e0ed (api:54) 2019-04-13 21:19:13,322+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:19:13,414+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=4ca20810-cc0a-4c76-8e2c-286a21d9776b (api:48) 2019-04-13 21:19:13,415+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000460985', 'lastCheck': '0.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000753895', 'lastCheck': '0.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000675159', 'lastCheck': '0.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000415758', 'lastCheck': '6.4', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=4ca20810-cc0a-4c76-8e2c-286a21d9776b (api:54) 2019-04-13 21:19:13,415+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=04901e96-c241-4709-aaf3-3207079c67b4 (api:48) 2019-04-13 21:19:13,416+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=04901e96-c241-4709-aaf3-3207079c67b4 (api:54) 2019-04-13 21:19:13,428+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '27.87', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '69.81'}, '21': {'cpuUser': '22.69', 'nodeIndex': 1, 'cpuSys': '2.59', 'cpuIdle': '74.72'}, '22': {'cpuUser': '17.91', 'nodeIndex': 1, 'cpuSys': '2.79', 'cpuIdle': '79.30'}, '23': {'cpuUser': '15.99', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '81.36'}, '1': {'cpuUser': '51.68', 'nodeIndex': 0, 'cpuSys': '4.05', 'cpuIdle': '44.27'}, '0': {'cpuUser': '56.86', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '41.28'}, '3': {'cpuUser': '54.20', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '43.81'}, '2': {'cpuUser': '50.49', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '47.59'}, '5': {'cpuUser': '51.55', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '46.66'}, '4': {'cpuUser': '47.04', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '51.17'}, '7': {'cpuUser': '37.15', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '60.13'}, '6': {'cpuUser': '39.41', 'nodeIndex': 1, 'cpuSys': '2.92', 'cpuIdle': '57.67'}, '9': {'cpuUser': '39.08', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '58.27'}, '8': {'cpuUser': '36.69', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '60.46'}, '11': {'cpuUser': '39.61', 'nodeIndex': 1, 'cpuSys': '2.92', 'cpuIdle': '57.47'}, '10': {'cpuUser': '39.21', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '58.07'}, '13': {'cpuUser': '23.09', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '74.72'}, '12': {'cpuUser': '23.69', 'nodeIndex': 0, 'cpuSys': '2.39', 'cpuIdle': '73.92'}, '15': {'cpuUser': '25.48', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '72.07'}, '14': {'cpuUser': '33.37', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '64.77'}, '17': {'cpuUser': '34.04', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '64.17'}, '16': {'cpuUser': '34.96', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '63.25'}, '19': {'cpuUser': '28.93', 'nodeIndex': 1, 'cpuSys': '2.26', 'cpuIdle': '68.81'}, '18': {'cpuUser': '17.78', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '79.50'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '70870'}, '0': {'memPercent': 22, 'memFree': '76848'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000460985', 'lastCheck': '0.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000675159', 'lastCheck': '0.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000415758', 'lastCheck': '6.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000753895', 'lastCheck': '0.1', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168844249', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '5091485658', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590435007638', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '2705572679137', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433764311122', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '258997998571', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239859409', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '366091070', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122789099', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121310596', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423609949784', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '253578091911', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215268395', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680358970', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '29546850845', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652287400', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '29289329146', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255766386512', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '7032919790', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121685149', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75264202217', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '61773837219', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122244079', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418916953', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '1012772039', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121496526', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183153.41348, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183153.41348, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183153.41348, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183153.41348, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121266557', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458729734679', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183153.41348, 'rx': '2576933384942', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322007793140', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183153.41348, 'rx': '1322007793140', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183153.41348, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940826716', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '2700621795678', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571494180922', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '4950883459', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329437092923', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183153.41348, 'rx': '979923643501', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183153.41348, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183153.41348, 'rx': '45348234', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183153.41348, 'rx': '39828688', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40572', 'ksmPages': 100, 'elapsedTime': '1727901.48', 'cpuLoad': '8.24', 'cpuSys': '2.41', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.65', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150378, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:19:13 GMT', 'cpuUser': '35.36', 'memFree': 150122, 'cpuIdle': '62.23', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.19'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:19:13,434+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.12 seconds (__init__:312) 2019-04-13 21:19:14,657+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=32ca9b7e-282d-4a25-8806-6e00d2efa5b8 (api:48) 2019-04-13 21:19:14,662+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=32ca9b7e-282d-4a25-8806-6e00d2efa5b8 (api:54) 2019-04-13 21:19:14,662+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:19:14,765+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=28b94dc5-acf2-4d61-9838-7a6d34a149cc (api:48) 2019-04-13 21:19:14,773+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707197456384', 'isoprefix': '', 'alerts': [], 'disktotal': '3898294140928', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707197456384', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719946043392', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707197456384', 'isoprefix': '', 'alerts': [], 'disktotal': '4364585402368', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=28b94dc5-acf2-4d61-9838-7a6d34a149cc (api:54) 2019-04-13 21:19:14,774+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:19:15,960+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:19:16,002+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:19:16,003+0200 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:19:16,004+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.05 seconds (__init__:312) 2019-04-13 21:19:20,608+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:19:20,639+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:19:20,639+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:19:20,640+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:19:24,828+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=7a2f66a3-9718-44f2-87f4-1d7a7342e427 (api:48) 2019-04-13 21:19:24,833+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=7a2f66a3-9718-44f2-87f4-1d7a7342e427 (api:54) 2019-04-13 21:19:24,833+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:19:24,849+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=69c7691b-2d08-4814-a245-c95c744669ac (api:48) 2019-04-13 21:19:24,859+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707195359232', 'isoprefix': '', 'alerts': [], 'disktotal': '3898293092352', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707195359232', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719943946240', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707195359232', 'isoprefix': '', 'alerts': [], 'disktotal': '4364583305216', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=69c7691b-2d08-4814-a245-c95c744669ac (api:54) 2019-04-13 21:19:24,861+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:19:27,516+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=d798f1cc-b04d-4344-b6cb-262287131042 (api:48) 2019-04-13 21:19:27,517+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000759961', 'lastCheck': '3.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000736851', 'lastCheck': '3.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000715058', 'lastCheck': '3.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000389496', 'lastCheck': '0.5', 'valid': True}} from=internal, task_id=d798f1cc-b04d-4344-b6cb-262287131042 (api:54) 2019-04-13 21:19:28,393+0200 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:19:28,460+0200 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=4e34eb3a-9d52-43d0-99d9-80ff8b211612 (api:48) 2019-04-13 21:19:28,461+0200 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000759961', 'lastCheck': '4.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000736851', 'lastCheck': '4.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000715058', 'lastCheck': '4.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000389496', 'lastCheck': '1.4', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=4e34eb3a-9d52-43d0-99d9-80ff8b211612 (api:54) 2019-04-13 21:19:28,462+0200 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=064a13f7-dede-4dcb-bb95-72ca6c287f86 (api:48) 2019-04-13 21:19:28,462+0200 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=064a13f7-dede-4dcb-bb95-72ca6c287f86 (api:54) 2019-04-13 21:19:28,474+0200 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '30.08', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '67.80'}, '21': {'cpuUser': '23.58', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '73.90'}, '22': {'cpuUser': '21.93', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '75.22'}, '23': {'cpuUser': '27.29', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '70.39'}, '1': {'cpuUser': '51.67', 'nodeIndex': 0, 'cpuSys': '4.17', 'cpuIdle': '44.16'}, '0': {'cpuUser': '54.19', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '44.09'}, '3': {'cpuUser': '57.04', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '41.24'}, '2': {'cpuUser': '58.83', 'nodeIndex': 0, 'cpuSys': '1.46', 'cpuIdle': '39.71'}, '5': {'cpuUser': '61.81', 'nodeIndex': 0, 'cpuSys': '1.26', 'cpuIdle': '36.93'}, '4': {'cpuUser': '55.05', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '43.36'}, '7': {'cpuUser': '34.12', 'nodeIndex': 1, 'cpuSys': '3.91', 'cpuIdle': '61.97'}, '6': {'cpuUser': '33.45', 'nodeIndex': 1, 'cpuSys': '3.38', 'cpuIdle': '63.17'}, '9': {'cpuUser': '32.13', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '64.56'}, '8': {'cpuUser': '32.39', 'nodeIndex': 1, 'cpuSys': '3.44', 'cpuIdle': '64.17'}, '11': {'cpuUser': '29.61', 'nodeIndex': 1, 'cpuSys': '3.64', 'cpuIdle': '66.75'}, '10': {'cpuUser': '35.31', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '61.38'}, '13': {'cpuUser': '35.44', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '62.64'}, '12': {'cpuUser': '32.66', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '65.68'}, '15': {'cpuUser': '23.45', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '74.36'}, '14': {'cpuUser': '27.43', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '70.25'}, '17': {'cpuUser': '33.59', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '64.69'}, '16': {'cpuUser': '42.20', 'nodeIndex': 0, 'cpuSys': '1.52', 'cpuIdle': '56.28'}, '19': {'cpuUser': '22.99', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '74.36'}, '18': {'cpuUser': '31.33', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '66.35'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '70895'}, '0': {'memPercent': 22, 'memFree': '76827'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000759961', 'lastCheck': '4.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000715058', 'lastCheck': '4.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000389496', 'lastCheck': '1.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000736851', 'lastCheck': '4.2', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168844818', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '5091485976', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590435276017', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '2705572703061', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433768264498', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '259000607811', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239859762', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '366091070', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122789452', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121310949', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423613864114', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '253580858778', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215268748', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680359323', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '29546850845', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652287510', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '29289329389', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255769022118', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '7035118383', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121685502', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75266468285', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '61778064159', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122244432', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418918177', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '1012772775', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121496879', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183168.459482, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183168.459482, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183168.459482, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183168.459482, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121266910', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458729965139', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183168.459482, 'rx': '2576933404405', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322020273863', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183168.459482, 'rx': '1322020273863', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183168.459482, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940827421', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '2700621819046', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571494448596', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '4950884015', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329446794436', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183168.459482, 'rx': '979934271014', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183168.459482, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183168.459482, 'rx': '45348624', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183168.459482, 'rx': '39829056', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40570', 'ksmPages': 100, 'elapsedTime': '1727916.55', 'cpuLoad': '8.31', 'cpuSys': '2.46', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.38', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150379, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:19:28 GMT', 'cpuUser': '36.97', 'memFree': 150123, 'cpuIdle': '60.57', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.39'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:19:28,480+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:19:31,026+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:19:31,058+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:19:31,058+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:19:31,059+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:19:34,911+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=50271a6f-37af-4d64-a765-cbfb55e4decb (api:48) 2019-04-13 21:19:34,916+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=50271a6f-37af-4d64-a765-cbfb55e4decb (api:54) 2019-04-13 21:19:34,917+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:19:34,930+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=ae3b9008-e8aa-4c02-8359-dc1e0d4e784c (api:48) 2019-04-13 21:19:34,940+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707193262080', 'isoprefix': '', 'alerts': [], 'disktotal': '3898289946624', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707193262080', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719941849088', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707193262080', 'isoprefix': '', 'alerts': [], 'disktotal': '4364581208064', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=ae3b9008-e8aa-4c02-8359-dc1e0d4e784c (api:54) 2019-04-13 21:19:34,942+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:19:35,646+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:19:35,676+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:19:35,677+0200 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:19:35,677+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:19:42,615+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=c16ce251-623c-4f87-8de0-4a1678f61553 (api:48) 2019-04-13 21:19:42,615+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000754914', 'lastCheck': '8.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000667376', 'lastCheck': '8.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000643339', 'lastCheck': '8.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000354813', 'lastCheck': '5.5', 'valid': True}} from=internal, task_id=c16ce251-623c-4f87-8de0-4a1678f61553 (api:54) 2019-04-13 21:19:43,436+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:19:43,498+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=c709bc85-0496-47b8-9948-57ffb923a5e2 (api:48) 2019-04-13 21:19:43,499+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000754914', 'lastCheck': '0.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000667376', 'lastCheck': '0.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000643339', 'lastCheck': '0.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000354813', 'lastCheck': '6.4', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=c709bc85-0496-47b8-9948-57ffb923a5e2 (api:54) 2019-04-13 21:19:43,500+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=64842b9c-ab84-49ec-aaad-6e81955cede1 (api:48) 2019-04-13 21:19:43,500+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=64842b9c-ab84-49ec-aaad-6e81955cede1 (api:54) 2019-04-13 21:19:43,513+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '26.65', 'nodeIndex': 1, 'cpuSys': '1.99', 'cpuIdle': '71.36'}, '21': {'cpuUser': '26.85', 'nodeIndex': 1, 'cpuSys': '2.39', 'cpuIdle': '70.76'}, '22': {'cpuUser': '21.74', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '75.81'}, '23': {'cpuUser': '25.06', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '72.75'}, '1': {'cpuUser': '42.56', 'nodeIndex': 0, 'cpuSys': '4.18', 'cpuIdle': '53.26'}, '0': {'cpuUser': '39.44', 'nodeIndex': 0, 'cpuSys': '2.78', 'cpuIdle': '57.78'}, '3': {'cpuUser': '44.08', 'nodeIndex': 0, 'cpuSys': '2.72', 'cpuIdle': '53.20'}, '2': {'cpuUser': '41.70', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '55.85'}, '5': {'cpuUser': '45.81', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '51.94'}, '4': {'cpuUser': '42.82', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '54.73'}, '7': {'cpuUser': '44.21', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '53.34'}, '6': {'cpuUser': '42.03', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '55.25'}, '9': {'cpuUser': '42.09', 'nodeIndex': 1, 'cpuSys': '2.59', 'cpuIdle': '55.32'}, '8': {'cpuUser': '44.68', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '53.00'}, '11': {'cpuUser': '41.76', 'nodeIndex': 1, 'cpuSys': '2.59', 'cpuIdle': '55.65'}, '10': {'cpuUser': '42.36', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '54.99'}, '13': {'cpuUser': '22.14', 'nodeIndex': 0, 'cpuSys': '2.59', 'cpuIdle': '75.27'}, '12': {'cpuUser': '24.26', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '73.88'}, '15': {'cpuUser': '21.01', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '76.74'}, '14': {'cpuUser': '21.01', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '76.67'}, '17': {'cpuUser': '20.28', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '77.60'}, '16': {'cpuUser': '24.26', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '73.42'}, '19': {'cpuUser': '27.05', 'nodeIndex': 1, 'cpuSys': '2.05', 'cpuIdle': '70.90'}, '18': {'cpuUser': '20.28', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '77.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '71372'}, '0': {'memPercent': 23, 'memFree': '76351'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000754914', 'lastCheck': '0.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000643339', 'lastCheck': '0.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000354813', 'lastCheck': '6.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000667376', 'lastCheck': '0.1', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168845226', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '5091486060', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590435737851', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '2705573823683', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433772527802', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '259003481021', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239860050', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '366091070', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122789740', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121311237', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423618147208', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '253583490852', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215269036', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680359611', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '29546850845', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652287634', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '29289329553', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255771904887', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '7037157308', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121685790', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75268289869', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '61780971360', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122244720', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418918597', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '1012772775', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121497167', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183183.497619, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183183.497619, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183183.497619, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183183.497619, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121267198', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458730461633', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183183.497619, 'rx': '2576934477163', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322030617582', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183183.497619, 'rx': '1322030617582', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183183.497619, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940827507', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '2700622939284', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571494910344', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '4950884399', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329454607640', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183183.497619, 'rx': '979944939543', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183183.497619, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183183.497619, 'rx': '45349014', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183183.497619, 'rx': '39829378', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40568', 'ksmPages': 100, 'elapsedTime': '1727931.60', 'cpuLoad': '8.38', 'cpuSys': '2.47', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '2.65', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150379, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:19:43 GMT', 'cpuUser': '33.09', 'memFree': 150123, 'cpuIdle': '64.44', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.19'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:19:43,519+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.08 seconds (__init__:312) 2019-04-13 21:19:44,992+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=cb5e91cc-8a11-4bd1-95ed-5847edc0f666 (api:48) 2019-04-13 21:19:44,997+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=cb5e91cc-8a11-4bd1-95ed-5847edc0f666 (api:54) 2019-04-13 21:19:44,998+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:19:45,010+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=588156b4-9a64-4384-8b57-7f92df3b4dd6 (api:48) 2019-04-13 21:19:45,018+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707190116352', 'isoprefix': '', 'alerts': [], 'disktotal': '3898287849472', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707190116352', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719938703360', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707190116352', 'isoprefix': '', 'alerts': [], 'disktotal': '4364578062336', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=588156b4-9a64-4384-8b57-7f92df3b4dd6 (api:54) 2019-04-13 21:19:45,019+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:19:46,091+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:19:46,122+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:19:46,122+0200 ERROR (jsonrpc/0) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:19:46,123+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:19:50,685+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:19:50,714+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:19:50,715+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:19:50,716+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:19:55,073+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=d8449d8e-8798-43a2-9703-6a5c8464a5dd (api:48) 2019-04-13 21:19:55,079+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=d8449d8e-8798-43a2-9703-6a5c8464a5dd (api:54) 2019-04-13 21:19:55,080+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:19:55,206+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=274e53d2-f3f5-403d-8f01-03c45572fef5 (api:48) 2019-04-13 21:19:55,213+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707186970624', 'isoprefix': '', 'alerts': [], 'disktotal': '3898284703744', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707186970624', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719935557632', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707186970624', 'isoprefix': '', 'alerts': [], 'disktotal': '4364574916608', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=274e53d2-f3f5-403d-8f01-03c45572fef5 (api:54) 2019-04-13 21:19:55,214+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:19:57,076+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_isos) Closing client (__init__:584) 2019-04-13 21:19:57,076+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/glusterSD/onode-2.discovery.intern:vol-node-2) Closing client (__init__:584) 2019-04-13 21:19:57,076+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_vms) Closing client (__init__:584) 2019-04-13 21:19:57,076+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/onode-1.discovery.intern:_mnt_store) Closing client (__init__:584) 2019-04-13 21:19:57,077+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Closing client (__init__:584) 2019-04-13 21:19:57,722+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=71002624-f986-4661-96dd-56cd2dc54083 (api:48) 2019-04-13 21:19:57,723+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000664393', 'lastCheck': '3.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000723957', 'lastCheck': '3.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000663062', 'lastCheck': '3.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000399304', 'lastCheck': '0.6', 'valid': True}} from=internal, task_id=71002624-f986-4661-96dd-56cd2dc54083 (api:54) 2019-04-13 21:19:58,507+0200 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:19:58,569+0200 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=e130ae9c-43e5-44f3-9096-d3b5633063e6 (api:48) 2019-04-13 21:19:58,570+0200 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000664393', 'lastCheck': '4.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000723957', 'lastCheck': '4.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000663062', 'lastCheck': '4.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000399304', 'lastCheck': '1.5', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=e130ae9c-43e5-44f3-9096-d3b5633063e6 (api:54) 2019-04-13 21:19:58,571+0200 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=6df8633f-5cd7-4989-a3f0-27ac76100fcf (api:48) 2019-04-13 21:19:58,571+0200 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=6df8633f-5cd7-4989-a3f0-27ac76100fcf (api:54) 2019-04-13 21:19:58,584+0200 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '24.11', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '73.37'}, '21': {'cpuUser': '26.29', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '71.46'}, '22': {'cpuUser': '33.18', 'nodeIndex': 1, 'cpuSys': '1.99', 'cpuIdle': '64.83'}, '23': {'cpuUser': '34.77', 'nodeIndex': 1, 'cpuSys': '1.92', 'cpuIdle': '63.31'}, '1': {'cpuUser': '39.01', 'nodeIndex': 0, 'cpuSys': '4.70', 'cpuIdle': '56.29'}, '0': {'cpuUser': '38.35', 'nodeIndex': 0, 'cpuSys': '2.38', 'cpuIdle': '59.27'}, '3': {'cpuUser': '42.72', 'nodeIndex': 0, 'cpuSys': '2.52', 'cpuIdle': '54.76'}, '2': {'cpuUser': '35.04', 'nodeIndex': 0, 'cpuSys': '2.58', 'cpuIdle': '62.38'}, '5': {'cpuUser': '41.40', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '56.55'}, '4': {'cpuUser': '43.71', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '54.10'}, '7': {'cpuUser': '40.00', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '57.15'}, '6': {'cpuUser': '43.32', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '53.83'}, '9': {'cpuUser': '41.66', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '55.76'}, '8': {'cpuUser': '40.14', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '57.01'}, '11': {'cpuUser': '40.87', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '56.61'}, '10': {'cpuUser': '40.00', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '57.28'}, '13': {'cpuUser': '18.48', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '79.60'}, '12': {'cpuUser': '32.12', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '65.89'}, '15': {'cpuUser': '17.22', 'nodeIndex': 0, 'cpuSys': '3.11', 'cpuIdle': '79.67'}, '14': {'cpuUser': '29.27', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '68.74'}, '17': {'cpuUser': '33.12', 'nodeIndex': 0, 'cpuSys': '1.26', 'cpuIdle': '65.62'}, '16': {'cpuUser': '24.51', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '73.50'}, '19': {'cpuUser': '27.16', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '70.19'}, '18': {'cpuUser': '23.18', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '74.63'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '71657'}, '0': {'memPercent': 23, 'memFree': '76073'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000664393', 'lastCheck': '4.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000663062', 'lastCheck': '4.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000399304', 'lastCheck': '1.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000723957', 'lastCheck': '4.3', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168845583', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '5091486060', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590436664945', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '2705575771092', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433776357180', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '259006172993', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239860407', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '366091070', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122790097', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121311594', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423622062912', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '253586384392', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215269393', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680359968', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '29546850845', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652287991', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '29289329553', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255774692682', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '7041681496', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121686147', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75270732863', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '61784148309', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122245077', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418919077', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '1012772775', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121497524', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183198.568631, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183198.568631, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183198.568631, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183198.568631, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121267555', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458731292873', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183198.568631, 'rx': '2576936337210', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322043101037', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183198.568631, 'rx': '1322043101037', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183198.568631, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940827635', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '2700624886411', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571495837310', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '4950884681', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329465499367', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183198.568631, 'rx': '979955571041', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183198.568631, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183198.568631, 'rx': '45349404', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183198.568631, 'rx': '39829746', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1727946.67', 'cpuLoad': '8.42', 'cpuSys': '2.44', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.18', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150388, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:19:58 GMT', 'cpuUser': '33.74', 'memFree': 150132, 'cpuIdle': '63.82', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.19'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:19:58,590+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.08 seconds (__init__:312) 2019-04-13 21:20:01,171+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:20:01,204+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:20:01,204+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:20:01,206+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:20:04,115+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID=u'ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', options=None) from=internal, task_id=e3ff2905-9e77-4688-b9af-a2d80b117cae (api:48) 2019-04-13 21:20:04,116+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', options=None) from=internal, task_id=bd1e542b-e092-4ada-aaf9-692d2bae146f (api:48) 2019-04-13 21:20:04,117+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', options=None) from=internal, task_id=b72e2681-cfb0-4f74-bfde-1de9ba6ece2f (api:48) 2019-04-13 21:20:04,121+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '4579690496', 'apparentsize': '16106127360'} from=internal, task_id=bd1e542b-e092-4ada-aaf9-692d2bae146f (api:54) 2019-04-13 21:20:04,122+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID='ef6741d1-380a-4736-a924-f88ed68e1270', options=None) from=internal, task_id=34411f8c-f708-4fe5-a3d6-75c7e1f05efd (api:48) 2019-04-13 21:20:04,125+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '5697695744', 'apparentsize': '16106127360'} from=internal, task_id=b72e2681-cfb0-4f74-bfde-1de9ba6ece2f (api:54) 2019-04-13 21:20:04,126+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID=u'67a4dd71-69ff-49a8-b8a4-c7101c75e105', options=None) from=internal, task_id=6ad00877-870a-49bc-9d31-13df7ec8c266 (api:48) 2019-04-13 21:20:04,130+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '53760942080', 'apparentsize': '139586437120'} from=internal, task_id=e3ff2905-9e77-4688-b9af-a2d80b117cae (api:54) 2019-04-13 21:20:04,131+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', options=None) from=internal, task_id=ffd64be5-a29d-4ef6-86fe-5adb48ab1c8d (api:48) 2019-04-13 21:20:04,134+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID=u'2b136ed7-8b9a-45c1-9edc-64ee96c87c43', options=None) from=internal, task_id=67550cc6-0ef5-4873-be89-dbbdf1ec2e96 (api:48) 2019-04-13 21:20:04,137+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '55059126272', 'apparentsize': '171798691840'} from=internal, task_id=6ad00877-870a-49bc-9d31-13df7ec8c266 (api:54) 2019-04-13 21:20:04,138+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID=u'44b3ceca-f0db-43da-af29-aa681df146ac', options=None) from=internal, task_id=dd33a9b1-fa76-47e6-bffc-d2d5ff6573d2 (api:48) 2019-04-13 21:20:04,139+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '6477467136', 'apparentsize': '21474836480'} from=internal, task_id=34411f8c-f708-4fe5-a3d6-75c7e1f05efd (api:54) 2019-04-13 21:20:04,140+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=8d1e8072-667d-4fd7-9c73-64c97961892d (api:48) 2019-04-13 21:20:04,145+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2024405504', 'apparentsize': '10737418240'} from=internal, task_id=67550cc6-0ef5-4873-be89-dbbdf1ec2e96 (api:54) 2019-04-13 21:20:04,146+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID='123e2cc8-8f24-4ddd-b401-68393535f555', options=None) from=internal, task_id=bcb45be0-8452-4dcb-aaed-0a526f17e441 (api:48) 2019-04-13 21:20:04,147+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2524568064', 'apparentsize': '12884901888'} from=internal, task_id=ffd64be5-a29d-4ef6-86fe-5adb48ab1c8d (api:54) 2019-04-13 21:20:04,148+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID='3f6c31c6-ec38-4979-a617-55537123ee61', options=None) from=internal, task_id=89d21655-f61b-49b9-a28c-f936de5b1daa (api:48) 2019-04-13 21:20:04,152+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '11196700160', 'apparentsize': '21474836480'} from=internal, task_id=dd33a9b1-fa76-47e6-bffc-d2d5ff6573d2 (api:54) 2019-04-13 21:20:04,153+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', options=None) from=internal, task_id=74a04e95-79ab-4e3b-abfd-287180e67b87 (api:48) 2019-04-13 21:20:04,158+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2116492288', 'apparentsize': '3948150784'} from=internal, task_id=8d1e8072-667d-4fd7-9c73-64c97961892d (api:54) 2019-04-13 21:20:04,159+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID=u'a1d583c1-e135-4e3f-b6fa-0f29a5261366', options=None) from=internal, task_id=85330cdc-6953-457d-b6e0-b4b10bdea447 (api:48) 2019-04-13 21:20:04,159+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '4998722048', 'apparentsize': '16106127360'} from=internal, task_id=89d21655-f61b-49b9-a28c-f936de5b1daa (api:54) 2019-04-13 21:20:04,160+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', options=None) from=internal, task_id=c9b18d84-cb1f-4183-87ec-353840f58e67 (api:48) 2019-04-13 21:20:04,161+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2283003904', 'apparentsize': '16106127360'} from=internal, task_id=bcb45be0-8452-4dcb-aaed-0a526f17e441 (api:54) 2019-04-13 21:20:04,162+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID='a884b4ce-381f-4653-be14-a31d15181f48', options=None) from=internal, task_id=6466b9ea-d173-4339-9290-3f64e738aa8c (api:48) 2019-04-13 21:20:04,166+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2109410304', 'apparentsize': '8589934592'} from=internal, task_id=74a04e95-79ab-4e3b-abfd-287180e67b87 (api:54) 2019-04-13 21:20:04,167+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', options=None) from=internal, task_id=a493beb3-caca-4c42-a336-28ebd4daf469 (api:48) 2019-04-13 21:20:04,181+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '1294894592', 'apparentsize': '12884901888'} from=internal, task_id=85330cdc-6953-457d-b6e0-b4b10bdea447 (api:54) 2019-04-13 21:20:04,184+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '3447292416', 'apparentsize': '16106127360'} from=internal, task_id=6466b9ea-d173-4339-9290-3f64e738aa8c (api:54) 2019-04-13 21:20:04,185+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '7263196672', 'apparentsize': '21474836480'} from=internal, task_id=c9b18d84-cb1f-4183-87ec-353840f58e67 (api:54) 2019-04-13 21:20:04,190+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2879759360', 'apparentsize': '8589934592'} from=internal, task_id=a493beb3-caca-4c42-a336-28ebd4daf469 (api:54) 2019-04-13 21:20:05,267+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=2d46b7fc-505c-4e88-97a8-913037b653d5 (api:48) 2019-04-13 21:20:05,273+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=2d46b7fc-505c-4e88-97a8-913037b653d5 (api:54) 2019-04-13 21:20:05,273+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:20:05,288+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=3ab098c8-c5b8-4bfb-9ee3-1f338eda592c (api:48) 2019-04-13 21:20:05,298+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707186970624', 'isoprefix': '', 'alerts': [], 'disktotal': '3898284703744', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707186970624', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719935557632', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707186970624', 'isoprefix': '', 'alerts': [], 'disktotal': '4364574916608', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=3ab098c8-c5b8-4bfb-9ee3-1f338eda592c (api:54) 2019-04-13 21:20:05,300+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:20:05,725+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:20:05,759+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:20:05,759+0200 ERROR (jsonrpc/0) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:20:05,761+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:20:12,814+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=596309ee-eb01-4734-8519-161262baab37 (api:48) 2019-04-13 21:20:12,815+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000641675', 'lastCheck': '8.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000459306', 'lastCheck': '8.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000649215', 'lastCheck': '8.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000352868', 'lastCheck': '5.7', 'valid': True}} from=internal, task_id=596309ee-eb01-4734-8519-161262baab37 (api:54) 2019-04-13 21:20:13,557+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:20:13,635+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=483ed574-fe07-4386-b83c-59ee15da22b0 (api:48) 2019-04-13 21:20:13,636+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000641675', 'lastCheck': '0.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000459306', 'lastCheck': '0.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000649215', 'lastCheck': '0.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000352868', 'lastCheck': '6.5', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=483ed574-fe07-4386-b83c-59ee15da22b0 (api:54) 2019-04-13 21:20:13,637+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=bdaba30d-e844-4161-928b-6a622f706e28 (api:48) 2019-04-13 21:20:13,637+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=bdaba30d-e844-4161-928b-6a622f706e28 (api:54) 2019-04-13 21:20:13,651+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '31.57', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '65.32'}, '21': {'cpuUser': '25.75', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '71.80'}, '22': {'cpuUser': '32.10', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '65.45'}, '23': {'cpuUser': '24.16', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '73.06'}, '1': {'cpuUser': '41.30', 'nodeIndex': 0, 'cpuSys': '4.17', 'cpuIdle': '54.53'}, '0': {'cpuUser': '42.62', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '55.13'}, '3': {'cpuUser': '52.62', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '45.53'}, '2': {'cpuUser': '44.34', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '53.67'}, '5': {'cpuUser': '46.20', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '51.75'}, '4': {'cpuUser': '46.60', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '51.35'}, '7': {'cpuUser': '33.42', 'nodeIndex': 1, 'cpuSys': '3.64', 'cpuIdle': '62.94'}, '6': {'cpuUser': '42.16', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '54.86'}, '9': {'cpuUser': '41.17', 'nodeIndex': 1, 'cpuSys': '3.04', 'cpuIdle': '55.79'}, '8': {'cpuUser': '33.42', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '63.47'}, '11': {'cpuUser': '42.29', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '55.13'}, '10': {'cpuUser': '36.93', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '60.09'}, '13': {'cpuUser': '27.07', 'nodeIndex': 0, 'cpuSys': '2.38', 'cpuIdle': '70.55'}, '12': {'cpuUser': '30.25', 'nodeIndex': 0, 'cpuSys': '1.65', 'cpuIdle': '68.10'}, '15': {'cpuUser': '21.58', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '76.17'}, '14': {'cpuUser': '26.54', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '71.41'}, '17': {'cpuUser': '21.64', 'nodeIndex': 0, 'cpuSys': '2.18', 'cpuIdle': '76.18'}, '16': {'cpuUser': '20.12', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '77.76'}, '19': {'cpuUser': '28.79', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '68.89'}, '18': {'cpuUser': '23.10', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '74.25'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '71613'}, '0': {'memPercent': 23, 'memFree': '76109'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000641675', 'lastCheck': '0.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000649215', 'lastCheck': '0.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000352868', 'lastCheck': '6.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000459306', 'lastCheck': '0.2', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168845665', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '5091486060', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590437777203', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '2705576934356', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433780494084', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '259008813051', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239860825', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '366091468', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122790179', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121311676', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423626157104', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '253589013025', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215269475', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680360158', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '29546850942', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652288088', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '29289329743', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255777346453', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '7043695459', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121686229', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75273127171', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '61788901515', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122245159', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418919497', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '1012772775', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121497606', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183213.634305, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183213.634305, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183213.634305, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183213.634305, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121267637', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458732442835', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183213.634305, 'rx': '2576937449178', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322054498316', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183213.634305, 'rx': '1322054498316', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183213.634305, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940827635', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '2700626049131', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571496949568', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '4950885225', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329475209218', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183213.634305, 'rx': '979966576708', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183213.634305, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183213.634305, 'rx': '45349794', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183213.634305, 'rx': '39830068', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40570', 'ksmPages': 100, 'elapsedTime': '1727961.72', 'cpuLoad': '8.32', 'cpuSys': '2.56', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.24', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150379, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:20:13 GMT', 'cpuUser': '33.99', 'memFree': 150123, 'cpuIdle': '63.45', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.32'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:20:13,656+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.10 seconds (__init__:312) 2019-04-13 21:20:15,346+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=3614a540-54fb-45c4-a9bd-f15913be4355 (api:48) 2019-04-13 21:20:15,351+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=3614a540-54fb-45c4-a9bd-f15913be4355 (api:54) 2019-04-13 21:20:15,351+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:20:15,363+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=127cb07b-ab71-4a00-965b-5234c466798d (api:48) 2019-04-13 21:20:15,371+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707181727744', 'isoprefix': '', 'alerts': [], 'disktotal': '3898279460864', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707181727744', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719930314752', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707181727744', 'isoprefix': '', 'alerts': [], 'disktotal': '4364570722304', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=127cb07b-ab71-4a00-965b-5234c466798d (api:54) 2019-04-13 21:20:15,372+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:20:16,251+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:20:16,290+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:20:16,291+0200 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:20:16,293+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:20:20,769+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:20:20,812+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:20:20,812+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:20:20,814+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.05 seconds (__init__:312) 2019-04-13 21:20:25,416+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=5baad4df-de85-45fa-9ca0-58a967827454 (api:48) 2019-04-13 21:20:25,421+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=5baad4df-de85-45fa-9ca0-58a967827454 (api:54) 2019-04-13 21:20:25,421+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:20:25,527+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=2aef8093-6225-47c7-816b-a9bbb2b91a8f (api:48) 2019-04-13 21:20:25,534+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707177533440', 'isoprefix': '', 'alerts': [], 'disktotal': '3898275266560', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707177533440', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719926120448', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707177533440', 'isoprefix': '', 'alerts': [], 'disktotal': '4364565479424', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=2aef8093-6225-47c7-816b-a9bbb2b91a8f (api:54) 2019-04-13 21:20:25,536+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:20:27,937+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=c114e322-70f8-43a7-afd4-69ca5dd40c20 (api:48) 2019-04-13 21:20:27,938+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000740422', 'lastCheck': '3.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000786502', 'lastCheck': '3.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000718224', 'lastCheck': '3.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000426412', 'lastCheck': '0.8', 'valid': True}} from=internal, task_id=c114e322-70f8-43a7-afd4-69ca5dd40c20 (api:54) 2019-04-13 21:20:28,627+0200 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:20:28,697+0200 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=074c69ac-0f03-48cc-ba3c-7890067d1531 (api:48) 2019-04-13 21:20:28,698+0200 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000740422', 'lastCheck': '4.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000786502', 'lastCheck': '4.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000718224', 'lastCheck': '4.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000426412', 'lastCheck': '1.6', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=074c69ac-0f03-48cc-ba3c-7890067d1531 (api:54) 2019-04-13 21:20:28,698+0200 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=9e216ef9-a155-4265-b9d2-e5e78d792b1c (api:48) 2019-04-13 21:20:28,699+0200 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=9e216ef9-a155-4265-b9d2-e5e78d792b1c (api:54) 2019-04-13 21:20:28,713+0200 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '19.33', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '77.82'}, '21': {'cpuUser': '21.72', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '75.43'}, '22': {'cpuUser': '20.13', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '76.76'}, '23': {'cpuUser': '22.97', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '74.05'}, '1': {'cpuUser': '42.17', 'nodeIndex': 0, 'cpuSys': '4.83', 'cpuIdle': '53.00'}, '0': {'cpuUser': '47.01', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '50.94'}, '3': {'cpuUser': '51.24', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '46.64'}, '2': {'cpuUser': '46.81', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '51.34'}, '5': {'cpuUser': '43.76', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '54.52'}, '4': {'cpuUser': '47.21', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '51.13'}, '7': {'cpuUser': '38.40', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '58.55'}, '6': {'cpuUser': '50.25', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '47.50'}, '9': {'cpuUser': '44.09', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '53.06'}, '8': {'cpuUser': '42.70', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '54.25'}, '11': {'cpuUser': '42.17', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '54.72'}, '10': {'cpuUser': '44.36', 'nodeIndex': 1, 'cpuSys': '2.91', 'cpuIdle': '52.73'}, '13': {'cpuUser': '37.61', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '60.47'}, '12': {'cpuUser': '39.79', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '58.22'}, '15': {'cpuUser': '26.35', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '71.73'}, '14': {'cpuUser': '40.32', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '57.76'}, '17': {'cpuUser': '48.07', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '50.14'}, '16': {'cpuUser': '34.76', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '63.65'}, '19': {'cpuUser': '24.89', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '72.53'}, '18': {'cpuUser': '24.43', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '73.12'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '71196'}, '0': {'memPercent': 23, 'memFree': '76534'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000740422', 'lastCheck': '4.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000718224', 'lastCheck': '4.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000426412', 'lastCheck': '1.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000786502', 'lastCheck': '4.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168846171', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '5091486470', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590438320151', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '2705576967468', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433784692150', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '259011636037', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239860999', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '366091468', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122790353', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121311850', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423630251414', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '253591809649', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215269649', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680360332', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '29546850942', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652288180', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '29289329825', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255780208523', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '7045831838', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121686403', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75275676877', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '61796546457', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122245333', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418919977', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '1012772775', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121497780', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183228.696192, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183228.696192, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183228.696192, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183228.696192, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121267811', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458732881553', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183228.696192, 'rx': '2576937473954', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322066976179', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183228.696192, 'rx': '1322066976179', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183228.696192, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940828179', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '2700626081801', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571497491972', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '4950885667', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329488161499', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183228.696192, 'rx': '979977968815', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183228.696192, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183228.696192, 'rx': '45350184', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183228.696192, 'rx': '39830436', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1727976.79', 'cpuLoad': '8.27', 'cpuSys': '2.47', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.44', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150387, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:20:28 GMT', 'cpuUser': '37.52', 'memFree': 150131, 'cpuIdle': '60.01', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.39'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:20:28,719+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:20:31,331+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:20:31,372+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:20:31,373+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:20:31,374+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.05 seconds (__init__:312) 2019-04-13 21:20:35,591+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=7bedd792-c2c7-478a-b435-f6261f5f5384 (api:48) 2019-04-13 21:20:35,596+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=7bedd792-c2c7-478a-b435-f6261f5f5384 (api:54) 2019-04-13 21:20:35,597+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:20:35,614+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=744f7cec-982f-4bcd-8f5d-60c2ed9a7876 (api:48) 2019-04-13 21:20:35,621+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707174387712', 'isoprefix': '', 'alerts': [], 'disktotal': '3898272120832', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707174387712', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719922974720', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707174387712', 'isoprefix': '', 'alerts': [], 'disktotal': '4364563382272', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=744f7cec-982f-4bcd-8f5d-60c2ed9a7876 (api:54) 2019-04-13 21:20:35,622+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:20:35,823+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:20:35,912+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:20:35,913+0200 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:20:35,914+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.10 seconds (__init__:312) 2019-04-13 21:20:43,042+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=84a4cf01-dfb3-4f98-92c3-2acbd1c591b7 (api:48) 2019-04-13 21:20:43,043+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000567761', 'lastCheck': '8.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.0007326', 'lastCheck': '8.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000564115', 'lastCheck': '8.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000403847', 'lastCheck': '5.9', 'valid': True}} from=internal, task_id=84a4cf01-dfb3-4f98-92c3-2acbd1c591b7 (api:54) 2019-04-13 21:20:43,673+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:20:43,752+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=0cec521b-523e-4843-a234-9a433d11ea81 (api:48) 2019-04-13 21:20:43,752+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000567761', 'lastCheck': '0.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.0007326', 'lastCheck': '0.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000564115', 'lastCheck': '0.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000403847', 'lastCheck': '6.6', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=0cec521b-523e-4843-a234-9a433d11ea81 (api:54) 2019-04-13 21:20:43,753+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=63554fb4-324e-4c0f-bba3-522e18c24606 (api:48) 2019-04-13 21:20:43,753+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=63554fb4-324e-4c0f-bba3-522e18c24606 (api:54) 2019-04-13 21:20:43,765+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '19.11', 'nodeIndex': 1, 'cpuSys': '2.51', 'cpuIdle': '78.38'}, '21': {'cpuUser': '27.25', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '70.04'}, '22': {'cpuUser': '26.52', 'nodeIndex': 1, 'cpuSys': '3.17', 'cpuIdle': '70.31'}, '23': {'cpuUser': '30.56', 'nodeIndex': 1, 'cpuSys': '1.85', 'cpuIdle': '67.59'}, '1': {'cpuUser': '38.96', 'nodeIndex': 0, 'cpuSys': '4.56', 'cpuIdle': '56.48'}, '0': {'cpuUser': '49.34', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '48.81'}, '3': {'cpuUser': '64.02', 'nodeIndex': 0, 'cpuSys': '1.39', 'cpuIdle': '34.59'}, '2': {'cpuUser': '59.66', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '38.75'}, '5': {'cpuUser': '50.80', 'nodeIndex': 0, 'cpuSys': '1.52', 'cpuIdle': '47.68'}, '4': {'cpuUser': '47.36', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '50.92'}, '7': {'cpuUser': '39.42', 'nodeIndex': 1, 'cpuSys': '3.51', 'cpuIdle': '57.07'}, '6': {'cpuUser': '41.21', 'nodeIndex': 1, 'cpuSys': '3.04', 'cpuIdle': '55.75'}, '9': {'cpuUser': '45.64', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '51.38'}, '8': {'cpuUser': '48.22', 'nodeIndex': 1, 'cpuSys': '2.84', 'cpuIdle': '48.94'}, '11': {'cpuUser': '42.07', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '55.35'}, '10': {'cpuUser': '38.16', 'nodeIndex': 1, 'cpuSys': '3.44', 'cpuIdle': '58.40'}, '13': {'cpuUser': '31.75', 'nodeIndex': 0, 'cpuSys': '1.98', 'cpuIdle': '66.27'}, '12': {'cpuUser': '25.66', 'nodeIndex': 0, 'cpuSys': '2.31', 'cpuIdle': '72.03'}, '15': {'cpuUser': '12.04', 'nodeIndex': 0, 'cpuSys': '3.04', 'cpuIdle': '84.92'}, '14': {'cpuUser': '15.68', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '82.40'}, '17': {'cpuUser': '27.32', 'nodeIndex': 0, 'cpuSys': '1.98', 'cpuIdle': '70.70'}, '16': {'cpuUser': '40.28', 'nodeIndex': 0, 'cpuSys': '1.65', 'cpuIdle': '58.07'}, '19': {'cpuUser': '29.43', 'nodeIndex': 1, 'cpuSys': '2.91', 'cpuIdle': '67.66'}, '18': {'cpuUser': '27.91', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '69.84'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '70964'}, '0': {'memPercent': 22, 'memFree': '76765'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000567761', 'lastCheck': '0.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000564115', 'lastCheck': '0.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000403847', 'lastCheck': '6.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.0007326', 'lastCheck': '0.3', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168846351', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '5091486554', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590438772361', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '2705578087622', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433788400300', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '259014343722', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239861059', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '366091468', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122790413', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121311910', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423634288232', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '253594554650', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215269709', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680360392', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '29546850942', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652288442', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '29289330013', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255782999283', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '7050162370', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121686463', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75277940311', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '61800772815', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122245393', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418920397', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '1012772775', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121497840', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183243.750965, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183243.750965, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183243.750965, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183243.750965, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121267871', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458733313095', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183243.750965, 'rx': '2576938549552', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322076288706', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183243.750965, 'rx': '1322076288706', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183243.750965, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940828307', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '2700627201209', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571497944054', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '4950886413', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329499359281', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183243.750965, 'rx': '979988513475', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183243.750965, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183243.750965, 'rx': '45350574', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183243.750965, 'rx': '39830758', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1727991.84', 'cpuLoad': '8.33', 'cpuSys': '2.47', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.44', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150386, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:20:43 GMT', 'cpuUser': '36.61', 'memFree': 150130, 'cpuIdle': '60.92', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '0.99'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:20:43,772+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.10 seconds (__init__:312) 2019-04-13 21:20:45,665+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=11153bff-fb1c-4584-b941-ba75b3b6ed04 (api:48) 2019-04-13 21:20:45,669+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=11153bff-fb1c-4584-b941-ba75b3b6ed04 (api:54) 2019-04-13 21:20:45,670+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:20:45,680+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=e48d6c2d-ffa5-4b9f-b925-104e4a3e1e58 (api:48) 2019-04-13 21:20:45,688+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707174387712', 'isoprefix': '', 'alerts': [], 'disktotal': '3898272120832', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707174387712', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719922974720', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707174387712', 'isoprefix': '', 'alerts': [], 'disktotal': '4364563382272', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=e48d6c2d-ffa5-4b9f-b925-104e4a3e1e58 (api:54) 2019-04-13 21:20:45,689+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:20:46,411+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:20:46,454+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:20:46,455+0200 ERROR (jsonrpc/0) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:20:46,456+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.05 seconds (__init__:312) 2019-04-13 21:20:50,920+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:20:50,948+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:20:50,949+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:20:50,950+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:20:55,730+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] 1430 requests processed during 3600 seconds (__init__:299) 2019-04-13 21:20:55,731+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=be2e030e-01ce-4754-8ff4-27e3992a354b (api:48) 2019-04-13 21:20:55,737+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=be2e030e-01ce-4754-8ff4-27e3992a354b (api:54) 2019-04-13 21:20:55,738+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:20:55,754+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=46b6c1af-9e17-4980-9809-f0e6ca96d0be (api:48) 2019-04-13 21:20:55,762+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707170193408', 'isoprefix': '', 'alerts': [], 'disktotal': '3898267926528', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707170193408', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719918780416', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707170193408', 'isoprefix': '', 'alerts': [], 'disktotal': '4364559187968', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=46b6c1af-9e17-4980-9809-f0e6ca96d0be (api:54) 2019-04-13 21:20:55,763+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:20:57,105+0200 WARN (qgapoller/5) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:20:57,571+0200 WARN (qgapoller/5) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090050> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:20:57,805+0200 WARN (qgapoller/9) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc086ed8> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:20:58,136+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=b90f3222-7d37-4e70-8f90-246e95de694d (api:48) 2019-04-13 21:20:58,137+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000682685', 'lastCheck': '3.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000653553', 'lastCheck': '3.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000668934', 'lastCheck': '3.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000427812', 'lastCheck': '1.0', 'valid': True}} from=internal, task_id=b90f3222-7d37-4e70-8f90-246e95de694d (api:54) 2019-04-13 21:20:58,788+0200 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:20:58,857+0200 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=413dc468-1f19-454b-9da8-8e15ecf1bfba (api:48) 2019-04-13 21:20:58,857+0200 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000682685', 'lastCheck': '4.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000653553', 'lastCheck': '4.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000668934', 'lastCheck': '4.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000427812', 'lastCheck': '1.7', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=413dc468-1f19-454b-9da8-8e15ecf1bfba (api:54) 2019-04-13 21:20:58,858+0200 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=8c147d34-13ea-4362-bb3e-6a757a56de29 (api:48) 2019-04-13 21:20:58,858+0200 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=8c147d34-13ea-4362-bb3e-6a757a56de29 (api:54) 2019-04-13 21:20:58,872+0200 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '24.70', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '72.65'}, '21': {'cpuUser': '28.34', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '68.88'}, '22': {'cpuUser': '24.83', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '72.59'}, '23': {'cpuUser': '20.33', 'nodeIndex': 1, 'cpuSys': '2.91', 'cpuIdle': '76.76'}, '1': {'cpuUser': '41.06', 'nodeIndex': 0, 'cpuSys': '4.64', 'cpuIdle': '54.30'}, '0': {'cpuUser': '44.90', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '53.05'}, '3': {'cpuUser': '42.71', 'nodeIndex': 0, 'cpuSys': '2.58', 'cpuIdle': '54.71'}, '2': {'cpuUser': '30.39', 'nodeIndex': 0, 'cpuSys': '2.71', 'cpuIdle': '66.90'}, '5': {'cpuUser': '48.08', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '50.00'}, '4': {'cpuUser': '45.82', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '52.19'}, '7': {'cpuUser': '36.02', 'nodeIndex': 1, 'cpuSys': '3.51', 'cpuIdle': '60.47'}, '6': {'cpuUser': '38.80', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '58.09'}, '9': {'cpuUser': '36.02', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '60.93'}, '8': {'cpuUser': '41.19', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '55.76'}, '11': {'cpuUser': '42.78', 'nodeIndex': 1, 'cpuSys': '2.91', 'cpuIdle': '54.31'}, '10': {'cpuUser': '41.65', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '55.37'}, '13': {'cpuUser': '21.12', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '77.09'}, '12': {'cpuUser': '15.16', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '82.52'}, '15': {'cpuUser': '30.73', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '67.42'}, '14': {'cpuUser': '36.88', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '61.53'}, '17': {'cpuUser': '22.32', 'nodeIndex': 0, 'cpuSys': '1.52', 'cpuIdle': '76.16'}, '16': {'cpuUser': '23.31', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '74.97'}, '19': {'cpuUser': '24.30', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '73.18'}, '18': {'cpuUser': '29.34', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '68.41'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '71112'}, '0': {'memPercent': 23, 'memFree': '76618'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000682685', 'lastCheck': '4.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000668934', 'lastCheck': '4.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000427812', 'lastCheck': '1.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000653553', 'lastCheck': '4.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168846515', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '5091486554', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590439343621', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '2705579677204', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433792350838', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '259017025063', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239861223', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '366091468', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122790577', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121312074', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423638258900', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '253597254388', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215269873', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680360556', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '29546850942', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652288442', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '29289330177', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255785699500', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '7052300927', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121686627', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75279636901', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '61803678768', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122245557', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418920877', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '1012772775', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121498004', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183258.855828, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183258.855828, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183258.855828, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183258.855828, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121268035', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458733862937', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183258.855828, 'rx': '2576940064118', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322089798897', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183258.855828, 'rx': '1322089798897', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183258.855828, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940828521', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '2700628790573', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571498515100', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '4950886631', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329507789991', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183258.855828, 'rx': '979998228298', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183258.855828, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183258.855828, 'rx': '45350964', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183258.855828, 'rx': '39831126', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1728006.95', 'cpuLoad': '8.47', 'cpuSys': '2.54', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.71', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150388, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:20:58 GMT', 'cpuUser': '32.95', 'memFree': 150132, 'cpuIdle': '64.51', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.59'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:20:58,878+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:21:01,489+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:21:01,518+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:21:01,519+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:21:01,519+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:21:04,112+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID=u'ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', options=None) from=internal, task_id=cd5f8a03-7271-4331-973f-e38fb7fcbe4b (api:48) 2019-04-13 21:21:04,114+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', options=None) from=internal, task_id=48637878-21f0-4549-89f3-099b3ff00363 (api:48) 2019-04-13 21:21:04,115+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', options=None) from=internal, task_id=24ac8f43-8b47-4067-832b-286a61d051f1 (api:48) 2019-04-13 21:21:04,119+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '5697701376', 'apparentsize': '16106127360'} from=internal, task_id=24ac8f43-8b47-4067-832b-286a61d051f1 (api:54) 2019-04-13 21:21:04,120+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID='ef6741d1-380a-4736-a924-f88ed68e1270', options=None) from=internal, task_id=63846907-5ea6-4b88-afc2-2de6d7a9aee3 (api:48) 2019-04-13 21:21:04,121+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '4579690496', 'apparentsize': '16106127360'} from=internal, task_id=48637878-21f0-4549-89f3-099b3ff00363 (api:54) 2019-04-13 21:21:04,122+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID=u'67a4dd71-69ff-49a8-b8a4-c7101c75e105', options=None) from=internal, task_id=39c62a50-92ad-4db9-9bf1-ae35a449b26c (api:48) 2019-04-13 21:21:04,124+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '53760947712', 'apparentsize': '139586437120'} from=internal, task_id=cd5f8a03-7271-4331-973f-e38fb7fcbe4b (api:54) 2019-04-13 21:21:04,125+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', options=None) from=internal, task_id=e851cd2d-35ae-431e-9512-b2d45acb6f32 (api:48) 2019-04-13 21:21:04,131+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID=u'2b136ed7-8b9a-45c1-9edc-64ee96c87c43', options=None) from=internal, task_id=3054f4e1-8357-4bac-bece-3a578f06aee2 (api:48) 2019-04-13 21:21:04,133+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '55059126272', 'apparentsize': '171798691840'} from=internal, task_id=39c62a50-92ad-4db9-9bf1-ae35a449b26c (api:54) 2019-04-13 21:21:04,134+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID=u'44b3ceca-f0db-43da-af29-aa681df146ac', options=None) from=internal, task_id=bce253b5-e81d-421e-bb32-f54938b2852f (api:48) 2019-04-13 21:21:04,135+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '6477472768', 'apparentsize': '21474836480'} from=internal, task_id=63846907-5ea6-4b88-afc2-2de6d7a9aee3 (api:54) 2019-04-13 21:21:04,136+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=d66609a4-eed5-4daf-a251-66e25868d8b9 (api:48) 2019-04-13 21:21:04,140+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2524573696', 'apparentsize': '12884901888'} from=internal, task_id=e851cd2d-35ae-431e-9512-b2d45acb6f32 (api:54) 2019-04-13 21:21:04,140+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID='123e2cc8-8f24-4ddd-b401-68393535f555', options=None) from=internal, task_id=b950221b-5657-40da-bf88-69812f943f0f (api:48) 2019-04-13 21:21:04,143+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2024405504', 'apparentsize': '10737418240'} from=internal, task_id=3054f4e1-8357-4bac-bece-3a578f06aee2 (api:54) 2019-04-13 21:21:04,144+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID='3f6c31c6-ec38-4979-a617-55537123ee61', options=None) from=internal, task_id=44b2e9ff-474f-4b2b-a6a1-8b8f89fa5f91 (api:48) 2019-04-13 21:21:04,146+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2116481024', 'apparentsize': '3948150784'} from=internal, task_id=d66609a4-eed5-4daf-a251-66e25868d8b9 (api:54) 2019-04-13 21:21:04,146+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', options=None) from=internal, task_id=434eb7b8-9202-47ea-af8e-a4532d2ca609 (api:48) 2019-04-13 21:21:04,147+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '11196694528', 'apparentsize': '21474836480'} from=internal, task_id=bce253b5-e81d-421e-bb32-f54938b2852f (api:54) 2019-04-13 21:21:04,148+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID=u'a1d583c1-e135-4e3f-b6fa-0f29a5261366', options=None) from=internal, task_id=be56c107-88c9-4a6c-a4ca-945ea674a4b5 (api:48) 2019-04-13 21:21:04,152+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2283003904', 'apparentsize': '16106127360'} from=internal, task_id=b950221b-5657-40da-bf88-69812f943f0f (api:54) 2019-04-13 21:21:04,153+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', options=None) from=internal, task_id=19a52747-01c7-46e7-b3d8-be3a2a1bfd29 (api:48) 2019-04-13 21:21:04,157+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '4998722048', 'apparentsize': '16106127360'} from=internal, task_id=44b2e9ff-474f-4b2b-a6a1-8b8f89fa5f91 (api:54) 2019-04-13 21:21:04,158+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID='a884b4ce-381f-4653-be14-a31d15181f48', options=None) from=internal, task_id=99520aab-6594-4289-ae00-c7c68e466b22 (api:48) 2019-04-13 21:21:04,159+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '1294894592', 'apparentsize': '12884901888'} from=internal, task_id=be56c107-88c9-4a6c-a4ca-945ea674a4b5 (api:54) 2019-04-13 21:21:04,159+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', options=None) from=internal, task_id=d1900996-2468-4272-843a-4880d050b1e0 (api:48) 2019-04-13 21:21:04,160+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2109410304', 'apparentsize': '8589934592'} from=internal, task_id=434eb7b8-9202-47ea-af8e-a4532d2ca609 (api:54) 2019-04-13 21:21:04,163+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '7263196672', 'apparentsize': '21474836480'} from=internal, task_id=19a52747-01c7-46e7-b3d8-be3a2a1bfd29 (api:54) 2019-04-13 21:21:04,166+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '3447292416', 'apparentsize': '16106127360'} from=internal, task_id=99520aab-6594-4289-ae00-c7c68e466b22 (api:54) 2019-04-13 21:21:04,167+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2879759360', 'apparentsize': '8589934592'} from=internal, task_id=d1900996-2468-4272-843a-4880d050b1e0 (api:54) 2019-04-13 21:21:05,815+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=79eda566-c611-416f-a1f6-9377b7c8769f (api:48) 2019-04-13 21:21:05,819+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=79eda566-c611-416f-a1f6-9377b7c8769f (api:54) 2019-04-13 21:21:05,820+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:21:05,956+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:21:05,980+0200 INFO (jsonrpc/0) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=f53120c8-d002-420b-805c-50eae95b37c8 (api:48) 2019-04-13 21:21:06,002+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707167047680', 'isoprefix': '', 'alerts': [], 'disktotal': '3898264780800', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707167047680', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719915634688', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707167047680', 'isoprefix': '', 'alerts': [], 'disktotal': '4364556042240', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=f53120c8-d002-420b-805c-50eae95b37c8 (api:54) 2019-04-13 21:21:06,004+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:21:06,011+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:21:06,012+0200 ERROR (jsonrpc/6) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:21:06,013+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.05 seconds (__init__:312) 2019-04-13 21:21:07,102+0200 WARN (qgapoller/10) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:21:13,237+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=c12e7242-8e33-4dbe-9396-493d0839bc21 (api:48) 2019-04-13 21:21:13,238+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000664812', 'lastCheck': '8.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000535909', 'lastCheck': '9.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000702948', 'lastCheck': '8.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00222809', 'lastCheck': '6.1', 'valid': True}} from=internal, task_id=c12e7242-8e33-4dbe-9396-493d0839bc21 (api:54) 2019-04-13 21:21:13,832+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:21:13,892+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=b3ce75e0-0149-42da-9ea5-3c820e9348ab (api:48) 2019-04-13 21:21:13,892+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000664812', 'lastCheck': '0.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000535909', 'lastCheck': '0.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000702948', 'lastCheck': '0.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00222809', 'lastCheck': '6.7', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=b3ce75e0-0149-42da-9ea5-3c820e9348ab (api:54) 2019-04-13 21:21:13,893+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=337bee9f-7a86-4109-8222-677833998519 (api:48) 2019-04-13 21:21:13,893+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=337bee9f-7a86-4109-8222-677833998519 (api:54) 2019-04-13 21:21:13,906+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '23.04', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '74.51'}, '21': {'cpuUser': '25.76', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '72.12'}, '22': {'cpuUser': '30.86', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '66.95'}, '23': {'cpuUser': '22.12', 'nodeIndex': 1, 'cpuSys': '2.38', 'cpuIdle': '75.50'}, '1': {'cpuUser': '53.90', 'nodeIndex': 0, 'cpuSys': '4.04', 'cpuIdle': '42.06'}, '0': {'cpuUser': '60.19', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '38.15'}, '3': {'cpuUser': '61.06', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '36.95'}, '2': {'cpuUser': '56.35', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '41.66'}, '5': {'cpuUser': '58.47', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '39.74'}, '4': {'cpuUser': '58.21', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '40.13'}, '7': {'cpuUser': '40.20', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '57.02'}, '6': {'cpuUser': '35.96', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '61.19'}, '9': {'cpuUser': '36.55', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '60.60'}, '8': {'cpuUser': '44.83', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '52.59'}, '11': {'cpuUser': '36.02', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '61.13'}, '10': {'cpuUser': '40.99', 'nodeIndex': 1, 'cpuSys': '2.38', 'cpuIdle': '56.63'}, '13': {'cpuUser': '25.50', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '72.05'}, '12': {'cpuUser': '23.04', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '74.71'}, '15': {'cpuUser': '12.91', 'nodeIndex': 0, 'cpuSys': '2.91', 'cpuIdle': '84.18'}, '14': {'cpuUser': '18.34', 'nodeIndex': 0, 'cpuSys': '3.18', 'cpuIdle': '78.48'}, '17': {'cpuUser': '28.94', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '69.07'}, '16': {'cpuUser': '27.15', 'nodeIndex': 0, 'cpuSys': '2.65', 'cpuIdle': '70.20'}, '19': {'cpuUser': '27.35', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '70.33'}, '18': {'cpuUser': '27.22', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '70.66'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '70618'}, '0': {'memPercent': 22, 'memFree': '77114'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000664812', 'lastCheck': '0.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000702948', 'lastCheck': '0.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00222809', 'lastCheck': '6.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000535909', 'lastCheck': '0.5', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168846635', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '5091486554', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590439829851', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '2705580806466', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433795930994', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '259019536991', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239861679', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '366091866', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122790697', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121312194', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423641920158', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '253599950515', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215269993', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680360676', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '29546850942', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652288562', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '29289330177', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255788230149', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '7054392476', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121686747', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75282210693', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '61807907067', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122245677', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418921405', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '1012772949', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121498124', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183273.891123, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183273.891123, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183273.891123, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183273.891123, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121268155', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458734331643', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183273.891123, 'rx': '2576941144346', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322100168282', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183273.891123, 'rx': '1322100168282', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183273.891123, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940828521', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '2700629919227', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571499001330', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '4950887239', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329517074430', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183273.891123, 'rx': '980008583497', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183273.891123, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183273.891123, 'rx': '45351354', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183273.891123, 'rx': '39831448', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1728021.99', 'cpuLoad': '8.50', 'cpuSys': '2.43', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.24', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150390, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:21:13 GMT', 'cpuUser': '36.45', 'memFree': 150134, 'cpuIdle': '61.11', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.19'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:21:13,912+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.08 seconds (__init__:312) 2019-04-13 21:21:16,063+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=d9fe4f45-d769-4663-b62e-732603c38e6d (api:48) 2019-04-13 21:21:16,068+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=d9fe4f45-d769-4663-b62e-732603c38e6d (api:54) 2019-04-13 21:21:16,069+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:21:16,088+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=e72a0682-a565-486a-b6ff-76474e468b0a (api:48) 2019-04-13 21:21:16,097+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707162853376', 'isoprefix': '', 'alerts': [], 'disktotal': '3898260586496', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707162853376', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719911440384', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707162853376', 'isoprefix': '', 'alerts': [], 'disktotal': '4364551847936', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=e72a0682-a565-486a-b6ff-76474e468b0a (api:54) 2019-04-13 21:21:16,099+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:21:16,567+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:21:16,609+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:21:16,610+0200 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:21:16,611+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:21:17,105+0200 WARN (qgapoller/5) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:21:21,022+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:21:21,054+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:21:21,055+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:21:21,056+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:21:23,431+0200 INFO (itmap/0) [IOProcessClient] (/onode-1.discovery.intern:_mnt_store) Starting client (__init__:308) 2019-04-13 21:21:23,464+0200 INFO (itmap/1) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_vms) Starting client (__init__:308) 2019-04-13 21:21:23,471+0200 INFO (ioprocess/28131) [IOProcess] (/onode-1.discovery.intern:_mnt_store) Starting ioprocess (__init__:434) 2019-04-13 21:21:23,496+0200 INFO (itmap/2) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_isos) Starting client (__init__:308) 2019-04-13 21:21:23,501+0200 INFO (ioprocess/28148) [IOProcess] (/nas.discovery.intern:_mnt_d2600_vms) Starting ioprocess (__init__:434) 2019-04-13 21:21:23,529+0200 INFO (itmap/3) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting client (__init__:308) 2019-04-13 21:21:23,551+0200 INFO (ioprocess/28157) [IOProcess] (/nas.discovery.intern:_mnt_d2600_isos) Starting ioprocess (__init__:434) 2019-04-13 21:21:23,559+0200 INFO (itmap/4) [IOProcessClient] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting client (__init__:308) 2019-04-13 21:21:23,564+0200 INFO (ioprocess/28162) [IOProcess] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting ioprocess (__init__:434) 2019-04-13 21:21:23,593+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:21:23,594+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:21:23,595+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:21:23,593+0200 INFO (ioprocess/28175) [IOProcess] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting ioprocess (__init__:434) 2019-04-13 21:21:26,146+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=afca8afa-4f45-4b01-ac45-5cb10823e355 (api:48) 2019-04-13 21:21:26,152+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=afca8afa-4f45-4b01-ac45-5cb10823e355 (api:54) 2019-04-13 21:21:26,152+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:21:26,166+0200 INFO (jsonrpc/0) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=736eb542-8367-4179-94c0-27e440396d94 (api:48) 2019-04-13 21:21:26,176+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707160756224', 'isoprefix': '', 'alerts': [], 'disktotal': '3898257440768', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707160756224', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719908294656', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707160756224', 'isoprefix': '', 'alerts': [], 'disktotal': '4364548702208', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=736eb542-8367-4179-94c0-27e440396d94 (api:54) 2019-04-13 21:21:26,178+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:21:27,122+0200 WARN (qgapoller/9) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:21:28,326+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=c9fa8cd6-ed58-42cd-bf54-444509f8e021 (api:48) 2019-04-13 21:21:28,326+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000744581', 'lastCheck': '4.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000643504', 'lastCheck': '4.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000779334', 'lastCheck': '4.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000338006', 'lastCheck': '1.2', 'valid': True}} from=internal, task_id=c9fa8cd6-ed58-42cd-bf54-444509f8e021 (api:54) 2019-04-13 21:21:28,877+0200 INFO (jsonrpc/6) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:21:28,944+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=03c2b583-496f-4b00-a84e-d2a932402b06 (api:48) 2019-04-13 21:21:28,945+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000744581', 'lastCheck': '4.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000643504', 'lastCheck': '4.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000779334', 'lastCheck': '4.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000338006', 'lastCheck': '1.8', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=03c2b583-496f-4b00-a84e-d2a932402b06 (api:54) 2019-04-13 21:21:28,945+0200 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=3dc63bc0-3182-4831-ad20-bb65e51fdb14 (api:48) 2019-04-13 21:21:28,946+0200 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=3dc63bc0-3182-4831-ad20-bb65e51fdb14 (api:54) 2019-04-13 21:21:28,958+0200 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '28.48', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '69.33'}, '21': {'cpuUser': '19.87', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '77.88'}, '22': {'cpuUser': '25.37', 'nodeIndex': 1, 'cpuSys': '1.85', 'cpuIdle': '72.78'}, '23': {'cpuUser': '22.65', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '74.90'}, '1': {'cpuUser': '30.40', 'nodeIndex': 0, 'cpuSys': '4.97', 'cpuIdle': '64.63'}, '0': {'cpuUser': '46.30', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '51.45'}, '3': {'cpuUser': '52.79', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '45.62'}, '2': {'cpuUser': '45.44', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '52.64'}, '5': {'cpuUser': '52.66', 'nodeIndex': 0, 'cpuSys': '1.52', 'cpuIdle': '45.82'}, '4': {'cpuUser': '36.63', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '61.45'}, '7': {'cpuUser': '30.20', 'nodeIndex': 1, 'cpuSys': '3.58', 'cpuIdle': '66.22'}, '6': {'cpuUser': '36.23', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '60.79'}, '9': {'cpuUser': '34.97', 'nodeIndex': 1, 'cpuSys': '3.25', 'cpuIdle': '61.78'}, '8': {'cpuUser': '33.91', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '63.31'}, '11': {'cpuUser': '32.59', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '64.30'}, '10': {'cpuUser': '30.40', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '66.49'}, '13': {'cpuUser': '50.74', 'nodeIndex': 0, 'cpuSys': '1.19', 'cpuIdle': '48.07'}, '12': {'cpuUser': '13.31', 'nodeIndex': 0, 'cpuSys': '2.78', 'cpuIdle': '83.91'}, '15': {'cpuUser': '30.93', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '67.15'}, '14': {'cpuUser': '32.26', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '65.95'}, '17': {'cpuUser': '20.33', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '77.42'}, '16': {'cpuUser': '49.88', 'nodeIndex': 0, 'cpuSys': '1.46', 'cpuIdle': '48.66'}, '19': {'cpuUser': '23.51', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '74.17'}, '18': {'cpuUser': '20.27', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '77.48'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '70593'}, '0': {'memPercent': 22, 'memFree': '77131'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000744581', 'lastCheck': '4.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000779334', 'lastCheck': '4.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000338006', 'lastCheck': '1.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000643504', 'lastCheck': '4.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168846961', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '5091486872', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590440069515', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '2705580843874', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433799448622', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '259021742582', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239861789', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '366091866', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122790807', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121312304', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423645515924', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '253602229557', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215270103', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680360786', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '29546850942', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652288672', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '29289330177', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255790490684', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '7058154716', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121686857', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75284711257', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '61810822563', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122245787', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418922075', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '1012773099', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121498234', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183288.943361, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183288.943361, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183288.943361, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183288.943361, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121268265', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458734584133', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183288.943361, 'rx': '2576941176704', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322111603989', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183288.943361, 'rx': '1322111603989', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183288.943361, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940828979', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '2700629956079', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571499240536', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '4950887795', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329526309925', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183288.943361, 'rx': '980018503287', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183288.943361, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183288.943361, 'rx': '45351744', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183288.943361, 'rx': '39831816', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40566', 'ksmPages': 100, 'elapsedTime': '1728037.04', 'cpuLoad': '8.55', 'cpuSys': '2.41', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.51', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150382, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:21:28 GMT', 'cpuUser': '33.34', 'memFree': 150126, 'cpuIdle': '64.25', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.85'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:21:28,964+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:21:31,648+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:21:31,683+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:21:31,683+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:21:31,684+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:21:36,063+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:21:36,093+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:21:36,094+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:21:36,095+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:21:36,218+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=b198a772-6460-4bd2-83fb-50355358ece2 (api:48) 2019-04-13 21:21:36,224+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=b198a772-6460-4bd2-83fb-50355358ece2 (api:54) 2019-04-13 21:21:36,225+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:21:36,240+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=e9630d05-87a4-4779-9de7-37b955800791 (api:48) 2019-04-13 21:21:36,248+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707160756224', 'isoprefix': '', 'alerts': [], 'disktotal': '3898258489344', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707160756224', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719909343232', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707160756224', 'isoprefix': '', 'alerts': [], 'disktotal': '4364548702208', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=e9630d05-87a4-4779-9de7-37b955800791 (api:54) 2019-04-13 21:21:36,249+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:21:37,110+0200 WARN (qgapoller/10) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:21:43,427+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=2046ed65-35ec-4482-9fdd-a37e6b5e0d5c (api:48) 2019-04-13 21:21:43,427+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000751711', 'lastCheck': '9.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000612742', 'lastCheck': '9.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000681904', 'lastCheck': '9.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000394733', 'lastCheck': '6.3', 'valid': True}} from=internal, task_id=2046ed65-35ec-4482-9fdd-a37e6b5e0d5c (api:54) 2019-04-13 21:21:43,953+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:21:44,021+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=d63ee8af-8f89-4790-99d0-8d72e3bae270 (api:48) 2019-04-13 21:21:44,021+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000751711', 'lastCheck': '0.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000612742', 'lastCheck': '0.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000681904', 'lastCheck': '0.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000394733', 'lastCheck': '6.9', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=d63ee8af-8f89-4790-99d0-8d72e3bae270 (api:54) 2019-04-13 21:21:44,022+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=c089b596-2142-4496-9a5f-7e0de8e84fec (api:48) 2019-04-13 21:21:44,022+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=c089b596-2142-4496-9a5f-7e0de8e84fec (api:54) 2019-04-13 21:21:44,036+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '27.13', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '70.55'}, '21': {'cpuUser': '17.31', 'nodeIndex': 1, 'cpuSys': '2.79', 'cpuIdle': '79.90'}, '22': {'cpuUser': '27.20', 'nodeIndex': 1, 'cpuSys': '1.92', 'cpuIdle': '70.88'}, '23': {'cpuUser': '25.47', 'nodeIndex': 1, 'cpuSys': '1.99', 'cpuIdle': '72.54'}, '1': {'cpuUser': '36.42', 'nodeIndex': 0, 'cpuSys': '4.18', 'cpuIdle': '59.40'}, '0': {'cpuUser': '43.85', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '53.96'}, '3': {'cpuUser': '45.77', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '52.24'}, '2': {'cpuUser': '46.03', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '52.11'}, '5': {'cpuUser': '47.76', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '50.52'}, '4': {'cpuUser': '42.32', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '55.96'}, '7': {'cpuUser': '32.77', 'nodeIndex': 1, 'cpuSys': '3.45', 'cpuIdle': '63.78'}, '6': {'cpuUser': '39.47', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '57.88'}, '9': {'cpuUser': '41.06', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '56.09'}, '8': {'cpuUser': '34.69', 'nodeIndex': 1, 'cpuSys': '2.92', 'cpuIdle': '62.39'}, '11': {'cpuUser': '37.41', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '59.54'}, '10': {'cpuUser': '33.37', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '63.58'}, '13': {'cpuUser': '27.46', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '70.95'}, '12': {'cpuUser': '26.86', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '71.28'}, '15': {'cpuUser': '23.68', 'nodeIndex': 0, 'cpuSys': '2.85', 'cpuIdle': '73.47'}, '14': {'cpuUser': '22.88', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '75.20'}, '17': {'cpuUser': '17.11', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '81.10'}, '16': {'cpuUser': '28.12', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '70.09'}, '19': {'cpuUser': '16.58', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '80.77'}, '18': {'cpuUser': '17.05', 'nodeIndex': 1, 'cpuSys': '2.26', 'cpuIdle': '80.69'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '70577'}, '0': {'memPercent': 22, 'memFree': '77149'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000751711', 'lastCheck': '0.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000681904', 'lastCheck': '0.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000394733', 'lastCheck': '6.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000612742', 'lastCheck': '0.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168847479', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '5091486956', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590440555573', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '2705582050982', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433803153018', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '259024277030', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239862187', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '366091866', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122791205', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121312702', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423649233142', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '253604670109', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215270501', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680361184', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '29546850942', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652289296', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '29289330695', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255793029812', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '7060009067', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121687255', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75286868159', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '61816886739', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122246185', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418922495', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '1012773099', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121498632', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183304.020013, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183304.020013, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183304.020013, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183304.020013, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121268663', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458735093073', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183304.020013, 'rx': '2576942333258', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322122998564', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183304.020013, 'rx': '1322122998564', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183304.020013, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940829065', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '2700631162397', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571499726508', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '4950888585', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329536978640', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183304.020013, 'rx': '980028619130', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183304.020013, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183304.020013, 'rx': '45352134', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183304.020013, 'rx': '39832138', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40566', 'ksmPages': 100, 'elapsedTime': '1728052.11', 'cpuLoad': '8.45', 'cpuSys': '2.39', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '2.85', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150383, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:21:44 GMT', 'cpuUser': '31.56', 'memFree': 150127, 'cpuIdle': '66.05', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '0.99'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:21:44,043+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:21:46,298+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=12c09943-3903-41ac-8e41-ff31e1af3684 (api:48) 2019-04-13 21:21:46,303+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=12c09943-3903-41ac-8e41-ff31e1af3684 (api:54) 2019-04-13 21:21:46,304+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:21:46,427+0200 INFO (jsonrpc/0) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=23b81534-ec2f-46c9-bded-d2e8e8df2660 (api:48) 2019-04-13 21:21:46,435+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707157610496', 'isoprefix': '', 'alerts': [], 'disktotal': '3898255343616', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707157610496', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719906197504', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707157610496', 'isoprefix': '', 'alerts': [], 'disktotal': '4364546605056', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=23b81534-ec2f-46c9-bded-d2e8e8df2660 (api:54) 2019-04-13 21:21:46,436+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:21:46,728+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:21:46,767+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:21:46,768+0200 ERROR (jsonrpc/6) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:21:46,768+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:21:47,107+0200 WARN (qgapoller/9) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:21:51,102+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:21:51,132+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:21:51,133+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:21:51,134+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:21:56,486+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=fb0ebeba-f1c9-4091-b1af-f13de18c274d (api:48) 2019-04-13 21:21:56,492+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=fb0ebeba-f1c9-4091-b1af-f13de18c274d (api:54) 2019-04-13 21:21:56,493+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:21:56,608+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=4aa39376-050e-461b-b5ba-1313e7d8a59a (api:48) 2019-04-13 21:21:56,617+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707154464768', 'isoprefix': '', 'alerts': [], 'disktotal': '3898252197888', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707154464768', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719903051776', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707154464768', 'isoprefix': '', 'alerts': [], 'disktotal': '4364543459328', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=4aa39376-050e-461b-b5ba-1313e7d8a59a (api:54) 2019-04-13 21:21:56,619+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:21:58,520+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=7fc3f57e-995a-408b-b6b1-83da4a133802 (api:48) 2019-04-13 21:21:58,521+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000610593', 'lastCheck': '4.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000626775', 'lastCheck': '4.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000690793', 'lastCheck': '4.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00034169', 'lastCheck': '1.3', 'valid': True}} from=internal, task_id=7fc3f57e-995a-408b-b6b1-83da4a133802 (api:54) 2019-04-13 21:21:59,030+0200 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:21:59,115+0200 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=dd12331a-c3d8-44d9-b139-8fa2d752ecd8 (api:48) 2019-04-13 21:21:59,116+0200 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000610593', 'lastCheck': '4.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000626775', 'lastCheck': '4.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000690793', 'lastCheck': '4.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00034169', 'lastCheck': '1.9', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=dd12331a-c3d8-44d9-b139-8fa2d752ecd8 (api:54) 2019-04-13 21:21:59,117+0200 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=57ce1c7f-72bf-4b68-83a9-a1f3ead5b358 (api:48) 2019-04-13 21:21:59,117+0200 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=57ce1c7f-72bf-4b68-83a9-a1f3ead5b358 (api:54) 2019-04-13 21:21:59,130+0200 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '20.86', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '76.95'}, '21': {'cpuUser': '17.22', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '80.06'}, '22': {'cpuUser': '26.03', 'nodeIndex': 1, 'cpuSys': '1.99', 'cpuIdle': '71.98'}, '23': {'cpuUser': '21.19', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '76.69'}, '1': {'cpuUser': '33.24', 'nodeIndex': 0, 'cpuSys': '5.17', 'cpuIdle': '61.59'}, '0': {'cpuUser': '39.60', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '58.08'}, '3': {'cpuUser': '42.65', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '55.16'}, '2': {'cpuUser': '40.60', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '57.08'}, '5': {'cpuUser': '48.15', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '50.26'}, '4': {'cpuUser': '43.64', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '54.37'}, '7': {'cpuUser': '35.96', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '61.26'}, '6': {'cpuUser': '35.50', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '61.65'}, '9': {'cpuUser': '38.41', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '58.87'}, '8': {'cpuUser': '31.52', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '65.50'}, '11': {'cpuUser': '37.28', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '60.14'}, '10': {'cpuUser': '32.65', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '64.50'}, '13': {'cpuUser': '20.66', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '77.35'}, '12': {'cpuUser': '22.05', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '75.83'}, '15': {'cpuUser': '23.58', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '74.50'}, '14': {'cpuUser': '25.89', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '72.06'}, '17': {'cpuUser': '21.39', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '76.89'}, '16': {'cpuUser': '21.19', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '77.02'}, '19': {'cpuUser': '19.27', 'nodeIndex': 1, 'cpuSys': '2.38', 'cpuIdle': '78.35'}, '18': {'cpuUser': '15.50', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '82.31'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '70552'}, '0': {'memPercent': 22, 'memFree': '77172'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000610593', 'lastCheck': '4.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000690793', 'lastCheck': '4.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00034169', 'lastCheck': '1.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000626775', 'lastCheck': '4.8', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168847606', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '5091486956', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590441219331', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '2705583616658', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433807417266', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '259026916533', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239862314', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '366091866', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122791332', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121312829', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423653014270', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '253607189432', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215270628', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680361311', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '29546850942', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652294358', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '29289332965', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255795708740', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '7062048458', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121687382', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75288888541', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '61821373389', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122246312', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418922975', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '1012773099', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121498759', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183319.114699, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183319.114699, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183319.114699, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183319.114699, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121268790', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458735667197', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183319.114699, 'rx': '2576943823165', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322135480789', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183319.114699, 'rx': '1322135480789', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183319.114699, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940831475', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '2700632727855', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571500387856', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '4950888803', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329546309963', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183319.114699, 'rx': '980038981645', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183319.114699, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183319.114699, 'rx': '45352524', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183319.114699, 'rx': '39832506', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40566', 'ksmPages': 100, 'elapsedTime': '1728067.19', 'cpuLoad': '8.36', 'cpuSys': '2.41', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.18', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150382, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:21:59 GMT', 'cpuUser': '29.76', 'memFree': 150126, 'cpuIdle': '67.83', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.59'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:21:59,136+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.11 seconds (__init__:312) 2019-04-13 21:22:01,809+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:22:01,841+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:22:01,842+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:22:01,843+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:22:04,112+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID=u'ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', options=None) from=internal, task_id=708b9562-11bf-4ead-9bf3-ae52ce32a491 (api:48) 2019-04-13 21:22:04,115+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', options=None) from=internal, task_id=ef6d924a-8b5c-49b1-90c8-50a92f08b993 (api:48) 2019-04-13 21:22:04,117+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', options=None) from=internal, task_id=d24ec8d9-9ecc-4670-827d-9c005e78f7b8 (api:48) 2019-04-13 21:22:04,121+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '4579690496', 'apparentsize': '16106127360'} from=internal, task_id=ef6d924a-8b5c-49b1-90c8-50a92f08b993 (api:54) 2019-04-13 21:22:04,122+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID='ef6741d1-380a-4736-a924-f88ed68e1270', options=None) from=internal, task_id=3d7d097a-ed4b-4f4a-9556-53052164d5e2 (api:48) 2019-04-13 21:22:04,125+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '5697701376', 'apparentsize': '16106127360'} from=internal, task_id=d24ec8d9-9ecc-4670-827d-9c005e78f7b8 (api:54) 2019-04-13 21:22:04,126+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID=u'67a4dd71-69ff-49a8-b8a4-c7101c75e105', options=None) from=internal, task_id=f5766a66-8501-43fb-ae2a-965c63d19554 (api:48) 2019-04-13 21:22:04,129+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '53760947712', 'apparentsize': '139586437120'} from=internal, task_id=708b9562-11bf-4ead-9bf3-ae52ce32a491 (api:54) 2019-04-13 21:22:04,130+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', options=None) from=internal, task_id=13594909-8fbb-4a40-b496-c9d0e567111e (api:48) 2019-04-13 21:22:04,132+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '6477472768', 'apparentsize': '21474836480'} from=internal, task_id=3d7d097a-ed4b-4f4a-9556-53052164d5e2 (api:54) 2019-04-13 21:22:04,133+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID=u'2b136ed7-8b9a-45c1-9edc-64ee96c87c43', options=None) from=internal, task_id=1eb6d926-d7de-4eab-b316-61342140e8a3 (api:48) 2019-04-13 21:22:04,135+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID=u'44b3ceca-f0db-43da-af29-aa681df146ac', options=None) from=internal, task_id=7f7e322a-2239-426e-9545-53ad8783b5ea (api:48) 2019-04-13 21:22:04,138+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '55059115520', 'apparentsize': '171798691840'} from=internal, task_id=f5766a66-8501-43fb-ae2a-965c63d19554 (api:54) 2019-04-13 21:22:04,138+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=9cd93acf-ae7f-4c88-869d-dc6cdaf5232e (api:48) 2019-04-13 21:22:04,142+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2524573696', 'apparentsize': '12884901888'} from=internal, task_id=13594909-8fbb-4a40-b496-c9d0e567111e (api:54) 2019-04-13 21:22:04,143+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID='123e2cc8-8f24-4ddd-b401-68393535f555', options=None) from=internal, task_id=794a6200-6247-40ac-b269-1fc10465ae94 (api:48) 2019-04-13 21:22:04,145+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2024405504', 'apparentsize': '10737418240'} from=internal, task_id=1eb6d926-d7de-4eab-b316-61342140e8a3 (api:54) 2019-04-13 21:22:04,145+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID='3f6c31c6-ec38-4979-a617-55537123ee61', options=None) from=internal, task_id=f7467641-5e49-4183-98c2-396ce4e44c8b (api:48) 2019-04-13 21:22:04,148+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '11196694528', 'apparentsize': '21474836480'} from=internal, task_id=7f7e322a-2239-426e-9545-53ad8783b5ea (api:54) 2019-04-13 21:22:04,149+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', options=None) from=internal, task_id=5dd17d7f-663b-4524-94ed-b28c26e79614 (api:48) 2019-04-13 21:22:04,150+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2116481024', 'apparentsize': '3948150784'} from=internal, task_id=9cd93acf-ae7f-4c88-869d-dc6cdaf5232e (api:54) 2019-04-13 21:22:04,151+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID=u'a1d583c1-e135-4e3f-b6fa-0f29a5261366', options=None) from=internal, task_id=5016d85e-6317-4568-889f-bce5fc6915e7 (api:48) 2019-04-13 21:22:04,154+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2283003904', 'apparentsize': '16106127360'} from=internal, task_id=794a6200-6247-40ac-b269-1fc10465ae94 (api:54) 2019-04-13 21:22:04,155+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', options=None) from=internal, task_id=e3865a97-e49e-4f8c-95f5-e899b7d72346 (api:48) 2019-04-13 21:22:04,159+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2109410304', 'apparentsize': '8589934592'} from=internal, task_id=5dd17d7f-663b-4524-94ed-b28c26e79614 (api:54) 2019-04-13 21:22:04,160+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID='a884b4ce-381f-4653-be14-a31d15181f48', options=None) from=internal, task_id=163843fc-138d-4e9f-9dd2-1bd1c84e2313 (api:48) 2019-04-13 21:22:04,161+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '1294894592', 'apparentsize': '12884901888'} from=internal, task_id=5016d85e-6317-4568-889f-bce5fc6915e7 (api:54) 2019-04-13 21:22:04,161+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', options=None) from=internal, task_id=905bc657-cd74-4cc8-9dca-691967d5273c (api:48) 2019-04-13 21:22:04,162+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '4998722048', 'apparentsize': '16106127360'} from=internal, task_id=f7467641-5e49-4183-98c2-396ce4e44c8b (api:54) 2019-04-13 21:22:04,166+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '7263196672', 'apparentsize': '21474836480'} from=internal, task_id=e3865a97-e49e-4f8c-95f5-e899b7d72346 (api:54) 2019-04-13 21:22:04,168+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '3447286784', 'apparentsize': '16106127360'} from=internal, task_id=163843fc-138d-4e9f-9dd2-1bd1c84e2313 (api:54) 2019-04-13 21:22:04,169+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2879759360', 'apparentsize': '8589934592'} from=internal, task_id=905bc657-cd74-4cc8-9dca-691967d5273c (api:54) 2019-04-13 21:22:06,140+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:22:06,171+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:22:06,172+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:22:06,173+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:22:06,674+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=06dcbc1e-e982-49d2-9a68-0e743baa2ae5 (api:48) 2019-04-13 21:22:06,679+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=06dcbc1e-e982-49d2-9a68-0e743baa2ae5 (api:54) 2019-04-13 21:22:06,680+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:22:06,695+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=4197d163-97fa-42a5-b852-7895a33400a5 (api:48) 2019-04-13 21:22:06,702+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707150270464', 'isoprefix': '', 'alerts': [], 'disktotal': '3898248003584', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707150270464', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719898857472', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707150270464', 'isoprefix': '', 'alerts': [], 'disktotal': '4364538216448', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=4197d163-97fa-42a5-b852-7895a33400a5 (api:54) 2019-04-13 21:22:06,703+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:22:13,602+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=092edafa-6cfd-4667-9bc1-50724ff6a515 (api:48) 2019-04-13 21:22:13,603+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00053959', 'lastCheck': '9.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000462717', 'lastCheck': '9.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000699117', 'lastCheck': '9.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000345741', 'lastCheck': '6.4', 'valid': True}} from=internal, task_id=092edafa-6cfd-4667-9bc1-50724ff6a515 (api:54) 2019-04-13 21:22:14,077+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:22:14,148+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=daa88820-6130-4df5-a8d9-1a2bbb14caf5 (api:48) 2019-04-13 21:22:14,149+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00053959', 'lastCheck': '0.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000462717', 'lastCheck': '0.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000699117', 'lastCheck': '0.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000345741', 'lastCheck': '7.0', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=daa88820-6130-4df5-a8d9-1a2bbb14caf5 (api:54) 2019-04-13 21:22:14,150+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=b9b8c209-ba74-4c11-843a-1dc66edcc3b9 (api:48) 2019-04-13 21:22:14,151+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=b9b8c209-ba74-4c11-843a-1dc66edcc3b9 (api:54) 2019-04-13 21:22:14,168+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '15.69', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '82.12'}, '21': {'cpuUser': '21.06', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '76.49'}, '22': {'cpuUser': '22.38', 'nodeIndex': 1, 'cpuSys': '1.92', 'cpuIdle': '75.70'}, '23': {'cpuUser': '19.14', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '78.74'}, '1': {'cpuUser': '37.68', 'nodeIndex': 0, 'cpuSys': '4.17', 'cpuIdle': '58.15'}, '0': {'cpuUser': '38.07', 'nodeIndex': 0, 'cpuSys': '2.58', 'cpuIdle': '59.35'}, '3': {'cpuUser': '39.00', 'nodeIndex': 0, 'cpuSys': '2.71', 'cpuIdle': '58.29'}, '2': {'cpuUser': '36.88', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '60.80'}, '5': {'cpuUser': '43.04', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '54.84'}, '4': {'cpuUser': '39.20', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '58.48'}, '7': {'cpuUser': '27.68', 'nodeIndex': 1, 'cpuSys': '3.38', 'cpuIdle': '68.94'}, '6': {'cpuUser': '30.79', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '66.16'}, '9': {'cpuUser': '31.78', 'nodeIndex': 1, 'cpuSys': '2.91', 'cpuIdle': '65.31'}, '8': {'cpuUser': '27.55', 'nodeIndex': 1, 'cpuSys': '3.58', 'cpuIdle': '68.87'}, '11': {'cpuUser': '35.16', 'nodeIndex': 1, 'cpuSys': '2.91', 'cpuIdle': '61.93'}, '10': {'cpuUser': '29.66', 'nodeIndex': 1, 'cpuSys': '3.24', 'cpuIdle': '67.10'}, '13': {'cpuUser': '18.87', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '78.68'}, '12': {'cpuUser': '22.58', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '75.23'}, '15': {'cpuUser': '18.94', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '79.14'}, '14': {'cpuUser': '31.45', 'nodeIndex': 0, 'cpuSys': '1.52', 'cpuIdle': '67.03'}, '17': {'cpuUser': '29.73', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '68.48'}, '16': {'cpuUser': '28.08', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '69.93'}, '19': {'cpuUser': '25.69', 'nodeIndex': 1, 'cpuSys': '1.59', 'cpuIdle': '72.72'}, '18': {'cpuUser': '20.13', 'nodeIndex': 1, 'cpuSys': '1.99', 'cpuIdle': '77.88'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '70739'}, '0': {'memPercent': 22, 'memFree': '76987'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00053959', 'lastCheck': '0.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000699117', 'lastCheck': '0.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000345741', 'lastCheck': '7.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000462717', 'lastCheck': '0.5', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168847726', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '5091486956', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590442412513', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '2705584964086', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433811617050', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '259029381010', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239862770', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '366092264', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122791452', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121312949', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423657606322', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '253609805650', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215270748', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680361539', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '29546851039', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652294575', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '29289333073', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255798231875', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '7066014188', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121687502', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75291276699', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '61825860549', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122246432', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418923563', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '1012773273', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121498879', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183334.146175, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183334.146175, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183334.146175, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183334.146175, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121268910', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458736894811', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183334.146175, 'rx': '2576945106361', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322145850206', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183334.146175, 'rx': '1322145850206', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183334.146175, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940831475', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '2700634074675', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571501581038', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '4950889411', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329557717365', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183334.146175, 'rx': '980050541871', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183334.146175, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183334.146175, 'rx': '45352914', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183334.146175, 'rx': '39832874', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40566', 'ksmPages': 100, 'elapsedTime': '1728082.24', 'cpuLoad': '8.29', 'cpuSys': '2.47', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.24', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150373, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:22:14 GMT', 'cpuUser': '28.76', 'memFree': 150117, 'cpuIdle': '68.77', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.13'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:22:14,181+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.10 seconds (__init__:312) 2019-04-13 21:22:16,749+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=d5558664-f2f5-47a1-97fb-ea83903450ef (api:48) 2019-04-13 21:22:16,754+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=d5558664-f2f5-47a1-97fb-ea83903450ef (api:54) 2019-04-13 21:22:16,754+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:22:16,766+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=88bbf22f-4ec8-40e2-99f3-a3c116026088 (api:48) 2019-04-13 21:22:16,774+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707150270464', 'isoprefix': '', 'alerts': [], 'disktotal': '3898248003584', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707150270464', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719898857472', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707150270464', 'isoprefix': '', 'alerts': [], 'disktotal': '4364538216448', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=88bbf22f-4ec8-40e2-99f3-a3c116026088 (api:54) 2019-04-13 21:22:16,775+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:22:16,874+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:22:16,905+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:22:16,905+0200 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:22:16,906+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:22:21,180+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:22:21,211+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:22:21,212+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:22:21,213+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:22:23,624+0200 INFO (monitor/93f02ed) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_isos) Closing client (__init__:584) 2019-04-13 21:22:23,625+0200 INFO (monitor/93f02ed) [IOProcessClient] (/glusterSD/onode-2.discovery.intern:vol-node-2) Closing client (__init__:584) 2019-04-13 21:22:23,625+0200 INFO (monitor/93f02ed) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_vms) Closing client (__init__:584) 2019-04-13 21:22:23,625+0200 INFO (monitor/93f02ed) [IOProcessClient] (/onode-1.discovery.intern:_mnt_store) Closing client (__init__:584) 2019-04-13 21:22:23,625+0200 INFO (monitor/93f02ed) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Closing client (__init__:584) 2019-04-13 21:22:26,821+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=e38aed53-6587-4b13-84e0-7ea1382db7bd (api:48) 2019-04-13 21:22:26,826+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=e38aed53-6587-4b13-84e0-7ea1382db7bd (api:54) 2019-04-13 21:22:26,826+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:22:26,837+0200 INFO (jsonrpc/0) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=48ee4501-2ddd-44a0-8cda-72836e668d70 (api:48) 2019-04-13 21:22:26,846+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707146076160', 'isoprefix': '', 'alerts': [], 'disktotal': '3898242760704', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707146076160', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719893614592', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707146076160', 'isoprefix': '', 'alerts': [], 'disktotal': '4364534022144', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=48ee4501-2ddd-44a0-8cda-72836e668d70 (api:54) 2019-04-13 21:22:26,848+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:22:28,685+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=b09723bb-9f28-4e79-b7d1-541b2ee16b5b (api:48) 2019-04-13 21:22:28,686+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00070254', 'lastCheck': '4.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000548665', 'lastCheck': '4.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000501233', 'lastCheck': '4.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00040922', 'lastCheck': '1.5', 'valid': True}} from=internal, task_id=b09723bb-9f28-4e79-b7d1-541b2ee16b5b (api:54) 2019-04-13 21:22:29,127+0200 INFO (jsonrpc/6) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:22:29,201+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=206ca87d-94f0-4f74-9286-b1f734772479 (api:48) 2019-04-13 21:22:29,201+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00070254', 'lastCheck': '4.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000548665', 'lastCheck': '4.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000501233', 'lastCheck': '4.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00040922', 'lastCheck': '2.0', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=206ca87d-94f0-4f74-9286-b1f734772479 (api:54) 2019-04-13 21:22:29,202+0200 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=f052d0ae-6143-4852-8ce0-20019b9f79d3 (api:48) 2019-04-13 21:22:29,202+0200 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=f052d0ae-6143-4852-8ce0-20019b9f79d3 (api:54) 2019-04-13 21:22:29,216+0200 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '16.45', 'nodeIndex': 1, 'cpuSys': '2.92', 'cpuIdle': '80.63'}, '21': {'cpuUser': '24.08', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '73.73'}, '22': {'cpuUser': '22.35', 'nodeIndex': 1, 'cpuSys': '1.99', 'cpuIdle': '75.66'}, '23': {'cpuUser': '27.92', 'nodeIndex': 1, 'cpuSys': '1.86', 'cpuIdle': '70.22'}, '1': {'cpuUser': '36.48', 'nodeIndex': 0, 'cpuSys': '4.97', 'cpuIdle': '58.55'}, '0': {'cpuUser': '50.67', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '46.88'}, '3': {'cpuUser': '44.97', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '52.58'}, '2': {'cpuUser': '37.14', 'nodeIndex': 0, 'cpuSys': '2.92', 'cpuIdle': '59.94'}, '5': {'cpuUser': '53.46', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '44.62'}, '4': {'cpuUser': '48.68', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '49.33'}, '7': {'cpuUser': '33.96', 'nodeIndex': 1, 'cpuSys': '3.38', 'cpuIdle': '62.66'}, '6': {'cpuUser': '33.83', 'nodeIndex': 1, 'cpuSys': '3.38', 'cpuIdle': '62.79'}, '9': {'cpuUser': '30.58', 'nodeIndex': 1, 'cpuSys': '3.25', 'cpuIdle': '66.17'}, '8': {'cpuUser': '32.96', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '63.99'}, '11': {'cpuUser': '32.10', 'nodeIndex': 1, 'cpuSys': '2.92', 'cpuIdle': '64.98'}, '10': {'cpuUser': '36.81', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '60.47'}, '13': {'cpuUser': '27.92', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '69.96'}, '12': {'cpuUser': '19.83', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '78.18'}, '15': {'cpuUser': '24.14', 'nodeIndex': 0, 'cpuSys': '2.39', 'cpuIdle': '73.47'}, '14': {'cpuUser': '29.05', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '68.63'}, '17': {'cpuUser': '19.10', 'nodeIndex': 0, 'cpuSys': '2.26', 'cpuIdle': '78.64'}, '16': {'cpuUser': '22.22', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '75.92'}, '19': {'cpuUser': '19.23', 'nodeIndex': 1, 'cpuSys': '2.26', 'cpuIdle': '78.51'}, '18': {'cpuUser': '18.04', 'nodeIndex': 1, 'cpuSys': '2.06', 'cpuIdle': '79.90'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '71194'}, '0': {'memPercent': 23, 'memFree': '76539'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00070254', 'lastCheck': '4.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000501233', 'lastCheck': '4.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00040922', 'lastCheck': '2.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000548665', 'lastCheck': '4.9', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168848106', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '5091487274', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590443139005', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '2705585022422', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433815816904', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '259031924732', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239862934', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '366092264', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122791616', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121313113', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423661328910', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '253612362184', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215270912', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680361703', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '29546851039', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652294575', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '29289333237', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255800785732', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '7067880455', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121687666', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75293361693', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '61830348174', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122246596', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418924103', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '1012773357', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121499043', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183349.19991, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183349.19991, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183349.19991, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183349.19991, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121269074', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458737544169', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183349.19991, 'rx': '2576945158979', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322158311361', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183349.19991, 'rx': '1322158311361', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183349.19991, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940832019', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '2700634132793', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571502306986', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '4950889629', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329567086128', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183349.19991, 'rx': '980061610491', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183349.19991, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183349.19991, 'rx': '45353304', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183349.19991, 'rx': '39833196', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1728097.29', 'cpuLoad': '8.23', 'cpuSys': '2.57', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.25', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150391, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:22:29 GMT', 'cpuUser': '30.91', 'memFree': 150135, 'cpuIdle': '66.52', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.39'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:22:29,223+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:22:31,955+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:22:31,988+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:22:31,989+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:22:31,990+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:22:36,220+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:22:36,258+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:22:36,259+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:22:36,260+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:22:36,899+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=115c3381-f88a-4d6e-91a6-ae9e62a9b82e (api:48) 2019-04-13 21:22:36,904+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=115c3381-f88a-4d6e-91a6-ae9e62a9b82e (api:54) 2019-04-13 21:22:36,904+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:22:36,918+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=5125c01f-3ce1-49b4-b5c9-919c39577ea2 (api:48) 2019-04-13 21:22:36,927+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707140833280', 'isoprefix': '', 'alerts': [], 'disktotal': '3898238566400', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707140833280', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719889420288', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707140833280', 'isoprefix': '', 'alerts': [], 'disktotal': '4364529827840', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=5125c01f-3ce1-49b4-b5c9-919c39577ea2 (api:54) 2019-04-13 21:22:36,929+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:22:43,774+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=fa249478-bd6c-4ba9-9806-3aa157aff413 (api:48) 2019-04-13 21:22:43,775+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000696364', 'lastCheck': '0.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000551559', 'lastCheck': '0.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000596898', 'lastCheck': '0.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00038961', 'lastCheck': '6.6', 'valid': True}} from=internal, task_id=fa249478-bd6c-4ba9-9806-3aa157aff413 (api:54) 2019-04-13 21:22:44,184+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:22:44,290+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=9bb827d1-7297-4dcf-862c-1cf8cbe1829d (api:48) 2019-04-13 21:22:44,291+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000696364', 'lastCheck': '0.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000551559', 'lastCheck': '0.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000596898', 'lastCheck': '0.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00038961', 'lastCheck': '7.1', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=9bb827d1-7297-4dcf-862c-1cf8cbe1829d (api:54) 2019-04-13 21:22:44,292+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=c0829460-dedd-4638-a9d9-fd320be618ab (api:48) 2019-04-13 21:22:44,293+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=c0829460-dedd-4638-a9d9-fd320be618ab (api:54) 2019-04-13 21:22:44,319+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '21.28', 'nodeIndex': 1, 'cpuSys': '2.39', 'cpuIdle': '76.33'}, '21': {'cpuUser': '17.30', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '80.05'}, '22': {'cpuUser': '22.61', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '74.94'}, '23': {'cpuUser': '25.79', 'nodeIndex': 1, 'cpuSys': '1.86', 'cpuIdle': '72.35'}, '1': {'cpuUser': '45.41', 'nodeIndex': 0, 'cpuSys': '3.58', 'cpuIdle': '51.01'}, '0': {'cpuUser': '49.12', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '48.76'}, '3': {'cpuUser': '62.85', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '35.56'}, '2': {'cpuUser': '59.07', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '39.14'}, '5': {'cpuUser': '47.20', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '51.08'}, '4': {'cpuUser': '53.63', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '44.78'}, '7': {'cpuUser': '31.16', 'nodeIndex': 1, 'cpuSys': '3.51', 'cpuIdle': '65.33'}, '6': {'cpuUser': '32.02', 'nodeIndex': 1, 'cpuSys': '3.45', 'cpuIdle': '64.53'}, '9': {'cpuUser': '35.33', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '61.62'}, '8': {'cpuUser': '33.88', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '62.94'}, '11': {'cpuUser': '36.06', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '60.96'}, '10': {'cpuUser': '35.00', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '61.69'}, '13': {'cpuUser': '37.99', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '60.29'}, '12': {'cpuUser': '27.84', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '69.97'}, '15': {'cpuUser': '13.79', 'nodeIndex': 0, 'cpuSys': '2.72', 'cpuIdle': '83.49'}, '14': {'cpuUser': '18.03', 'nodeIndex': 0, 'cpuSys': '2.52', 'cpuIdle': '79.45'}, '17': {'cpuUser': '41.37', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '56.97'}, '16': {'cpuUser': '29.30', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '68.51'}, '19': {'cpuUser': '26.25', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '71.30'}, '18': {'cpuUser': '20.55', 'nodeIndex': 1, 'cpuSys': '2.39', 'cpuIdle': '77.06'}}, 'numaNodeMemFree': {'1': {'memPercent': 28, 'memFree': '70822'}, '0': {'memPercent': 22, 'memFree': '76911'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000696364', 'lastCheck': '0.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000596898', 'lastCheck': '0.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00038961', 'lastCheck': '7.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000551559', 'lastCheck': '0.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168848226', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '5091487358', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590443545475', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '2705586152536', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433819335924', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '259034730284', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239862934', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '366092264', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122791616', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121313113', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423665552676', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '253615108599', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215270912', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680361703', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '29546851039', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652294575', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '29289333237', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255803607008', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '7070060066', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121687666', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75295738457', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '61833260430', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122246596', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418924583', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '1012773357', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121499043', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183364.287276, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183364.287276, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183364.287276, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183364.287276, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121269074', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458737911013', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183364.287276, 'rx': '2576946238371', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322168679302', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183364.287276, 'rx': '1322168679302', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183364.287276, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940832019', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '2700635262427', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571502713456', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '4950890109', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329575163254', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183364.287276, 'rx': '980072087531', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183364.287276, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183364.287276, 'rx': '45353694', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183364.287276, 'rx': '39833564', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1728112.35', 'cpuLoad': '8.25', 'cpuSys': '2.46', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.05', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150376, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:22:44 GMT', 'cpuUser': '34.30', 'memFree': 150120, 'cpuIdle': '63.25', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.13'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:22:44,329+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.15 seconds (__init__:312) 2019-04-13 21:22:46,981+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=1e2e7391-c1d2-4952-8977-d23ba23340f1 (api:48) 2019-04-13 21:22:46,986+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=1e2e7391-c1d2-4952-8977-d23ba23340f1 (api:54) 2019-04-13 21:22:46,987+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:22:47,035+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:22:47,078+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:22:47,079+0200 ERROR (jsonrpc/0) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:22:47,080+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:22:47,120+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=8d43b59e-1cba-4ab9-a848-f2ae644f83db (api:48) 2019-04-13 21:22:47,130+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707138736128', 'isoprefix': '', 'alerts': [], 'disktotal': '3898235420672', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707138736128', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719886274560', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707138736128', 'isoprefix': '', 'alerts': [], 'disktotal': '4364526682112', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=8d43b59e-1cba-4ab9-a848-f2ae644f83db (api:54) 2019-04-13 21:22:47,131+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:22:51,268+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:22:51,309+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:22:51,309+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:22:51,311+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:22:57,191+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=ace7d27e-e40c-4bfa-a0da-e5cba433da43 (api:48) 2019-04-13 21:22:57,200+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=ace7d27e-e40c-4bfa-a0da-e5cba433da43 (api:54) 2019-04-13 21:22:57,200+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:22:57,215+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=67f01b50-a656-49e7-a90d-6a9ea3edec39 (api:48) 2019-04-13 21:22:57,224+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707138736128', 'isoprefix': '', 'alerts': [], 'disktotal': '3898235420672', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707138736128', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719886274560', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707138736128', 'isoprefix': '', 'alerts': [], 'disktotal': '4364526682112', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=67f01b50-a656-49e7-a90d-6a9ea3edec39 (api:54) 2019-04-13 21:22:57,225+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:22:58,854+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=53692112-0a41-4031-8db7-22d1f192f48d (api:48) 2019-04-13 21:22:58,855+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000795499', 'lastCheck': '4.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000683293', 'lastCheck': '4.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000633864', 'lastCheck': '4.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000361604', 'lastCheck': '1.6', 'valid': True}} from=internal, task_id=53692112-0a41-4031-8db7-22d1f192f48d (api:54) 2019-04-13 21:22:59,225+0200 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:22:59,287+0200 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=f51b81bc-0580-4818-b3d0-681f3608ae88 (api:48) 2019-04-13 21:22:59,287+0200 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000795499', 'lastCheck': '5.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000683293', 'lastCheck': '5.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000633864', 'lastCheck': '5.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000361604', 'lastCheck': '2.1', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=f51b81bc-0580-4818-b3d0-681f3608ae88 (api:54) 2019-04-13 21:22:59,288+0200 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=d73794a8-08d9-4f1e-b7d5-b6c2e1796738 (api:48) 2019-04-13 21:22:59,288+0200 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=d73794a8-08d9-4f1e-b7d5-b6c2e1796738 (api:54) 2019-04-13 21:22:59,302+0200 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '21.01', 'nodeIndex': 1, 'cpuSys': '2.39', 'cpuIdle': '76.60'}, '21': {'cpuUser': '14.98', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '82.50'}, '22': {'cpuUser': '27.23', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '70.65'}, '23': {'cpuUser': '24.38', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '73.43'}, '1': {'cpuUser': '40.29', 'nodeIndex': 0, 'cpuSys': '4.44', 'cpuIdle': '55.27'}, '0': {'cpuUser': '42.67', 'nodeIndex': 0, 'cpuSys': '2.39', 'cpuIdle': '54.94'}, '3': {'cpuUser': '47.97', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '50.17'}, '2': {'cpuUser': '49.70', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '48.18'}, '5': {'cpuUser': '41.48', 'nodeIndex': 0, 'cpuSys': '2.65', 'cpuIdle': '55.87'}, '4': {'cpuUser': '46.85', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '51.16'}, '7': {'cpuUser': '34.99', 'nodeIndex': 1, 'cpuSys': '3.45', 'cpuIdle': '61.56'}, '6': {'cpuUser': '37.31', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '59.64'}, '9': {'cpuUser': '37.84', 'nodeIndex': 1, 'cpuSys': '3.51', 'cpuIdle': '58.65'}, '8': {'cpuUser': '33.99', 'nodeIndex': 1, 'cpuSys': '3.51', 'cpuIdle': '62.50'}, '11': {'cpuUser': '35.38', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '61.51'}, '10': {'cpuUser': '33.93', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '62.89'}, '13': {'cpuUser': '37.44', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '60.90'}, '12': {'cpuUser': '25.18', 'nodeIndex': 0, 'cpuSys': '1.46', 'cpuIdle': '73.36'}, '15': {'cpuUser': '24.25', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '73.70'}, '14': {'cpuUser': '12.66', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '84.89'}, '17': {'cpuUser': '29.09', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '68.99'}, '16': {'cpuUser': '29.22', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '68.92'}, '19': {'cpuUser': '24.98', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '72.90'}, '18': {'cpuUser': '22.26', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '75.09'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '70506'}, '0': {'memPercent': 22, 'memFree': '77228'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000795499', 'lastCheck': '5.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000633864', 'lastCheck': '5.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000361604', 'lastCheck': '2.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000683293', 'lastCheck': '5.0', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168848390', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '5091487358', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590444038391', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '2705587693086', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433822605694', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '259037078411', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239863098', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '366092264', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122791780', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121313277', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423668841340', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '253617697678', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215271076', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680361867', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '29546851039', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652294755', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '29289333593', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255806037540', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '7074336637', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121687830', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75297697017', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '61837744905', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122246760', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418925003', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '1012773357', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121499207', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183379.285724, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183379.285724, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183379.285724, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183379.285724, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121269238', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458738385803', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183379.285724, 'rx': '2576947712389', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322182188453', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183379.285724, 'rx': '1322182188453', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183379.285724, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940832361', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '2700636802977', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571503206030', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '4950890109', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329586850175', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183379.285724, 'rx': '980081088833', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183379.285724, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183379.285724, 'rx': '45354084', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183379.285724, 'rx': '39833886', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1728127.39', 'cpuLoad': '8.26', 'cpuSys': '2.53', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.78', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150390, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:22:59 GMT', 'cpuUser': '32.29', 'memFree': 150134, 'cpuIdle': '65.18', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.59'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:22:59,308+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.08 seconds (__init__:312) 2019-04-13 21:23:02,110+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:23:02,141+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:23:02,141+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:23:02,142+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:23:04,106+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID=u'ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', options=None) from=internal, task_id=a64dcec1-6e65-4cb4-862e-081989991037 (api:48) 2019-04-13 21:23:04,107+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', options=None) from=internal, task_id=0c54130c-ee5e-48da-9af0-ede7b9c84554 (api:48) 2019-04-13 21:23:04,109+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', options=None) from=internal, task_id=c9566fa6-d19c-4b44-b870-37cc5f8c8987 (api:48) 2019-04-13 21:23:04,111+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '4579696128', 'apparentsize': '16106127360'} from=internal, task_id=0c54130c-ee5e-48da-9af0-ede7b9c84554 (api:54) 2019-04-13 21:23:04,112+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID='ef6741d1-380a-4736-a924-f88ed68e1270', options=None) from=internal, task_id=c5c51e1d-b07b-4a91-affe-dc720992f83e (api:48) 2019-04-13 21:23:04,114+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '5697701376', 'apparentsize': '16106127360'} from=internal, task_id=c9566fa6-d19c-4b44-b870-37cc5f8c8987 (api:54) 2019-04-13 21:23:04,114+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID=u'67a4dd71-69ff-49a8-b8a4-c7101c75e105', options=None) from=internal, task_id=d6fb68ff-2b43-4095-b7a2-107f8a58248b (api:48) 2019-04-13 21:23:04,115+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '53760936448', 'apparentsize': '139586437120'} from=internal, task_id=a64dcec1-6e65-4cb4-862e-081989991037 (api:54) 2019-04-13 21:23:04,116+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', options=None) from=internal, task_id=9869e0f5-158e-4e49-9b62-e9bd1c21bdfc (api:48) 2019-04-13 21:23:04,119+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '6477472768', 'apparentsize': '21474836480'} from=internal, task_id=c5c51e1d-b07b-4a91-affe-dc720992f83e (api:54) 2019-04-13 21:23:04,120+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID=u'2b136ed7-8b9a-45c1-9edc-64ee96c87c43', options=None) from=internal, task_id=348223fa-98bd-46d3-9fad-21738c80d905 (api:48) 2019-04-13 21:23:04,121+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID=u'44b3ceca-f0db-43da-af29-aa681df146ac', options=None) from=internal, task_id=fcf39844-04da-4405-bade-ed3dfb2575d9 (api:48) 2019-04-13 21:23:04,123+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '55059120640', 'apparentsize': '171798691840'} from=internal, task_id=d6fb68ff-2b43-4095-b7a2-107f8a58248b (api:54) 2019-04-13 21:23:04,123+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=7dd007f6-830a-47db-82a1-de8967e55339 (api:48) 2019-04-13 21:23:04,124+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2524573696', 'apparentsize': '12884901888'} from=internal, task_id=9869e0f5-158e-4e49-9b62-e9bd1c21bdfc (api:54) 2019-04-13 21:23:04,124+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID='123e2cc8-8f24-4ddd-b401-68393535f555', options=None) from=internal, task_id=02087862-50a5-4856-8f08-fea097e77ec8 (api:48) 2019-04-13 21:23:04,127+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2024405504', 'apparentsize': '10737418240'} from=internal, task_id=348223fa-98bd-46d3-9fad-21738c80d905 (api:54) 2019-04-13 21:23:04,128+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID='3f6c31c6-ec38-4979-a617-55537123ee61', options=None) from=internal, task_id=d5b2f98a-9e7c-4d3c-9ebe-92d4f9daef97 (api:48) 2019-04-13 21:23:04,130+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '11196694528', 'apparentsize': '21474836480'} from=internal, task_id=fcf39844-04da-4405-bade-ed3dfb2575d9 (api:54) 2019-04-13 21:23:04,130+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', options=None) from=internal, task_id=be8abc8f-0879-4c2d-9e21-497d98df92ef (api:48) 2019-04-13 21:23:04,131+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2116486656', 'apparentsize': '3948150784'} from=internal, task_id=7dd007f6-830a-47db-82a1-de8967e55339 (api:54) 2019-04-13 21:23:04,132+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID=u'a1d583c1-e135-4e3f-b6fa-0f29a5261366', options=None) from=internal, task_id=335f9992-4834-4168-983e-8ee627673978 (api:48) 2019-04-13 21:23:04,133+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2283003904', 'apparentsize': '16106127360'} from=internal, task_id=02087862-50a5-4856-8f08-fea097e77ec8 (api:54) 2019-04-13 21:23:04,133+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', options=None) from=internal, task_id=9648a214-7492-4eca-8e6d-fb074c7028e5 (api:48) 2019-04-13 21:23:04,136+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '4998716928', 'apparentsize': '16106127360'} from=internal, task_id=d5b2f98a-9e7c-4d3c-9ebe-92d4f9daef97 (api:54) 2019-04-13 21:23:04,136+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID='a884b4ce-381f-4653-be14-a31d15181f48', options=None) from=internal, task_id=665a9cf8-c2c1-4ed0-add5-1231f5a1ccd1 (api:48) 2019-04-13 21:23:04,138+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2109410304', 'apparentsize': '8589934592'} from=internal, task_id=be8abc8f-0879-4c2d-9e21-497d98df92ef (api:54) 2019-04-13 21:23:04,139+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', options=None) from=internal, task_id=93008813-12ca-4089-ac3a-566f0e5f5bcd (api:48) 2019-04-13 21:23:04,139+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '1294894592', 'apparentsize': '12884901888'} from=internal, task_id=335f9992-4834-4168-983e-8ee627673978 (api:54) 2019-04-13 21:23:04,141+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '7263202304', 'apparentsize': '21474836480'} from=internal, task_id=9648a214-7492-4eca-8e6d-fb074c7028e5 (api:54) 2019-04-13 21:23:04,142+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '3447286784', 'apparentsize': '16106127360'} from=internal, task_id=665a9cf8-c2c1-4ed0-add5-1231f5a1ccd1 (api:54) 2019-04-13 21:23:04,143+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2879759360', 'apparentsize': '8589934592'} from=internal, task_id=93008813-12ca-4089-ac3a-566f0e5f5bcd (api:54) 2019-04-13 21:23:06,318+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:23:06,347+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:23:06,347+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:23:06,348+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:23:07,281+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=aa1b37ec-f8d8-4d93-b449-cd8a4cc95fe5 (api:48) 2019-04-13 21:23:07,285+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=aa1b37ec-f8d8-4d93-b449-cd8a4cc95fe5 (api:54) 2019-04-13 21:23:07,286+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:23:07,298+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=802cd626-d3b5-4547-bbed-bb3a92b89e6a (api:48) 2019-04-13 21:23:07,304+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707133493248', 'isoprefix': '', 'alerts': [], 'disktotal': '3898231226368', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707133493248', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719882080256', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707133493248', 'isoprefix': '', 'alerts': [], 'disktotal': '4364522487808', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=802cd626-d3b5-4547-bbed-bb3a92b89e6a (api:54) 2019-04-13 21:23:07,305+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:23:13,937+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=086311e2-aa92-4d9a-b104-3322589626e9 (api:48) 2019-04-13 21:23:13,937+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000793394', 'lastCheck': '0.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000626732', 'lastCheck': '0.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000681102', 'lastCheck': '0.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000380507', 'lastCheck': '6.7', 'valid': True}} from=internal, task_id=086311e2-aa92-4d9a-b104-3322589626e9 (api:54) 2019-04-13 21:23:14,272+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:23:14,346+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=055892f0-0c68-4841-a935-fd72b29bfb0a (api:48) 2019-04-13 21:23:14,347+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000793394', 'lastCheck': '0.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000528571', 'lastCheck': '0.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000681102', 'lastCheck': '0.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000457667', 'lastCheck': '0.2', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=055892f0-0c68-4841-a935-fd72b29bfb0a (api:54) 2019-04-13 21:23:14,349+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=7985dca4-851b-4fb0-a9f8-204124cdfb55 (api:48) 2019-04-13 21:23:14,350+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=7985dca4-851b-4fb0-a9f8-204124cdfb55 (api:54) 2019-04-13 21:23:14,368+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '20.22', 'nodeIndex': 1, 'cpuSys': '2.59', 'cpuIdle': '77.19'}, '21': {'cpuUser': '26.45', 'nodeIndex': 1, 'cpuSys': '2.39', 'cpuIdle': '71.16'}, '22': {'cpuUser': '24.07', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '73.41'}, '23': {'cpuUser': '26.12', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '71.76'}, '1': {'cpuUser': '33.75', 'nodeIndex': 0, 'cpuSys': '4.18', 'cpuIdle': '62.07'}, '0': {'cpuUser': '29.97', 'nodeIndex': 0, 'cpuSys': '2.92', 'cpuIdle': '67.11'}, '3': {'cpuUser': '33.22', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '64.33'}, '2': {'cpuUser': '34.74', 'nodeIndex': 0, 'cpuSys': '2.39', 'cpuIdle': '62.87'}, '5': {'cpuUser': '46.87', 'nodeIndex': 0, 'cpuSys': '2.06', 'cpuIdle': '51.07'}, '4': {'cpuUser': '38.32', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '59.36'}, '7': {'cpuUser': '37.13', 'nodeIndex': 1, 'cpuSys': '3.51', 'cpuIdle': '59.36'}, '6': {'cpuUser': '38.65', 'nodeIndex': 1, 'cpuSys': '3.38', 'cpuIdle': '57.97'}, '9': {'cpuUser': '37.13', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '59.89'}, '8': {'cpuUser': '36.73', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '60.09'}, '11': {'cpuUser': '34.74', 'nodeIndex': 1, 'cpuSys': '3.38', 'cpuIdle': '61.88'}, '10': {'cpuUser': '34.87', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '62.08'}, '13': {'cpuUser': '15.32', 'nodeIndex': 0, 'cpuSys': '2.72', 'cpuIdle': '81.96'}, '12': {'cpuUser': '23.87', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '74.27'}, '15': {'cpuUser': '30.30', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '68.04'}, '14': {'cpuUser': '23.14', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '74.87'}, '17': {'cpuUser': '12.86', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '84.82'}, '16': {'cpuUser': '20.16', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '77.72'}, '19': {'cpuUser': '13.19', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '84.09'}, '18': {'cpuUser': '17.83', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '79.72'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '70103'}, '0': {'memPercent': 22, 'memFree': '77631'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000793394', 'lastCheck': '0.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000681102', 'lastCheck': '0.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000457667', 'lastCheck': '0.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000528571', 'lastCheck': '0.1', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168848510', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '5091487358', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590444259257', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '2705587712572', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433825998400', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '259039431154', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239863554', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '366092662', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122791900', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121313397', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423672189056', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '253620012618', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215271196', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680361987', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '29546851039', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652294875', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '29289333593', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255808400705', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '7076198182', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121687950', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75299530121', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '61842491868', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122246880', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418925591', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '1012773531', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121499327', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183394.343609, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183394.343609, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183394.343609, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183394.343609, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121269358', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458738599215', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183394.343609, 'rx': '2576947729701', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322192557508', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183394.343609, 'rx': '1322192557508', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183394.343609, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940832361', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '2700636821637', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571503426896', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '4950890935', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329596075235', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183394.343609, 'rx': '980090080406', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183394.343609, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183394.343609, 'rx': '45354474', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183394.343609, 'rx': '39834254', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1728142.43', 'cpuLoad': '8.19', 'cpuSys': '2.64', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '2.98', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150381, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:23:14 GMT', 'cpuUser': '28.74', 'memFree': 150125, 'cpuIdle': '68.62', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.06'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:23:14,375+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.11 seconds (__init__:312) 2019-04-13 21:23:17,190+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:23:17,225+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:23:17,225+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:23:17,226+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:23:17,359+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=a5aaa6b0-1749-4ed8-bebf-07fa9b6bfb43 (api:48) 2019-04-13 21:23:17,363+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=a5aaa6b0-1749-4ed8-bebf-07fa9b6bfb43 (api:54) 2019-04-13 21:23:17,363+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:23:17,377+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=0f97e8ff-0451-4614-b5fd-642aaca6c4c8 (api:48) 2019-04-13 21:23:17,384+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707133493248', 'isoprefix': '', 'alerts': [], 'disktotal': '3898231226368', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707133493248', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719882080256', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707133493248', 'isoprefix': '', 'alerts': [], 'disktotal': '4364522487808', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=0f97e8ff-0451-4614-b5fd-642aaca6c4c8 (api:54) 2019-04-13 21:23:17,385+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:23:21,355+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:23:21,386+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:23:21,386+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:23:21,387+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:23:27,430+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=d365ffe8-e870-4507-841f-40ea95974b50 (api:48) 2019-04-13 21:23:27,434+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=d365ffe8-e870-4507-841f-40ea95974b50 (api:54) 2019-04-13 21:23:27,435+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:23:27,447+0200 INFO (jsonrpc/0) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=d253bcf2-c48a-4009-8fce-12155304985a (api:48) 2019-04-13 21:23:27,459+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707129298944', 'isoprefix': '', 'alerts': [], 'disktotal': '3898227032064', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707129298944', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719877885952', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707129298944', 'isoprefix': '', 'alerts': [], 'disktotal': '4364517244928', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=d253bcf2-c48a-4009-8fce-12155304985a (api:54) 2019-04-13 21:23:27,460+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:23:29,032+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=398b620f-3109-48c1-af47-9d2f95004d18 (api:48) 2019-04-13 21:23:29,032+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000620331', 'lastCheck': '4.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000577492', 'lastCheck': '4.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000597239', 'lastCheck': '4.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000442495', 'lastCheck': '1.8', 'valid': True}} from=internal, task_id=398b620f-3109-48c1-af47-9d2f95004d18 (api:54) 2019-04-13 21:23:29,314+0200 INFO (jsonrpc/6) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:23:29,381+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=375201ec-06f8-4886-af8b-6b1cf4ba3df7 (api:48) 2019-04-13 21:23:29,382+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000620331', 'lastCheck': '5.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000577492', 'lastCheck': '5.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000597239', 'lastCheck': '5.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000442495', 'lastCheck': '2.2', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=375201ec-06f8-4886-af8b-6b1cf4ba3df7 (api:54) 2019-04-13 21:23:29,383+0200 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=8af41d74-1fa5-4624-afed-051e1e0ffab2 (api:48) 2019-04-13 21:23:29,384+0200 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=8af41d74-1fa5-4624-afed-051e1e0ffab2 (api:54) 2019-04-13 21:23:29,407+0200 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '26.60', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '71.21'}, '21': {'cpuUser': '12.73', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '84.75'}, '22': {'cpuUser': '25.14', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '72.67'}, '23': {'cpuUser': '23.68', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '73.87'}, '1': {'cpuUser': '33.89', 'nodeIndex': 0, 'cpuSys': '5.17', 'cpuIdle': '60.94'}, '0': {'cpuUser': '31.77', 'nodeIndex': 0, 'cpuSys': '2.85', 'cpuIdle': '65.38'}, '3': {'cpuUser': '50.81', 'nodeIndex': 0, 'cpuSys': '2.26', 'cpuIdle': '46.93'}, '2': {'cpuUser': '30.05', 'nodeIndex': 0, 'cpuSys': '2.72', 'cpuIdle': '67.23'}, '5': {'cpuUser': '39.99', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '57.82'}, '4': {'cpuUser': '47.82', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '50.06'}, '7': {'cpuUser': '38.60', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '58.42'}, '6': {'cpuUser': '34.02', 'nodeIndex': 1, 'cpuSys': '2.79', 'cpuIdle': '63.19'}, '9': {'cpuUser': '41.59', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '55.23'}, '8': {'cpuUser': '34.22', 'nodeIndex': 1, 'cpuSys': '3.45', 'cpuIdle': '62.33'}, '11': {'cpuUser': '39.99', 'nodeIndex': 1, 'cpuSys': '2.79', 'cpuIdle': '57.22'}, '10': {'cpuUser': '38.40', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '58.75'}, '13': {'cpuUser': '14.26', 'nodeIndex': 0, 'cpuSys': '2.59', 'cpuIdle': '83.15'}, '12': {'cpuUser': '25.00', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '72.55'}, '15': {'cpuUser': '14.79', 'nodeIndex': 0, 'cpuSys': '2.79', 'cpuIdle': '82.42'}, '14': {'cpuUser': '31.24', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '66.90'}, '17': {'cpuUser': '28.32', 'nodeIndex': 0, 'cpuSys': '1.46', 'cpuIdle': '70.22'}, '16': {'cpuUser': '25.40', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '72.74'}, '19': {'cpuUser': '19.96', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '77.32'}, '18': {'cpuUser': '30.51', 'nodeIndex': 1, 'cpuSys': '1.86', 'cpuIdle': '67.63'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '70046'}, '0': {'memPercent': 21, 'memFree': '77685'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000620331', 'lastCheck': '5.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000597239', 'lastCheck': '5.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000442495', 'lastCheck': '2.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000577492', 'lastCheck': '5.1', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168849654', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '5091488616', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590444785471', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '2705588836900', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433829268440', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '259041863239', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239863554', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '366092662', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122791900', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121313397', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423675599738', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '253622443234', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215271196', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680361987', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '29546851039', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652295509', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '29289334197', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255810886464', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '7078824436', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121687950', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75301799775', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '61847243862', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122246880', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418926131', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '1012773615', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121499327', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183409.379902, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183409.379902, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183409.379902, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183409.379902, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121269358', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458739104303', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183409.379902, 'rx': '2576948802979', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322203989039', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183409.379902, 'rx': '1322203989039', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183409.379902, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940833129', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '2700637945417', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571503952342', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '4950891483', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329605782227', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183409.379902, 'rx': '980099351932', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183409.379902, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183409.379902, 'rx': '45354864', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183409.379902, 'rx': '39834576', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1728157.48', 'cpuLoad': '8.14', 'cpuSys': '2.59', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '2.72', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150388, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:23:29 GMT', 'cpuUser': '30.78', 'memFree': 150132, 'cpuIdle': '66.63', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.33'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:23:29,417+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.10 seconds (__init__:312) 2019-04-13 21:23:32,257+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:23:32,281+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:23:32,281+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:23:32,282+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.02 seconds (__init__:312) 2019-04-13 21:23:36,394+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:23:36,425+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:23:36,426+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:23:36,427+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:23:37,509+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=f03de1a0-b642-416f-8900-3bb40b51c928 (api:48) 2019-04-13 21:23:37,513+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=f03de1a0-b642-416f-8900-3bb40b51c928 (api:54) 2019-04-13 21:23:37,514+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:23:37,529+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=22425fcb-3e40-4c1e-b8e2-23af0d061005 (api:48) 2019-04-13 21:23:37,536+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707124056064', 'isoprefix': '', 'alerts': [], 'disktotal': '3898221789184', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707124056064', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719872643072', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707124056064', 'isoprefix': '', 'alerts': [], 'disktotal': '4364513050624', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=22425fcb-3e40-4c1e-b8e2-23af0d061005 (api:54) 2019-04-13 21:23:37,537+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:23:44,192+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=dd104c4a-f706-41fa-b683-97ca5a92fd7e (api:48) 2019-04-13 21:23:44,194+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000658494', 'lastCheck': '0.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000684123', 'lastCheck': '0.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000600248', 'lastCheck': '0.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000348223', 'lastCheck': '7.0', 'valid': True}} from=internal, task_id=dd104c4a-f706-41fa-b683-97ca5a92fd7e (api:54) 2019-04-13 21:23:44,362+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:23:44,425+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=0d9aade6-2abb-46ab-b7c2-051901e89113 (api:48) 2019-04-13 21:23:44,425+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000678034', 'lastCheck': '0.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000709603', 'lastCheck': '0.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000571258', 'lastCheck': '0.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000361671', 'lastCheck': '0.2', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=0d9aade6-2abb-46ab-b7c2-051901e89113 (api:54) 2019-04-13 21:23:44,426+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=8696c29b-eb3f-43b7-bbfc-5f2892332cba (api:48) 2019-04-13 21:23:44,426+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=8696c29b-eb3f-43b7-bbfc-5f2892332cba (api:54) 2019-04-13 21:23:44,440+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '20.85', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '76.30'}, '21': {'cpuUser': '28.40', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '69.28'}, '22': {'cpuUser': '26.48', 'nodeIndex': 1, 'cpuSys': '2.18', 'cpuIdle': '71.34'}, '23': {'cpuUser': '26.28', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '71.40'}, '1': {'cpuUser': '38.13', 'nodeIndex': 0, 'cpuSys': '4.10', 'cpuIdle': '57.77'}, '0': {'cpuUser': '54.61', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '43.60'}, '3': {'cpuUser': '46.34', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '51.74'}, '2': {'cpuUser': '44.68', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '53.07'}, '5': {'cpuUser': '48.92', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '49.09'}, '4': {'cpuUser': '51.30', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '46.85'}, '7': {'cpuUser': '34.03', 'nodeIndex': 1, 'cpuSys': '3.77', 'cpuIdle': '62.20'}, '6': {'cpuUser': '37.86', 'nodeIndex': 1, 'cpuSys': '3.44', 'cpuIdle': '58.70'}, '9': {'cpuUser': '30.38', 'nodeIndex': 1, 'cpuSys': '3.71', 'cpuIdle': '65.91'}, '8': {'cpuUser': '32.70', 'nodeIndex': 1, 'cpuSys': '3.77', 'cpuIdle': '63.53'}, '11': {'cpuUser': '38.79', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '58.10'}, '10': {'cpuUser': '36.61', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '60.21'}, '13': {'cpuUser': '24.56', 'nodeIndex': 0, 'cpuSys': '2.18', 'cpuIdle': '73.26'}, '12': {'cpuUser': '18.14', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '79.74'}, '15': {'cpuUser': '21.84', 'nodeIndex': 0, 'cpuSys': '2.52', 'cpuIdle': '75.64'}, '14': {'cpuUser': '31.11', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '67.17'}, '17': {'cpuUser': '19.13', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '78.88'}, '16': {'cpuUser': '33.10', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '65.31'}, '19': {'cpuUser': '28.93', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '68.62'}, '18': {'cpuUser': '28.86', 'nodeIndex': 1, 'cpuSys': '2.18', 'cpuIdle': '68.96'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '70022'}, '0': {'memPercent': 21, 'memFree': '77710'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000678034', 'lastCheck': '0.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000571258', 'lastCheck': '0.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000361671', 'lastCheck': '0.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000709603', 'lastCheck': '0.2', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168850282', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '5091488700', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590446681931', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '2705592198512', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433832477492', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '259044170907', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239864062', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '366092662', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122792408', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121313905', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423678881592', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '253624817489', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215271704', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680373031', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '29546858697', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652303511', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '29289344897', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255813253652', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '7081882654', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121688458', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75304002029', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '61851732087', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122247388', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418926611', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '1012773615', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121499835', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183424.423989, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183424.423989, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183424.423989, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183424.423989, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121269866', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458740934207', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183424.423989, 'rx': '2576952016337', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322215388066', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183424.423989, 'rx': '1322215388066', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183424.423989, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940833215', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '2700641306313', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571505848716', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '4950892199', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329616389738', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183424.423989, 'rx': '980108325184', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183424.423989, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183424.423989, 'rx': '45355254', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183424.423989, 'rx': '39834944', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1728172.52', 'cpuLoad': '8.13', 'cpuSys': '2.55', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '2.98', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150390, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:23:44 GMT', 'cpuUser': '33.42', 'memFree': 150134, 'cpuIdle': '64.02', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '0.99'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:23:44,445+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:23:47,302+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:23:47,345+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:23:47,346+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:23:47,347+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.05 seconds (__init__:312) 2019-04-13 21:23:47,594+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=3f1d3244-6de0-4ea7-a00d-3a968eae58d3 (api:48) 2019-04-13 21:23:47,599+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=3f1d3244-6de0-4ea7-a00d-3a968eae58d3 (api:54) 2019-04-13 21:23:47,600+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:23:47,613+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=e1154c73-aca2-4d91-87b2-7b0b29505240 (api:48) 2019-04-13 21:23:47,621+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707121958912', 'isoprefix': '', 'alerts': [], 'disktotal': '3898219692032', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707121958912', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719870545920', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707121958912', 'isoprefix': '', 'alerts': [], 'disktotal': '4364510953472', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=e1154c73-aca2-4d91-87b2-7b0b29505240 (api:54) 2019-04-13 21:23:47,623+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:23:51,434+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:23:51,464+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:23:51,465+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:23:51,466+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:23:57,239+0200 INFO (itmap/0) [IOProcessClient] (/onode-1.discovery.intern:_mnt_store) Starting client (__init__:308) 2019-04-13 21:23:57,257+0200 INFO (itmap/1) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_vms) Starting client (__init__:308) 2019-04-13 21:23:57,262+0200 INFO (ioprocess/28523) [IOProcess] (/onode-1.discovery.intern:_mnt_store) Starting ioprocess (__init__:434) 2019-04-13 21:23:57,278+0200 INFO (itmap/2) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_isos) Starting client (__init__:308) 2019-04-13 21:23:57,284+0200 INFO (ioprocess/28531) [IOProcess] (/nas.discovery.intern:_mnt_d2600_vms) Starting ioprocess (__init__:434) 2019-04-13 21:23:57,296+0200 INFO (itmap/3) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting client (__init__:308) 2019-04-13 21:23:57,305+0200 INFO (ioprocess/28539) [IOProcess] (/nas.discovery.intern:_mnt_d2600_isos) Starting ioprocess (__init__:434) 2019-04-13 21:23:57,323+0200 INFO (itmap/4) [IOProcessClient] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting client (__init__:308) 2019-04-13 21:23:57,334+0200 INFO (ioprocess/28546) [IOProcess] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting ioprocess (__init__:434) 2019-04-13 21:23:57,354+0200 INFO (ioprocess/28553) [IOProcess] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting ioprocess (__init__:434) 2019-04-13 21:23:57,364+0200 INFO (monitor/6a4cde7) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:23:57,668+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=a3462476-e24d-4b25-9741-4a53b2c6e305 (api:48) 2019-04-13 21:23:57,673+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=a3462476-e24d-4b25-9741-4a53b2c6e305 (api:54) 2019-04-13 21:23:57,674+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:23:57,688+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=6ae3621f-793c-44ab-a3b2-c727e07de349 (api:48) 2019-04-13 21:23:57,698+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707119861760', 'isoprefix': '', 'alerts': [], 'disktotal': '3898217594880', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707119861760', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719868448768', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707119861760', 'isoprefix': '', 'alerts': [], 'disktotal': '4364508856320', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=6ae3621f-793c-44ab-a3b2-c727e07de349 (api:54) 2019-04-13 21:23:57,700+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:23:59,321+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=30769c6c-b6e1-4ae1-82f9-571558397da4 (api:48) 2019-04-13 21:23:59,323+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00052654', 'lastCheck': '4.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000742713', 'lastCheck': '5.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000643117', 'lastCheck': '4.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000376217', 'lastCheck': '2.0', 'valid': True}} from=internal, task_id=30769c6c-b6e1-4ae1-82f9-571558397da4 (api:54) 2019-04-13 21:23:59,466+0200 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:23:59,533+0200 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=1ec653c8-9042-486b-95c6-37a4355926ea (api:48) 2019-04-13 21:23:59,534+0200 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00052654', 'lastCheck': '5.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000742713', 'lastCheck': '5.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000643117', 'lastCheck': '5.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000376217', 'lastCheck': '2.2', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=1ec653c8-9042-486b-95c6-37a4355926ea (api:54) 2019-04-13 21:23:59,535+0200 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=594e9adb-56e4-4b73-a8d5-252cf4c790e9 (api:48) 2019-04-13 21:23:59,535+0200 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=594e9adb-56e4-4b73-a8d5-252cf4c790e9 (api:54) 2019-04-13 21:23:59,547+0200 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '26.37', 'nodeIndex': 1, 'cpuSys': '2.97', 'cpuIdle': '70.66'}, '21': {'cpuUser': '19.12', 'nodeIndex': 1, 'cpuSys': '3.16', 'cpuIdle': '77.72'}, '22': {'cpuUser': '17.01', 'nodeIndex': 1, 'cpuSys': '3.16', 'cpuIdle': '79.83'}, '23': {'cpuUser': '21.03', 'nodeIndex': 1, 'cpuSys': '2.64', 'cpuIdle': '76.33'}, '1': {'cpuUser': '38.96', 'nodeIndex': 0, 'cpuSys': '5.14', 'cpuIdle': '55.90'}, '0': {'cpuUser': '54.26', 'nodeIndex': 0, 'cpuSys': '1.65', 'cpuIdle': '44.09'}, '3': {'cpuUser': '44.24', 'nodeIndex': 0, 'cpuSys': '1.65', 'cpuIdle': '54.11'}, '2': {'cpuUser': '57.75', 'nodeIndex': 0, 'cpuSys': '1.25', 'cpuIdle': '41.00'}, '5': {'cpuUser': '47.40', 'nodeIndex': 0, 'cpuSys': '2.04', 'cpuIdle': '50.56'}, '4': {'cpuUser': '56.44', 'nodeIndex': 0, 'cpuSys': '1.71', 'cpuIdle': '41.85'}, '7': {'cpuUser': '29.34', 'nodeIndex': 1, 'cpuSys': '4.62', 'cpuIdle': '66.04'}, '6': {'cpuUser': '34.94', 'nodeIndex': 1, 'cpuSys': '4.22', 'cpuIdle': '60.84'}, '9': {'cpuUser': '37.71', 'nodeIndex': 1, 'cpuSys': '3.82', 'cpuIdle': '58.47'}, '8': {'cpuUser': '32.04', 'nodeIndex': 1, 'cpuSys': '4.02', 'cpuIdle': '63.94'}, '11': {'cpuUser': '32.37', 'nodeIndex': 1, 'cpuSys': '4.29', 'cpuIdle': '63.34'}, '10': {'cpuUser': '32.64', 'nodeIndex': 1, 'cpuSys': '4.35', 'cpuIdle': '63.01'}, '13': {'cpuUser': '51.43', 'nodeIndex': 0, 'cpuSys': '1.32', 'cpuIdle': '47.25'}, '12': {'cpuUser': '25.38', 'nodeIndex': 0, 'cpuSys': '2.37', 'cpuIdle': '72.25'}, '15': {'cpuUser': '38.64', 'nodeIndex': 0, 'cpuSys': '1.45', 'cpuIdle': '59.91'}, '14': {'cpuUser': '26.11', 'nodeIndex': 0, 'cpuSys': '1.78', 'cpuIdle': '72.11'}, '17': {'cpuUser': '31.12', 'nodeIndex': 0, 'cpuSys': '2.18', 'cpuIdle': '66.70'}, '16': {'cpuUser': '27.56', 'nodeIndex': 0, 'cpuSys': '1.52', 'cpuIdle': '70.92'}, '19': {'cpuUser': '24.00', 'nodeIndex': 1, 'cpuSys': '3.16', 'cpuIdle': '72.84'}, '18': {'cpuUser': '15.96', 'nodeIndex': 1, 'cpuSys': '3.43', 'cpuIdle': '80.61'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '69989'}, '0': {'memPercent': 21, 'memFree': '77739'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00052654', 'lastCheck': '5.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000643117', 'lastCheck': '5.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000376217', 'lastCheck': '2.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000742713', 'lastCheck': '5.3', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168850409', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '5091488700', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590447388185', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '2705594035391', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433836120578', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '259046516846', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239864189', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '366092662', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122792535', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121314032', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423682164778', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '253627211005', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215271831', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680373158', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '29546858697', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652303748', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '29289345001', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255815648596', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '7085464591', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121688585', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75306455783', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '61855960416', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122247515', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418927031', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '1012773615', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121499962', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183439.532604, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183439.532604, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183439.532604, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183439.532604, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121269993', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458741611463', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183439.532604, 'rx': '2576953771576', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322227874623', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183439.532604, 'rx': '1322227874623', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183439.532604, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940833343', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '2700643143192', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571506554842', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '4950892199', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329626845326', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183439.532604, 'rx': '980118066818', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183439.532604, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183439.532604, 'rx': '45355644', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183439.532604, 'rx': '39835266', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40566', 'ksmPages': 100, 'elapsedTime': '1728187.63', 'cpuLoad': '8.32', 'cpuSys': '2.83', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.49', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150386, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:23:59 GMT', 'cpuUser': '34.24', 'memFree': 150130, 'cpuIdle': '62.93', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.85'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:23:59,553+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.08 seconds (__init__:312) 2019-04-13 21:24:02,381+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:24:02,422+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:24:02,422+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:24:02,424+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.05 seconds (__init__:312) 2019-04-13 21:24:04,114+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID=u'ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', options=None) from=internal, task_id=24a55017-4ca8-4c60-a15c-f7d341f9aee4 (api:48) 2019-04-13 21:24:04,116+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', options=None) from=internal, task_id=c8e2b019-4394-4e8f-89e8-13b8bd77d131 (api:48) 2019-04-13 21:24:04,118+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', options=None) from=internal, task_id=d4f27b4c-92a6-468f-9953-7f63cc667f92 (api:48) 2019-04-13 21:24:04,121+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '4579696128', 'apparentsize': '16106127360'} from=internal, task_id=c8e2b019-4394-4e8f-89e8-13b8bd77d131 (api:54) 2019-04-13 21:24:04,122+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID='ef6741d1-380a-4736-a924-f88ed68e1270', options=None) from=internal, task_id=60c127b8-8a77-4551-8c43-5fb67ffe5e0f (api:48) 2019-04-13 21:24:04,123+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '53760936448', 'apparentsize': '139586437120'} from=internal, task_id=24a55017-4ca8-4c60-a15c-f7d341f9aee4 (api:54) 2019-04-13 21:24:04,123+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID=u'67a4dd71-69ff-49a8-b8a4-c7101c75e105', options=None) from=internal, task_id=92b856cf-95f9-454e-a220-4b7ea124f0dd (api:48) 2019-04-13 21:24:04,125+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '5697701376', 'apparentsize': '16106127360'} from=internal, task_id=d4f27b4c-92a6-468f-9953-7f63cc667f92 (api:54) 2019-04-13 21:24:04,125+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', options=None) from=internal, task_id=246d948b-7c4e-4300-a802-b1561ed266ff (api:48) 2019-04-13 21:24:04,128+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID=u'2b136ed7-8b9a-45c1-9edc-64ee96c87c43', options=None) from=internal, task_id=d9f201c6-2996-4ffc-aa77-4b83c59cd800 (api:48) 2019-04-13 21:24:04,130+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2524573696', 'apparentsize': '12884901888'} from=internal, task_id=246d948b-7c4e-4300-a802-b1561ed266ff (api:54) 2019-04-13 21:24:04,131+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID=u'44b3ceca-f0db-43da-af29-aa681df146ac', options=None) from=internal, task_id=cb4da184-86d5-471c-b279-6fc18cab4a64 (api:48) 2019-04-13 21:24:04,131+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '6477472768', 'apparentsize': '21474836480'} from=internal, task_id=60c127b8-8a77-4551-8c43-5fb67ffe5e0f (api:54) 2019-04-13 21:24:04,132+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=5a242683-b234-4d33-b4cc-91f4c7965076 (api:48) 2019-04-13 21:24:04,133+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '55059126272', 'apparentsize': '171798691840'} from=internal, task_id=92b856cf-95f9-454e-a220-4b7ea124f0dd (api:54) 2019-04-13 21:24:04,133+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID='123e2cc8-8f24-4ddd-b401-68393535f555', options=None) from=internal, task_id=4d14419e-01b5-4cf5-87ed-092ffa3b73e3 (api:48) 2019-04-13 21:24:04,135+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2024405504', 'apparentsize': '10737418240'} from=internal, task_id=d9f201c6-2996-4ffc-aa77-4b83c59cd800 (api:54) 2019-04-13 21:24:04,136+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID='3f6c31c6-ec38-4979-a617-55537123ee61', options=None) from=internal, task_id=ad298e32-34c3-44fb-905a-bd0c933545ce (api:48) 2019-04-13 21:24:04,138+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '11196678144', 'apparentsize': '21474836480'} from=internal, task_id=cb4da184-86d5-471c-b279-6fc18cab4a64 (api:54) 2019-04-13 21:24:04,139+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', options=None) from=internal, task_id=665666b7-7db0-460b-8a0d-6ebad1b31d69 (api:48) 2019-04-13 21:24:04,140+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2116486656', 'apparentsize': '3948150784'} from=internal, task_id=5a242683-b234-4d33-b4cc-91f4c7965076 (api:54) 2019-04-13 21:24:04,140+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID=u'a1d583c1-e135-4e3f-b6fa-0f29a5261366', options=None) from=internal, task_id=f26a0a6e-dde0-4c57-98fe-aa142e1f0ff2 (api:48) 2019-04-13 21:24:04,141+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2283003904', 'apparentsize': '16106127360'} from=internal, task_id=4d14419e-01b5-4cf5-87ed-092ffa3b73e3 (api:54) 2019-04-13 21:24:04,141+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', options=None) from=internal, task_id=a36c38d6-62be-49da-ae04-6c38accedd5b (api:48) 2019-04-13 21:24:04,144+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '4998716928', 'apparentsize': '16106127360'} from=internal, task_id=ad298e32-34c3-44fb-905a-bd0c933545ce (api:54) 2019-04-13 21:24:04,144+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID='a884b4ce-381f-4653-be14-a31d15181f48', options=None) from=internal, task_id=154e4b97-3c1d-4d60-bad5-40823a9d8ba2 (api:48) 2019-04-13 21:24:04,146+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2109410304', 'apparentsize': '8589934592'} from=internal, task_id=665666b7-7db0-460b-8a0d-6ebad1b31d69 (api:54) 2019-04-13 21:24:04,147+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', options=None) from=internal, task_id=921b1f40-cb01-4491-a19a-cbbaf4297000 (api:48) 2019-04-13 21:24:04,147+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '1294894592', 'apparentsize': '12884901888'} from=internal, task_id=f26a0a6e-dde0-4c57-98fe-aa142e1f0ff2 (api:54) 2019-04-13 21:24:04,149+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '7263202304', 'apparentsize': '21474836480'} from=internal, task_id=a36c38d6-62be-49da-ae04-6c38accedd5b (api:54) 2019-04-13 21:24:04,150+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '3447286784', 'apparentsize': '16106127360'} from=internal, task_id=154e4b97-3c1d-4d60-bad5-40823a9d8ba2 (api:54) 2019-04-13 21:24:04,151+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2879759360', 'apparentsize': '8589934592'} from=internal, task_id=921b1f40-cb01-4491-a19a-cbbaf4297000 (api:54) 2019-04-13 21:24:06,473+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:24:06,506+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:24:06,507+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:24:06,508+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:24:07,751+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=4a26ed58-4a43-4be3-9f80-6c2289f93a91 (api:48) 2019-04-13 21:24:07,756+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=4a26ed58-4a43-4be3-9f80-6c2289f93a91 (api:54) 2019-04-13 21:24:07,756+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:24:07,769+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=45ca44d0-c030-40fc-979f-e47964956645 (api:48) 2019-04-13 21:24:07,776+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707115667456', 'isoprefix': '', 'alerts': [], 'disktotal': '3898213400576', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707115667456', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719864254464', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707115667456', 'isoprefix': '', 'alerts': [], 'disktotal': '4364503613440', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=45ca44d0-c030-40fc-979f-e47964956645 (api:54) 2019-04-13 21:24:07,778+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:24:14,403+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=0ccfbf63-3660-4001-acff-a4a43c5549e1 (api:48) 2019-04-13 21:24:14,404+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000681985', 'lastCheck': '0.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000534477', 'lastCheck': '0.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000588261', 'lastCheck': '0.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000449224', 'lastCheck': '0.2', 'valid': True}} from=internal, task_id=0ccfbf63-3660-4001-acff-a4a43c5549e1 (api:54) 2019-04-13 21:24:14,575+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:24:14,645+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=e0aff2a2-7230-49b8-b5dd-17e3f0fe14ed (api:48) 2019-04-13 21:24:14,645+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000681985', 'lastCheck': '0.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000534477', 'lastCheck': '0.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000588261', 'lastCheck': '0.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000449224', 'lastCheck': '0.5', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=e0aff2a2-7230-49b8-b5dd-17e3f0fe14ed (api:54) 2019-04-13 21:24:14,646+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=7f1cc268-2520-450c-9791-ed87dd10eaef (api:48) 2019-04-13 21:24:14,646+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=7f1cc268-2520-450c-9791-ed87dd10eaef (api:54) 2019-04-13 21:24:14,660+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '23.16', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '74.52'}, '21': {'cpuUser': '18.06', 'nodeIndex': 1, 'cpuSys': '2.51', 'cpuIdle': '79.43'}, '22': {'cpuUser': '19.91', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '77.51'}, '23': {'cpuUser': '21.04', 'nodeIndex': 1, 'cpuSys': '2.38', 'cpuIdle': '76.58'}, '1': {'cpuUser': '40.09', 'nodeIndex': 0, 'cpuSys': '4.30', 'cpuIdle': '55.61'}, '0': {'cpuUser': '41.42', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '56.46'}, '3': {'cpuUser': '54.38', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '43.70'}, '2': {'cpuUser': '51.21', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '47.00'}, '5': {'cpuUser': '52.00', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '46.15'}, '4': {'cpuUser': '53.13', 'nodeIndex': 0, 'cpuSys': '1.65', 'cpuIdle': '45.22'}, '7': {'cpuUser': '32.22', 'nodeIndex': 1, 'cpuSys': '4.10', 'cpuIdle': '63.68'}, '6': {'cpuUser': '34.60', 'nodeIndex': 1, 'cpuSys': '3.84', 'cpuIdle': '61.56'}, '9': {'cpuUser': '33.74', 'nodeIndex': 1, 'cpuSys': '3.71', 'cpuIdle': '62.55'}, '8': {'cpuUser': '30.96', 'nodeIndex': 1, 'cpuSys': '4.04', 'cpuIdle': '65.00'}, '11': {'cpuUser': '33.41', 'nodeIndex': 1, 'cpuSys': '3.44', 'cpuIdle': '63.15'}, '10': {'cpuUser': '30.50', 'nodeIndex': 1, 'cpuSys': '4.10', 'cpuIdle': '65.40'}, '13': {'cpuUser': '22.56', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '75.39'}, '12': {'cpuUser': '27.52', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '70.76'}, '15': {'cpuUser': '14.29', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '83.86'}, '14': {'cpuUser': '16.67', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '81.21'}, '17': {'cpuUser': '32.55', 'nodeIndex': 0, 'cpuSys': '1.39', 'cpuIdle': '66.06'}, '16': {'cpuUser': '15.55', 'nodeIndex': 0, 'cpuSys': '2.18', 'cpuIdle': '82.27'}, '19': {'cpuUser': '15.02', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '82.27'}, '18': {'cpuUser': '12.17', 'nodeIndex': 1, 'cpuSys': '3.51', 'cpuIdle': '84.32'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69473'}, '0': {'memPercent': 21, 'memFree': '78253'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000681985', 'lastCheck': '0.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000588261', 'lastCheck': '0.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000449224', 'lastCheck': '0.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000534477', 'lastCheck': '0.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168850529', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '5091488700', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590448452711', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '2705594096455', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433840567894', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '259048978957', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239864969', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '366093362', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122792655', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121314152', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423686507026', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '253629691377', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215271951', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680373576', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '29546859226', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652304457', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '29289345341', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255818143262', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '7088057701', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121688705', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75308292781', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '61860707091', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122247635', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418927511', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '1012773615', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121500082', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183454.644209, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183454.644209, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183454.644209, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183454.644209, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121270113', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458742685487', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183454.644209, 'rx': '2576953826156', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322238243730', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183454.644209, 'rx': '1322238243730', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183454.644209, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940833407', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '2700643203106', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571507619304', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '4950893349', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329631253919', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183454.644209, 'rx': '980129095176', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183454.644209, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183454.644209, 'rx': '45356034', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183454.644209, 'rx': '39835634', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40566', 'ksmPages': 100, 'elapsedTime': '1728202.74', 'cpuLoad': '8.29', 'cpuSys': '2.67', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.11', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150384, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:24:14 GMT', 'cpuUser': '30.26', 'memFree': 150128, 'cpuIdle': '67.07', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.39'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:24:14,666+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:24:17,459+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:24:17,503+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:24:17,503+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:24:17,505+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.05 seconds (__init__:312) 2019-04-13 21:24:17,827+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=ab668342-8afb-4b7d-8a03-8ed2b315db05 (api:48) 2019-04-13 21:24:17,832+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=ab668342-8afb-4b7d-8a03-8ed2b315db05 (api:54) 2019-04-13 21:24:17,832+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:24:17,969+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=ddf2bfeb-e5d2-4faf-83eb-094dba39f436 (api:48) 2019-04-13 21:24:17,979+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707115667456', 'isoprefix': '', 'alerts': [], 'disktotal': '3898213400576', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707115667456', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719864254464', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707115667456', 'isoprefix': '', 'alerts': [], 'disktotal': '4364503613440', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=ddf2bfeb-e5d2-4faf-83eb-094dba39f436 (api:54) 2019-04-13 21:24:17,981+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:24:21,515+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:24:21,544+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:24:21,545+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:24:21,545+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:24:28,036+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=7157f7a5-e9ea-4894-bfe1-a1a3f6aaa90d (api:48) 2019-04-13 21:24:28,041+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=7157f7a5-e9ea-4894-bfe1-a1a3f6aaa90d (api:54) 2019-04-13 21:24:28,041+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:24:28,054+0200 INFO (jsonrpc/0) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=9f1e1997-f7e9-43ce-a556-360afc517579 (api:48) 2019-04-13 21:24:28,061+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707110424576', 'isoprefix': '', 'alerts': [], 'disktotal': '3898208157696', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707110424576', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719859011584', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707110424576', 'isoprefix': '', 'alerts': [], 'disktotal': '4364499419136', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=9f1e1997-f7e9-43ce-a556-360afc517579 (api:54) 2019-04-13 21:24:28,062+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:24:29,483+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=6d3bc3b2-1b04-4c43-bbe0-f12520738d13 (api:48) 2019-04-13 21:24:29,484+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000431206', 'lastCheck': '5.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000408465', 'lastCheck': '5.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000476176', 'lastCheck': '5.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000456225', 'lastCheck': '2.1', 'valid': True}} from=internal, task_id=6d3bc3b2-1b04-4c43-bbe0-f12520738d13 (api:54) 2019-04-13 21:24:29,624+0200 INFO (jsonrpc/6) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:24:29,692+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=fd392431-8062-4bf2-920b-7a746dad7461 (api:48) 2019-04-13 21:24:29,693+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000431206', 'lastCheck': '5.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000408465', 'lastCheck': '5.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000476176', 'lastCheck': '5.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000456225', 'lastCheck': '2.3', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=fd392431-8062-4bf2-920b-7a746dad7461 (api:54) 2019-04-13 21:24:29,693+0200 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=c830cab6-4683-4ab9-98db-3654cc9154a9 (api:48) 2019-04-13 21:24:29,694+0200 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=c830cab6-4683-4ab9-98db-3654cc9154a9 (api:54) 2019-04-13 21:24:29,706+0200 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '19.57', 'nodeIndex': 1, 'cpuSys': '2.26', 'cpuIdle': '78.17'}, '21': {'cpuUser': '27.39', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '70.42'}, '22': {'cpuUser': '23.21', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '74.60'}, '23': {'cpuUser': '25.93', 'nodeIndex': 1, 'cpuSys': '1.66', 'cpuIdle': '72.41'}, '1': {'cpuUser': '36.48', 'nodeIndex': 0, 'cpuSys': '4.18', 'cpuIdle': '59.34'}, '0': {'cpuUser': '47.56', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '50.52'}, '3': {'cpuUser': '43.58', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '54.43'}, '2': {'cpuUser': '45.63', 'nodeIndex': 0, 'cpuSys': '2.06', 'cpuIdle': '52.31'}, '5': {'cpuUser': '49.01', 'nodeIndex': 0, 'cpuSys': '2.06', 'cpuIdle': '48.93'}, '4': {'cpuUser': '43.58', 'nodeIndex': 0, 'cpuSys': '2.06', 'cpuIdle': '54.36'}, '7': {'cpuUser': '33.36', 'nodeIndex': 1, 'cpuSys': '3.52', 'cpuIdle': '63.12'}, '6': {'cpuUser': '32.70', 'nodeIndex': 1, 'cpuSys': '2.92', 'cpuIdle': '64.38'}, '9': {'cpuUser': '34.56', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '62.39'}, '8': {'cpuUser': '37.94', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '59.01'}, '11': {'cpuUser': '34.16', 'nodeIndex': 1, 'cpuSys': '2.79', 'cpuIdle': '63.05'}, '10': {'cpuUser': '35.62', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '61.53'}, '13': {'cpuUser': '24.47', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '73.81'}, '12': {'cpuUser': '19.57', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '78.24'}, '15': {'cpuUser': '23.68', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '74.20'}, '14': {'cpuUser': '22.88', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '75.33'}, '17': {'cpuUser': '20.76', 'nodeIndex': 0, 'cpuSys': '2.06', 'cpuIdle': '77.18'}, '16': {'cpuUser': '23.08', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '75.06'}, '19': {'cpuUser': '17.91', 'nodeIndex': 1, 'cpuSys': '2.79', 'cpuIdle': '79.30'}, '18': {'cpuUser': '24.47', 'nodeIndex': 1, 'cpuSys': '1.92', 'cpuIdle': '73.61'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69362'}, '0': {'memPercent': 21, 'memFree': '78367'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000431206', 'lastCheck': '5.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000476176', 'lastCheck': '5.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000456225', 'lastCheck': '2.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000408465', 'lastCheck': '5.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168850909', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '5091489018', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590449039159', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '2705595270071', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433843902272', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '259051717872', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239865133', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '366093362', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122792819', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121314316', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423690355540', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '253632246028', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215272115', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680373740', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '29546859226', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652304457', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '29289345505', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255820917679', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '7091491982', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121688869', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75310833609', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '61860728220', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122247799', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418928903', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '1012774609', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121500246', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183469.691459, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183469.691459, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183469.691459, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183469.691459, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121270277', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458743226473', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183469.691459, 'rx': '2576954945898', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322250705429', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183469.691459, 'rx': '1322250705429', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183469.691459, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940833951', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '2700644376498', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571508205208', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '4950893573', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329643065986', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183469.691459, 'rx': '980139247824', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183469.691459, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183469.691459, 'rx': '45356424', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183469.691459, 'rx': '39835956', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1728217.79', 'cpuLoad': '8.26', 'cpuSys': '2.39', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.12', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150387, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:24:29 GMT', 'cpuUser': '31.13', 'memFree': 150131, 'cpuIdle': '66.48', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.19'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:24:29,712+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.08 seconds (__init__:312) 2019-04-13 21:24:32,539+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:24:32,561+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:24:32,561+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:24:32,562+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.02 seconds (__init__:312) 2019-04-13 21:24:36,553+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:24:36,577+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:24:36,578+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:24:36,578+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:24:38,108+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=148b1fae-89f7-45c1-831d-7b838fbd4fc2 (api:48) 2019-04-13 21:24:38,112+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=148b1fae-89f7-45c1-831d-7b838fbd4fc2 (api:54) 2019-04-13 21:24:38,113+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:24:38,127+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=e44585dc-5a2e-4b40-b6fd-b75cb7a51b6b (api:48) 2019-04-13 21:24:38,134+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707110424576', 'isoprefix': '', 'alerts': [], 'disktotal': '3898208157696', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707110424576', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719859011584', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707110424576', 'isoprefix': '', 'alerts': [], 'disktotal': '4364499419136', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=e44585dc-5a2e-4b40-b6fd-b75cb7a51b6b (api:54) 2019-04-13 21:24:38,136+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:24:44,587+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=2bb4dc5b-e902-465e-a833-3b9599e84606 (api:48) 2019-04-13 21:24:44,588+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00058273', 'lastCheck': '0.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000580128', 'lastCheck': '0.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000696338', 'lastCheck': '0.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000382324', 'lastCheck': '0.4', 'valid': True}} from=internal, task_id=2bb4dc5b-e902-465e-a833-3b9599e84606 (api:54) 2019-04-13 21:24:44,673+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:24:44,760+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=0d79981e-c58c-4eeb-8c74-0357ffd18f18 (api:48) 2019-04-13 21:24:44,761+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00058273', 'lastCheck': '0.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000580128', 'lastCheck': '0.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000696338', 'lastCheck': '0.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000382324', 'lastCheck': '0.6', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=0d79981e-c58c-4eeb-8c74-0357ffd18f18 (api:54) 2019-04-13 21:24:44,762+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=c8d50cf3-72a7-48f4-add8-1a3e5dc5f946 (api:48) 2019-04-13 21:24:44,762+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=c8d50cf3-72a7-48f4-add8-1a3e5dc5f946 (api:54) 2019-04-13 21:24:44,775+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '22.87', 'nodeIndex': 1, 'cpuSys': '1.99', 'cpuIdle': '75.14'}, '21': {'cpuUser': '25.99', 'nodeIndex': 1, 'cpuSys': '2.06', 'cpuIdle': '71.95'}, '22': {'cpuUser': '21.28', 'nodeIndex': 1, 'cpuSys': '1.72', 'cpuIdle': '77.00'}, '23': {'cpuUser': '19.82', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '77.53'}, '1': {'cpuUser': '45.75', 'nodeIndex': 0, 'cpuSys': '4.24', 'cpuIdle': '50.01'}, '0': {'cpuUser': '42.30', 'nodeIndex': 0, 'cpuSys': '2.39', 'cpuIdle': '55.31'}, '3': {'cpuUser': '51.52', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '46.49'}, '2': {'cpuUser': '38.06', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '59.75'}, '5': {'cpuUser': '40.91', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '57.10'}, '4': {'cpuUser': '46.14', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '51.74'}, '7': {'cpuUser': '33.48', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '63.21'}, '6': {'cpuUser': '31.69', 'nodeIndex': 1, 'cpuSys': '3.51', 'cpuIdle': '64.80'}, '9': {'cpuUser': '28.91', 'nodeIndex': 1, 'cpuSys': '3.25', 'cpuIdle': '67.84'}, '8': {'cpuUser': '31.29', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '65.86'}, '11': {'cpuUser': '33.15', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '63.54'}, '10': {'cpuUser': '30.10', 'nodeIndex': 1, 'cpuSys': '3.38', 'cpuIdle': '66.52'}, '13': {'cpuUser': '16.97', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '81.24'}, '12': {'cpuUser': '19.23', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '78.85'}, '15': {'cpuUser': '22.21', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '75.47'}, '14': {'cpuUser': '33.81', 'nodeIndex': 0, 'cpuSys': '1.46', 'cpuIdle': '64.73'}, '17': {'cpuUser': '24.73', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '73.55'}, '16': {'cpuUser': '17.24', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '80.90'}, '19': {'cpuUser': '24.27', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '73.54'}, '18': {'cpuUser': '22.81', 'nodeIndex': 1, 'cpuSys': '1.86', 'cpuIdle': '75.33'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69492'}, '0': {'memPercent': 21, 'memFree': '78234'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00058273', 'lastCheck': '0.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000696338', 'lastCheck': '0.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000382324', 'lastCheck': '0.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000580128', 'lastCheck': '0.5', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168851029', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '5091489102', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590449571366', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '2705596394292', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433846924770', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '259053432422', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239865133', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '366093362', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122792819', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121314316', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423693398638', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '253634781862', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215272115', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680373837', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '29546859334', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652304750', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '29289345765', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255822677618', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '7093318941', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121688869', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75313223225', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '61865743089', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122247799', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418929383', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '1012774609', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121500246', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183484.759709, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183484.759709, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183484.759709, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183484.759709, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121270277', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458743712819', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183484.759709, 'rx': '2576956022870', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322261074482', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183484.759709, 'rx': '1322261074482', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183484.759709, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940833951', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '2700645500052', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571508737415', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '4950894240', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329652764923', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183484.759709, 'rx': '980147994102', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183484.759709, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183484.759709, 'rx': '45356814', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183484.759709, 'rx': '39836324', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40566', 'ksmPages': 100, 'elapsedTime': '1728232.83', 'cpuLoad': '8.24', 'cpuSys': '2.41', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '2.72', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150384, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:24:44 GMT', 'cpuUser': '30.19', 'memFree': 150128, 'cpuIdle': '67.41', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.13'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:24:44,781+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.11 seconds (__init__:312) 2019-04-13 21:24:47,572+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:24:47,600+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:24:47,601+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:24:47,602+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:24:48,177+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=25713c33-b8ce-4303-a31b-da0556224d80 (api:48) 2019-04-13 21:24:48,182+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=25713c33-b8ce-4303-a31b-da0556224d80 (api:54) 2019-04-13 21:24:48,182+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:24:48,194+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=0f09bfaf-4d23-48b5-98f3-04e99fc682dd (api:48) 2019-04-13 21:24:48,202+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707106230272', 'isoprefix': '', 'alerts': [], 'disktotal': '3898202914816', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707106230272', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719854817280', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707106230272', 'isoprefix': '', 'alerts': [], 'disktotal': '4364494176256', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=0f09bfaf-4d23-48b5-98f3-04e99fc682dd (api:54) 2019-04-13 21:24:48,203+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:24:51,585+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:24:51,618+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:24:51,619+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:24:51,619+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:24:57,401+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_isos) Closing client (__init__:584) 2019-04-13 21:24:57,402+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/glusterSD/onode-2.discovery.intern:vol-node-2) Closing client (__init__:584) 2019-04-13 21:24:57,402+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_vms) Closing client (__init__:584) 2019-04-13 21:24:57,402+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/onode-1.discovery.intern:_mnt_store) Closing client (__init__:584) 2019-04-13 21:24:57,402+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Closing client (__init__:584) 2019-04-13 21:24:58,250+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=c46ba963-6f3b-4035-af4b-5705fef1679a (api:48) 2019-04-13 21:24:58,254+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=c46ba963-6f3b-4035-af4b-5705fef1679a (api:54) 2019-04-13 21:24:58,254+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:24:58,354+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=d17db251-b77f-483a-a54c-33dc44736114 (api:48) 2019-04-13 21:24:58,362+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707100987392', 'isoprefix': '', 'alerts': [], 'disktotal': '3898198720512', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707100987392', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719849574400', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707100987392', 'isoprefix': '', 'alerts': [], 'disktotal': '4364489981952', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=d17db251-b77f-483a-a54c-33dc44736114 (api:54) 2019-04-13 21:24:58,364+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.02 seconds (__init__:312) 2019-04-13 21:24:59,676+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=2c5f7652-bd71-45da-b5eb-8efc8e1a0cc9 (api:48) 2019-04-13 21:24:59,677+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000866151', 'lastCheck': '5.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000832096', 'lastCheck': '5.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000660346', 'lastCheck': '5.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000465528', 'lastCheck': '2.3', 'valid': True}} from=internal, task_id=2c5f7652-bd71-45da-b5eb-8efc8e1a0cc9 (api:54) 2019-04-13 21:24:59,783+0200 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:24:59,846+0200 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=c1ce9bf4-7153-4284-8415-b0239915b896 (api:48) 2019-04-13 21:24:59,847+0200 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000866151', 'lastCheck': '5.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000832096', 'lastCheck': '5.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000660346', 'lastCheck': '5.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000465528', 'lastCheck': '2.4', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=c1ce9bf4-7153-4284-8415-b0239915b896 (api:54) 2019-04-13 21:24:59,848+0200 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=8a67d6b5-cb0c-4130-bc93-b219b12804fa (api:48) 2019-04-13 21:24:59,848+0200 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=8a67d6b5-cb0c-4130-bc93-b219b12804fa (api:54) 2019-04-13 21:24:59,860+0200 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '16.75', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '81.00'}, '21': {'cpuUser': '21.72', 'nodeIndex': 1, 'cpuSys': '1.99', 'cpuIdle': '76.29'}, '22': {'cpuUser': '13.44', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '83.85'}, '23': {'cpuUser': '22.71', 'nodeIndex': 1, 'cpuSys': '1.79', 'cpuIdle': '75.50'}, '1': {'cpuUser': '41.25', 'nodeIndex': 0, 'cpuSys': '3.77', 'cpuIdle': '54.98'}, '0': {'cpuUser': '44.16', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '53.79'}, '3': {'cpuUser': '50.65', 'nodeIndex': 0, 'cpuSys': '1.46', 'cpuIdle': '47.89'}, '2': {'cpuUser': '46.28', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '51.87'}, '5': {'cpuUser': '42.97', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '55.04'}, '4': {'cpuUser': '38.86', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '59.09'}, '7': {'cpuUser': '28.87', 'nodeIndex': 1, 'cpuSys': '3.58', 'cpuIdle': '67.55'}, '6': {'cpuUser': '25.29', 'nodeIndex': 1, 'cpuSys': '3.58', 'cpuIdle': '71.13'}, '9': {'cpuUser': '25.49', 'nodeIndex': 1, 'cpuSys': '3.71', 'cpuIdle': '70.80'}, '8': {'cpuUser': '26.55', 'nodeIndex': 1, 'cpuSys': '3.97', 'cpuIdle': '69.48'}, '11': {'cpuUser': '27.61', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '69.28'}, '10': {'cpuUser': '26.28', 'nodeIndex': 1, 'cpuSys': '3.91', 'cpuIdle': '69.81'}, '13': {'cpuUser': '14.23', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '83.85'}, '12': {'cpuUser': '22.64', 'nodeIndex': 0, 'cpuSys': '1.32', 'cpuIdle': '76.04'}, '15': {'cpuUser': '21.05', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '77.23'}, '14': {'cpuUser': '19.00', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '79.15'}, '17': {'cpuUser': '30.19', 'nodeIndex': 0, 'cpuSys': '1.39', 'cpuIdle': '68.42'}, '16': {'cpuUser': '24.63', 'nodeIndex': 0, 'cpuSys': '1.52', 'cpuIdle': '73.85'}, '19': {'cpuUser': '14.96', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '82.72'}, '18': {'cpuUser': '21.78', 'nodeIndex': 1, 'cpuSys': '1.92', 'cpuIdle': '76.30'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69473'}, '0': {'memPercent': 21, 'memFree': '78259'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000866151', 'lastCheck': '5.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000660346', 'lastCheck': '5.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000465528', 'lastCheck': '2.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000832096', 'lastCheck': '5.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168851111', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '5091489102', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590450509990', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '2705598332598', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433850444248', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '259055405758', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239865215', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '366093362', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122792901', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121314398', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423696736610', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '253636409410', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215272197', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680373919', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '29546859334', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652304902', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '29289345987', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255824666694', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '7095563301', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121688951', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75315923259', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '61870233447', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122247881', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418929803', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '1012774609', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121500328', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183499.845529, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183499.845529, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183499.845529, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183499.845529, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121270359', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458744616989', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183499.845529, 'rx': '2576957877812', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322274585433', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183499.845529, 'rx': '1322274585433', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183499.845529, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940834143', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '2700647438358', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571509675847', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '4950894240', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329660623753', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183499.845529, 'rx': '980157869268', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183499.845529, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183499.845529, 'rx': '45357204', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183499.845529, 'rx': '39836646', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1728247.94', 'cpuLoad': '8.12', 'cpuSys': '2.41', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.38', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150388, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:24:59 GMT', 'cpuUser': '27.80', 'memFree': 150132, 'cpuIdle': '69.78', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.32'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:24:59,867+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:25:02,653+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:25:02,696+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:25:02,697+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:25:02,698+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.05 seconds (__init__:312) 2019-04-13 21:25:04,112+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID=u'ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', options=None) from=internal, task_id=0baf42f5-edea-4305-8fdf-1193c4d4541b (api:48) 2019-04-13 21:25:04,114+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', options=None) from=internal, task_id=1b5a8e9e-15fe-42ca-9856-619e52792f4d (api:48) 2019-04-13 21:25:04,116+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', options=None) from=internal, task_id=9207b8c8-1898-4008-9690-c5912d98a2ca (api:48) 2019-04-13 21:25:04,122+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '4579696128', 'apparentsize': '16106127360'} from=internal, task_id=1b5a8e9e-15fe-42ca-9856-619e52792f4d (api:54) 2019-04-13 21:25:04,123+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID='ef6741d1-380a-4736-a924-f88ed68e1270', options=None) from=internal, task_id=a9a14acf-86f4-4bee-ba46-5037059014b5 (api:48) 2019-04-13 21:25:04,124+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '53760925696', 'apparentsize': '139586437120'} from=internal, task_id=0baf42f5-edea-4305-8fdf-1193c4d4541b (api:54) 2019-04-13 21:25:04,125+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID=u'67a4dd71-69ff-49a8-b8a4-c7101c75e105', options=None) from=internal, task_id=a7645a7a-9fe4-4c65-841e-c01a43a81fbd (api:48) 2019-04-13 21:25:04,127+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '5697701376', 'apparentsize': '16106127360'} from=internal, task_id=9207b8c8-1898-4008-9690-c5912d98a2ca (api:54) 2019-04-13 21:25:04,128+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', options=None) from=internal, task_id=b7b0c331-aa53-40b6-9e4c-88bcc4a7c235 (api:48) 2019-04-13 21:25:04,132+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID=u'2b136ed7-8b9a-45c1-9edc-64ee96c87c43', options=None) from=internal, task_id=e49c9644-da7f-4e57-8936-6a954e5fe575 (api:48) 2019-04-13 21:25:04,134+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '6477472768', 'apparentsize': '21474836480'} from=internal, task_id=a9a14acf-86f4-4bee-ba46-5037059014b5 (api:54) 2019-04-13 21:25:04,135+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID=u'44b3ceca-f0db-43da-af29-aa681df146ac', options=None) from=internal, task_id=f709313c-29bf-454c-a827-f3ab0ef5e765 (api:48) 2019-04-13 21:25:04,136+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '55059142656', 'apparentsize': '171798691840'} from=internal, task_id=a7645a7a-9fe4-4c65-841e-c01a43a81fbd (api:54) 2019-04-13 21:25:04,137+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=59123179-a07c-45f2-b37a-59336ad18619 (api:48) 2019-04-13 21:25:04,140+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2524573696', 'apparentsize': '12884901888'} from=internal, task_id=b7b0c331-aa53-40b6-9e4c-88bcc4a7c235 (api:54) 2019-04-13 21:25:04,140+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID='123e2cc8-8f24-4ddd-b401-68393535f555', options=None) from=internal, task_id=e3026e08-1818-44cd-b128-729b3082c0a6 (api:48) 2019-04-13 21:25:04,144+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2024405504', 'apparentsize': '10737418240'} from=internal, task_id=e49c9644-da7f-4e57-8936-6a954e5fe575 (api:54) 2019-04-13 21:25:04,144+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID='3f6c31c6-ec38-4979-a617-55537123ee61', options=None) from=internal, task_id=c022bebb-6f7a-47c4-85bd-62a2fcc98d1d (api:48) 2019-04-13 21:25:04,148+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2283003904', 'apparentsize': '16106127360'} from=internal, task_id=e3026e08-1818-44cd-b128-729b3082c0a6 (api:54) 2019-04-13 21:25:04,149+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', options=None) from=internal, task_id=84d0f76b-19af-47dd-ab81-056735155115 (api:48) 2019-04-13 21:25:04,149+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '11196678144', 'apparentsize': '21474836480'} from=internal, task_id=f709313c-29bf-454c-a827-f3ab0ef5e765 (api:54) 2019-04-13 21:25:04,150+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID=u'a1d583c1-e135-4e3f-b6fa-0f29a5261366', options=None) from=internal, task_id=d3d20781-81c1-4c7a-9699-0d8f5e9b2bc4 (api:48) 2019-04-13 21:25:04,154+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2116486656', 'apparentsize': '3948150784'} from=internal, task_id=59123179-a07c-45f2-b37a-59336ad18619 (api:54) 2019-04-13 21:25:04,155+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', options=None) from=internal, task_id=fe09c1ad-d027-4014-bda2-89ae0036cfc8 (api:48) 2019-04-13 21:25:04,155+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '4998716928', 'apparentsize': '16106127360'} from=internal, task_id=c022bebb-6f7a-47c4-85bd-62a2fcc98d1d (api:54) 2019-04-13 21:25:04,156+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID='a884b4ce-381f-4653-be14-a31d15181f48', options=None) from=internal, task_id=503ee849-08d1-4744-84f9-351e3f7b670e (api:48) 2019-04-13 21:25:04,161+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2109410304', 'apparentsize': '8589934592'} from=internal, task_id=84d0f76b-19af-47dd-ab81-056735155115 (api:54) 2019-04-13 21:25:04,162+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', options=None) from=internal, task_id=dcc08b16-821f-491f-b065-3ca735ac3594 (api:48) 2019-04-13 21:25:04,162+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '1294894592', 'apparentsize': '12884901888'} from=internal, task_id=d3d20781-81c1-4c7a-9699-0d8f5e9b2bc4 (api:54) 2019-04-13 21:25:04,165+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '7263196672', 'apparentsize': '21474836480'} from=internal, task_id=fe09c1ad-d027-4014-bda2-89ae0036cfc8 (api:54) 2019-04-13 21:25:04,166+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '3447286784', 'apparentsize': '16106127360'} from=internal, task_id=503ee849-08d1-4744-84f9-351e3f7b670e (api:54) 2019-04-13 21:25:04,168+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2879759360', 'apparentsize': '8589934592'} from=internal, task_id=dcc08b16-821f-491f-b065-3ca735ac3594 (api:54) 2019-04-13 21:25:06,627+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:25:06,657+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:25:06,658+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:25:06,658+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:25:08,420+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=8bbdb06a-8e35-4356-af3c-c6b5762733b5 (api:48) 2019-04-13 21:25:08,425+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=8bbdb06a-8e35-4356-af3c-c6b5762733b5 (api:54) 2019-04-13 21:25:08,425+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:25:08,438+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=ce947bff-0cda-4933-be3f-4765505f451e (api:48) 2019-04-13 21:25:08,446+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707100987392', 'isoprefix': '', 'alerts': [], 'disktotal': '3898198720512', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707100987392', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719849574400', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707100987392', 'isoprefix': '', 'alerts': [], 'disktotal': '4364489981952', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=ce947bff-0cda-4933-be3f-4765505f451e (api:54) 2019-04-13 21:25:08,447+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:25:14,800+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=c4507277-cf26-4dc3-b079-0a74fe5447be (api:48) 2019-04-13 21:25:14,801+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00072852', 'lastCheck': '0.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000784405', 'lastCheck': '0.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000727115', 'lastCheck': '0.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000429136', 'lastCheck': '0.6', 'valid': True}} from=internal, task_id=c4507277-cf26-4dc3-b079-0a74fe5447be (api:54) 2019-04-13 21:25:14,880+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:25:14,940+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=606b329c-2a63-4d2a-b88e-9b95222124b9 (api:48) 2019-04-13 21:25:14,940+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00072852', 'lastCheck': '0.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000784405', 'lastCheck': '0.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000727115', 'lastCheck': '0.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000429136', 'lastCheck': '0.7', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=606b329c-2a63-4d2a-b88e-9b95222124b9 (api:54) 2019-04-13 21:25:14,941+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=ce7d2343-2ec0-4410-8b51-2d124734995f (api:48) 2019-04-13 21:25:14,941+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=ce7d2343-2ec0-4410-8b51-2d124734995f (api:54) 2019-04-13 21:25:14,953+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '23.37', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '74.51'}, '21': {'cpuUser': '21.98', 'nodeIndex': 1, 'cpuSys': '2.05', 'cpuIdle': '75.97'}, '22': {'cpuUser': '18.07', 'nodeIndex': 1, 'cpuSys': '1.79', 'cpuIdle': '80.14'}, '23': {'cpuUser': '22.44', 'nodeIndex': 1, 'cpuSys': '1.85', 'cpuIdle': '75.71'}, '1': {'cpuUser': '38.07', 'nodeIndex': 0, 'cpuSys': '4.70', 'cpuIdle': '57.23'}, '0': {'cpuUser': '33.04', 'nodeIndex': 0, 'cpuSys': '2.52', 'cpuIdle': '64.44'}, '3': {'cpuUser': '39.72', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '57.96'}, '2': {'cpuUser': '44.36', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '53.39'}, '5': {'cpuUser': '44.09', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '53.79'}, '4': {'cpuUser': '40.19', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '57.69'}, '7': {'cpuUser': '26.22', 'nodeIndex': 1, 'cpuSys': '3.64', 'cpuIdle': '70.14'}, '6': {'cpuUser': '27.87', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '69.02'}, '9': {'cpuUser': '28.00', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '68.89'}, '8': {'cpuUser': '27.87', 'nodeIndex': 1, 'cpuSys': '3.44', 'cpuIdle': '68.69'}, '11': {'cpuUser': '31.58', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '65.57'}, '10': {'cpuUser': '31.51', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '65.64'}, '13': {'cpuUser': '19.80', 'nodeIndex': 0, 'cpuSys': '2.65', 'cpuIdle': '77.55'}, '12': {'cpuUser': '37.41', 'nodeIndex': 0, 'cpuSys': '1.26', 'cpuIdle': '61.33'}, '15': {'cpuUser': '27.48', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '70.80'}, '14': {'cpuUser': '15.96', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '81.59'}, '17': {'cpuUser': '25.42', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '72.73'}, '16': {'cpuUser': '28.93', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '69.48'}, '19': {'cpuUser': '19.33', 'nodeIndex': 1, 'cpuSys': '2.05', 'cpuIdle': '78.62'}, '18': {'cpuUser': '19.20', 'nodeIndex': 1, 'cpuSys': '1.92', 'cpuIdle': '78.88'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69516'}, '0': {'memPercent': 21, 'memFree': '78215'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00072852', 'lastCheck': '0.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000727115', 'lastCheck': '0.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000429136', 'lastCheck': '0.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000784405', 'lastCheck': '0.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168851313', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '5091489102', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590450795220', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '2705598363074', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433853776314', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '259057551008', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239865753', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '366093760', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122793103', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121314600', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423700022576', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '253638526538', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215272399', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680374218', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '29546859442', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652305130', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '29289346166', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255826845495', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '7097926492', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121689153', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75318374775', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '61874983770', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122248083', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418930283', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '1012774609', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121500530', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183514.939191, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183514.939191, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183514.939191, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183514.939191, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121270561', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458744891413', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183514.939191, 'rx': '2576957900924', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322283903266', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183514.939191, 'rx': '1322283903266', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183514.939191, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940834229', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '2700647468008', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571509960991', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '4950895066', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329670882928', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183514.939191, 'rx': '980167466778', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183514.939191, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183514.939191, 'rx': '45357594', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183514.939191, 'rx': '39837014', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1728263.04', 'cpuLoad': '8.06', 'cpuSys': '2.42', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.31', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150389, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:25:14 GMT', 'cpuUser': '28.83', 'memFree': 150133, 'cpuIdle': '68.75', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.32'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:25:14,960+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.08 seconds (__init__:312) 2019-04-13 21:25:17,719+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:25:17,763+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:25:17,764+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:25:17,765+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.05 seconds (__init__:312) 2019-04-13 21:25:18,490+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=82ea03d6-b6a0-498d-b5b1-1f167b7804c1 (api:48) 2019-04-13 21:25:18,495+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=82ea03d6-b6a0-498d-b5b1-1f167b7804c1 (api:54) 2019-04-13 21:25:18,496+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:25:18,509+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=708273ef-7c39-4f36-81b4-576145981ba0 (api:48) 2019-04-13 21:25:18,517+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707096793088', 'isoprefix': '', 'alerts': [], 'disktotal': '3898194526208', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707096793088', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719845380096', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707096793088', 'isoprefix': '', 'alerts': [], 'disktotal': '4364484739072', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=708273ef-7c39-4f36-81b4-576145981ba0 (api:54) 2019-04-13 21:25:18,518+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:25:21,665+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:25:21,695+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:25:21,695+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:25:21,696+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:25:28,562+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=7e3d8616-bcff-4894-9bfd-ff0269509a8f (api:48) 2019-04-13 21:25:28,568+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=7e3d8616-bcff-4894-9bfd-ff0269509a8f (api:54) 2019-04-13 21:25:28,569+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:25:28,586+0200 INFO (jsonrpc/0) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=33f68775-be57-4b80-ac1b-9300d70211ba (api:48) 2019-04-13 21:25:28,595+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707091550208', 'isoprefix': '', 'alerts': [], 'disktotal': '3898189283328', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707091550208', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719840137216', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707091550208', 'isoprefix': '', 'alerts': [], 'disktotal': '4364480544768', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=33f68775-be57-4b80-ac1b-9300d70211ba (api:54) 2019-04-13 21:25:28,597+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:25:29,901+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=b57c4cf7-9121-4f6d-93f6-1768eaa08030 (api:48) 2019-04-13 21:25:29,902+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000503125', 'lastCheck': '5.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000585365', 'lastCheck': '5.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000563203', 'lastCheck': '5.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000387607', 'lastCheck': '2.5', 'valid': True}} from=internal, task_id=b57c4cf7-9121-4f6d-93f6-1768eaa08030 (api:54) 2019-04-13 21:25:29,930+0200 INFO (jsonrpc/6) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:25:29,997+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=8eb03d72-0ba0-48ad-b56a-cc72edad0e63 (api:48) 2019-04-13 21:25:29,998+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000503125', 'lastCheck': '5.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000585365', 'lastCheck': '5.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000563203', 'lastCheck': '5.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000387607', 'lastCheck': '2.6', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=8eb03d72-0ba0-48ad-b56a-cc72edad0e63 (api:54) 2019-04-13 21:25:29,999+0200 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=d660c9cd-fcef-4e35-b52b-9d87bac40f44 (api:48) 2019-04-13 21:25:29,999+0200 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=d660c9cd-fcef-4e35-b52b-9d87bac40f44 (api:54) 2019-04-13 21:25:30,012+0200 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '14.63', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '82.66'}, '21': {'cpuUser': '20.65', 'nodeIndex': 1, 'cpuSys': '2.38', 'cpuIdle': '76.97'}, '22': {'cpuUser': '12.91', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '84.38'}, '23': {'cpuUser': '24.09', 'nodeIndex': 1, 'cpuSys': '1.59', 'cpuIdle': '74.32'}, '1': {'cpuUser': '37.66', 'nodeIndex': 0, 'cpuSys': '3.97', 'cpuIdle': '58.37'}, '0': {'cpuUser': '44.88', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '53.53'}, '3': {'cpuUser': '40.31', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '57.37'}, '2': {'cpuUser': '35.08', 'nodeIndex': 0, 'cpuSys': '2.38', 'cpuIdle': '62.54'}, '5': {'cpuUser': '40.71', 'nodeIndex': 0, 'cpuSys': '1.52', 'cpuIdle': '57.77'}, '4': {'cpuUser': '32.37', 'nodeIndex': 0, 'cpuSys': '2.18', 'cpuIdle': '65.45'}, '7': {'cpuUser': '28.73', 'nodeIndex': 1, 'cpuSys': '3.57', 'cpuIdle': '67.70'}, '6': {'cpuUser': '31.38', 'nodeIndex': 1, 'cpuSys': '3.38', 'cpuIdle': '65.24'}, '9': {'cpuUser': '28.73', 'nodeIndex': 1, 'cpuSys': '3.84', 'cpuIdle': '67.43'}, '8': {'cpuUser': '27.40', 'nodeIndex': 1, 'cpuSys': '4.24', 'cpuIdle': '68.36'}, '11': {'cpuUser': '28.07', 'nodeIndex': 1, 'cpuSys': '3.04', 'cpuIdle': '68.89'}, '10': {'cpuUser': '29.06', 'nodeIndex': 1, 'cpuSys': '3.64', 'cpuIdle': '67.30'}, '13': {'cpuUser': '19.26', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '78.49'}, '12': {'cpuUser': '19.79', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '78.22'}, '15': {'cpuUser': '25.22', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '72.79'}, '14': {'cpuUser': '25.15', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '72.80'}, '17': {'cpuUser': '40.31', 'nodeIndex': 0, 'cpuSys': '1.52', 'cpuIdle': '58.17'}, '16': {'cpuUser': '31.11', 'nodeIndex': 0, 'cpuSys': '1.65', 'cpuIdle': '67.24'}, '19': {'cpuUser': '18.34', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '78.95'}, '18': {'cpuUser': '18.00', 'nodeIndex': 1, 'cpuSys': '1.92', 'cpuIdle': '80.08'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69488'}, '0': {'memPercent': 21, 'memFree': '78244'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000503125', 'lastCheck': '5.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000563203', 'lastCheck': '5.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000387607', 'lastCheck': '2.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000585365', 'lastCheck': '5.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168851639', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '5091489420', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590451145148', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '2705599487296', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433857606216', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '259059586658', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239865863', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '366093760', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122793213', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121314710', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423703805378', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '253640587428', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215272509', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680374328', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '29546859442', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652305240', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '29289346166', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255828887119', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '7099466118', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121689263', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75320462487', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '61879994313', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122248193', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418930703', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '1012774609', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121500640', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183529.996668, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183529.996668, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183529.996668, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183529.996668, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121270671', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458745228567', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183529.996668, 'rx': '2576959073090', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322296387337', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183529.996668, 'rx': '1322296387337', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183529.996668, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940834781', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '2700648591892', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571510310367', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '4950895404', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329679800557', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183529.996668, 'rx': '980177359481', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183529.996668, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183529.996668, 'rx': '45357984', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183529.996668, 'rx': '39837336', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1728278.09', 'cpuLoad': '8.03', 'cpuSys': '2.54', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.11', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150389, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:25:30 GMT', 'cpuUser': '28.09', 'memFree': 150133, 'cpuIdle': '69.37', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.06'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:25:30,018+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:25:32,789+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:25:32,819+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:25:32,820+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:25:32,821+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:25:36,704+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:25:36,732+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:25:36,732+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:25:36,733+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:25:38,650+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=04847d5e-7020-4c4f-8126-8feeae290fb5 (api:48) 2019-04-13 21:25:38,654+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=04847d5e-7020-4c4f-8126-8feeae290fb5 (api:54) 2019-04-13 21:25:38,654+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:25:38,670+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=75fd2181-16dd-4899-8150-c6dd13e36505 (api:48) 2019-04-13 21:25:38,678+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707091550208', 'isoprefix': '', 'alerts': [], 'disktotal': '3898189283328', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707091550208', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719840137216', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707091550208', 'isoprefix': '', 'alerts': [], 'disktotal': '4364480544768', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=75fd2181-16dd-4899-8150-c6dd13e36505 (api:54) 2019-04-13 21:25:38,679+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:25:44,976+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:25:45,035+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=f86566a9-82f3-4e53-af7f-8a66c572ff41 (api:48) 2019-04-13 21:25:45,036+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000710776', 'lastCheck': '0.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000618291', 'lastCheck': '0.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000670107', 'lastCheck': '0.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000335129', 'lastCheck': '0.8', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=f86566a9-82f3-4e53-af7f-8a66c572ff41 (api:54) 2019-04-13 21:25:45,037+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=b3a97612-06fc-4c4a-8430-1cae2c547308 (api:48) 2019-04-13 21:25:45,037+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=b3a97612-06fc-4c4a-8430-1cae2c547308 (api:54) 2019-04-13 21:25:45,062+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=56220429-9462-4548-8127-42d0b8d27a20 (api:48) 2019-04-13 21:25:45,063+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000710776', 'lastCheck': '0.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000618291', 'lastCheck': '0.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000670107', 'lastCheck': '0.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000335129', 'lastCheck': '0.9', 'valid': True}} from=internal, task_id=56220429-9462-4548-8127-42d0b8d27a20 (api:54) 2019-04-13 21:25:45,082+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '20.06', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '77.49'}, '21': {'cpuUser': '16.88', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '80.14'}, '22': {'cpuUser': '15.29', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '82.00'}, '23': {'cpuUser': '20.39', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '77.29'}, '1': {'cpuUser': '47.20', 'nodeIndex': 0, 'cpuSys': '4.57', 'cpuIdle': '48.23'}, '0': {'cpuUser': '35.29', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '62.26'}, '3': {'cpuUser': '60.97', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '37.37'}, '2': {'cpuUser': '44.55', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '53.53'}, '5': {'cpuUser': '52.70', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '45.38'}, '4': {'cpuUser': '43.89', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '54.06'}, '7': {'cpuUser': '28.20', 'nodeIndex': 1, 'cpuSys': '4.30', 'cpuIdle': '67.50'}, '6': {'cpuUser': '31.64', 'nodeIndex': 1, 'cpuSys': '3.64', 'cpuIdle': '64.72'}, '9': {'cpuUser': '29.99', 'nodeIndex': 1, 'cpuSys': '3.77', 'cpuIdle': '66.24'}, '8': {'cpuUser': '32.57', 'nodeIndex': 1, 'cpuSys': '3.64', 'cpuIdle': '63.79'}, '11': {'cpuUser': '31.84', 'nodeIndex': 1, 'cpuSys': '3.24', 'cpuIdle': '64.92'}, '10': {'cpuUser': '31.58', 'nodeIndex': 1, 'cpuSys': '3.97', 'cpuIdle': '64.45'}, '13': {'cpuUser': '21.18', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '76.37'}, '12': {'cpuUser': '42.37', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '55.78'}, '15': {'cpuUser': '18.54', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '79.34'}, '14': {'cpuUser': '35.09', 'nodeIndex': 0, 'cpuSys': '1.52', 'cpuIdle': '63.39'}, '17': {'cpuUser': '25.22', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '72.99'}, '16': {'cpuUser': '35.75', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '62.40'}, '19': {'cpuUser': '20.13', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '77.02'}, '18': {'cpuUser': '17.87', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '79.81'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69493'}, '0': {'memPercent': 21, 'memFree': '78237'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000710776', 'lastCheck': '0.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000670107', 'lastCheck': '0.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000335129', 'lastCheck': '0.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000618291', 'lastCheck': '0.8', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168852492', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '5091489596', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590452143844', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '2705606201815', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433861123850', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '259061721345', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239866480', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '366093760', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122793830', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121315327', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423707523526', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '253642580879', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215273126', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680374945', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '29546859442', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652305693', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '29289346330', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255831046433', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '7102647088', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121689880', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75322855193', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '61884221301', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122248810', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418931183', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '1012774609', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121501257', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183545.034525, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183545.034525, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183545.034525, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183545.034525, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121271288', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458746234435', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183545.034525, 'rx': '2576965398031', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322307783264', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183545.034525, 'rx': '1322307783264', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183545.034525, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940834867', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '2700655305593', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571511308977', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '4950896222', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329688953300', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183545.034525, 'rx': '980187260437', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183545.034525, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183545.034525, 'rx': '45358374', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183545.034525, 'rx': '39837704', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1728293.14', 'cpuLoad': '7.96', 'cpuSys': '2.69', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '2.85', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150388, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:25:45 GMT', 'cpuUser': '31.63', 'memFree': 150132, 'cpuIdle': '65.68', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.32'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:25:45,088+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.11 seconds (__init__:312) 2019-04-13 21:25:47,868+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:25:47,907+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:25:47,908+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:25:47,909+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:25:48,733+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=4073bab3-9065-4f74-80bc-712ea5bec636 (api:48) 2019-04-13 21:25:48,737+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=4073bab3-9065-4f74-80bc-712ea5bec636 (api:54) 2019-04-13 21:25:48,738+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:25:48,845+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=0dfd2ba6-59b5-4638-9423-e250f453f847 (api:48) 2019-04-13 21:25:48,854+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707087355904', 'isoprefix': '', 'alerts': [], 'disktotal': '3898185089024', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707087355904', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719835942912', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707087355904', 'isoprefix': '', 'alerts': [], 'disktotal': '4364476350464', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=0dfd2ba6-59b5-4638-9423-e250f453f847 (api:54) 2019-04-13 21:25:48,856+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:25:51,741+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:25:51,779+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:25:51,780+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:25:51,781+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:25:57,114+0200 WARN (qgapoller/9) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:25:58,908+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=7f9b3c66-3f24-4431-a8ba-ba235415565c (api:48) 2019-04-13 21:25:58,913+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=7f9b3c66-3f24-4431-a8ba-ba235415565c (api:54) 2019-04-13 21:25:58,914+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:25:58,936+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=ac980fb9-cda3-446e-b821-ba16aac4798e (api:48) 2019-04-13 21:25:58,946+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707087355904', 'isoprefix': '', 'alerts': [], 'disktotal': '3898185089024', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707087355904', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719835942912', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707087355904', 'isoprefix': '', 'alerts': [], 'disktotal': '4364476350464', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=ac980fb9-cda3-446e-b821-ba16aac4798e (api:54) 2019-04-13 21:25:58,947+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:26:00,043+0200 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:26:00,237+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=12c7e75e-c9a1-4ec8-8816-469881437d46 (api:48) 2019-04-13 21:26:00,237+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000646285', 'lastCheck': '5.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000507952', 'lastCheck': '6.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000692515', 'lastCheck': '5.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000301674', 'lastCheck': '2.8', 'valid': True}} from=internal, task_id=12c7e75e-c9a1-4ec8-8816-469881437d46 (api:54) 2019-04-13 21:26:00,265+0200 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=e55ffb6a-27d8-431e-abb9-ff83fcdf5efa (api:48) 2019-04-13 21:26:00,266+0200 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000646285', 'lastCheck': '5.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000507952', 'lastCheck': '6.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000692515', 'lastCheck': '5.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000301674', 'lastCheck': '2.8', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=e55ffb6a-27d8-431e-abb9-ff83fcdf5efa (api:54) 2019-04-13 21:26:00,268+0200 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=05bc979d-423f-4b78-ac13-b10708133475 (api:48) 2019-04-13 21:26:00,269+0200 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=05bc979d-423f-4b78-ac13-b10708133475 (api:54) 2019-04-13 21:26:00,286+0200 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '20.06', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '77.49'}, '21': {'cpuUser': '16.88', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '80.14'}, '22': {'cpuUser': '15.29', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '82.00'}, '23': {'cpuUser': '20.39', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '77.29'}, '1': {'cpuUser': '47.20', 'nodeIndex': 0, 'cpuSys': '4.57', 'cpuIdle': '48.23'}, '0': {'cpuUser': '35.29', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '62.26'}, '3': {'cpuUser': '60.97', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '37.37'}, '2': {'cpuUser': '44.55', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '53.53'}, '5': {'cpuUser': '52.70', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '45.38'}, '4': {'cpuUser': '43.89', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '54.06'}, '7': {'cpuUser': '28.20', 'nodeIndex': 1, 'cpuSys': '4.30', 'cpuIdle': '67.50'}, '6': {'cpuUser': '31.64', 'nodeIndex': 1, 'cpuSys': '3.64', 'cpuIdle': '64.72'}, '9': {'cpuUser': '29.99', 'nodeIndex': 1, 'cpuSys': '3.77', 'cpuIdle': '66.24'}, '8': {'cpuUser': '32.57', 'nodeIndex': 1, 'cpuSys': '3.64', 'cpuIdle': '63.79'}, '11': {'cpuUser': '31.84', 'nodeIndex': 1, 'cpuSys': '3.24', 'cpuIdle': '64.92'}, '10': {'cpuUser': '31.58', 'nodeIndex': 1, 'cpuSys': '3.97', 'cpuIdle': '64.45'}, '13': {'cpuUser': '21.18', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '76.37'}, '12': {'cpuUser': '42.37', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '55.78'}, '15': {'cpuUser': '18.54', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '79.34'}, '14': {'cpuUser': '35.09', 'nodeIndex': 0, 'cpuSys': '1.52', 'cpuIdle': '63.39'}, '17': {'cpuUser': '25.22', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '72.99'}, '16': {'cpuUser': '35.75', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '62.40'}, '19': {'cpuUser': '20.13', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '77.02'}, '18': {'cpuUser': '17.87', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '79.81'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69493'}, '0': {'memPercent': 21, 'memFree': '78237'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000646285', 'lastCheck': '5.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000692515', 'lastCheck': '5.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000301674', 'lastCheck': '2.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000507952', 'lastCheck': '6.0', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168852492', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '5091489596', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590452749370', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '2705607756048', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433865016834', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '259064133456', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239866480', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '366093760', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122793830', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121315327', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423711181190', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '253644892453', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215273126', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680374945', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '29546859442', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652305693', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '29289346330', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255833491506', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '7104589365', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121689880', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75325508129', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '61889764293', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122248810', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418931663', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '1012774609', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121501257', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183560.262197, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183560.262197, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183560.262197, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183560.262197, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121271288', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458746771857', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183560.262197, 'rx': '2576966881701', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322320265491', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183560.262197, 'rx': '1322320265491', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183560.262197, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940834995', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '2700655759091', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571511914375', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '4950896222', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329699594551', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183560.262197, 'rx': '980197770245', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183560.262197, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183560.262197, 'rx': '45358764', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183560.262197, 'rx': '39838072', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1728308.20', 'cpuLoad': '7.96', 'cpuSys': '2.69', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '2.85', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150387, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:26:00 GMT', 'cpuUser': '31.63', 'memFree': 150131, 'cpuIdle': '65.68', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.32'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:26:00,293+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.25 seconds (__init__:312) 2019-04-13 21:26:02,949+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:26:02,980+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:26:02,981+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:26:02,982+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:26:04,111+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID=u'ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', options=None) from=internal, task_id=c6cf8914-1c0a-43d4-af79-dcd6f602870b (api:48) 2019-04-13 21:26:04,113+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', options=None) from=internal, task_id=f2d9dd55-dc93-4610-a778-646593c27694 (api:48) 2019-04-13 21:26:04,114+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', options=None) from=internal, task_id=7139c710-6dd4-459e-bac2-e7e8bb9cece9 (api:48) 2019-04-13 21:26:04,117+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '4579684864', 'apparentsize': '16106127360'} from=internal, task_id=f2d9dd55-dc93-4610-a778-646593c27694 (api:54) 2019-04-13 21:26:04,117+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID='ef6741d1-380a-4736-a924-f88ed68e1270', options=None) from=internal, task_id=285bbc13-e0a3-4fff-ae81-9e3fc60e0249 (api:48) 2019-04-13 21:26:04,119+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '5697701376', 'apparentsize': '16106127360'} from=internal, task_id=7139c710-6dd4-459e-bac2-e7e8bb9cece9 (api:54) 2019-04-13 21:26:04,120+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID=u'67a4dd71-69ff-49a8-b8a4-c7101c75e105', options=None) from=internal, task_id=e0677239-6d3d-47fb-9533-000580afec83 (api:48) 2019-04-13 21:26:04,121+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '53760914944', 'apparentsize': '139586437120'} from=internal, task_id=c6cf8914-1c0a-43d4-af79-dcd6f602870b (api:54) 2019-04-13 21:26:04,122+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', options=None) from=internal, task_id=ab31e486-78e2-4f66-ae94-0235a290ab8f (api:48) 2019-04-13 21:26:04,124+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '6477472768', 'apparentsize': '21474836480'} from=internal, task_id=285bbc13-e0a3-4fff-ae81-9e3fc60e0249 (api:54) 2019-04-13 21:26:04,124+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID=u'2b136ed7-8b9a-45c1-9edc-64ee96c87c43', options=None) from=internal, task_id=13efd57a-0522-4b7c-9076-fd797df05971 (api:48) 2019-04-13 21:26:04,126+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID=u'44b3ceca-f0db-43da-af29-aa681df146ac', options=None) from=internal, task_id=490b0563-cbb1-4010-979f-ff07291ff8e6 (api:48) 2019-04-13 21:26:04,128+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '55059120640', 'apparentsize': '171798691840'} from=internal, task_id=e0677239-6d3d-47fb-9533-000580afec83 (api:54) 2019-04-13 21:26:04,129+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=f48f8063-5aa4-48c2-8d56-f26a5fcad99e (api:48) 2019-04-13 21:26:04,130+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2524568064', 'apparentsize': '12884901888'} from=internal, task_id=ab31e486-78e2-4f66-ae94-0235a290ab8f (api:54) 2019-04-13 21:26:04,130+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID='123e2cc8-8f24-4ddd-b401-68393535f555', options=None) from=internal, task_id=0673e280-6112-4b06-99ad-1730c06ee975 (api:48) 2019-04-13 21:26:04,131+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2024405504', 'apparentsize': '10737418240'} from=internal, task_id=13efd57a-0522-4b7c-9076-fd797df05971 (api:54) 2019-04-13 21:26:04,131+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID='3f6c31c6-ec38-4979-a617-55537123ee61', options=None) from=internal, task_id=587dba2e-2f84-4048-b54d-cb14c4896f25 (api:48) 2019-04-13 21:26:04,133+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '11196683776', 'apparentsize': '21474836480'} from=internal, task_id=490b0563-cbb1-4010-979f-ff07291ff8e6 (api:54) 2019-04-13 21:26:04,134+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', options=None) from=internal, task_id=8930b669-bc9c-44f2-892c-a22f69a4b2a3 (api:48) 2019-04-13 21:26:04,136+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2116486656', 'apparentsize': '3948150784'} from=internal, task_id=f48f8063-5aa4-48c2-8d56-f26a5fcad99e (api:54) 2019-04-13 21:26:04,137+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID=u'a1d583c1-e135-4e3f-b6fa-0f29a5261366', options=None) from=internal, task_id=cbd4082f-ece4-47ba-bf9e-7cc091fa23ab (api:48) 2019-04-13 21:26:04,137+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2283003904', 'apparentsize': '16106127360'} from=internal, task_id=0673e280-6112-4b06-99ad-1730c06ee975 (api:54) 2019-04-13 21:26:04,138+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', options=None) from=internal, task_id=ae9be6f4-cc62-4efc-a6ce-3e57827be64b (api:48) 2019-04-13 21:26:04,140+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2109410304', 'apparentsize': '8589934592'} from=internal, task_id=8930b669-bc9c-44f2-892c-a22f69a4b2a3 (api:54) 2019-04-13 21:26:04,141+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID='a884b4ce-381f-4653-be14-a31d15181f48', options=None) from=internal, task_id=8b1d10ca-fc80-4fd5-b72f-cce621921640 (api:48) 2019-04-13 21:26:04,141+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '4998716928', 'apparentsize': '16106127360'} from=internal, task_id=587dba2e-2f84-4048-b54d-cb14c4896f25 (api:54) 2019-04-13 21:26:04,142+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', options=None) from=internal, task_id=3b31ff4e-42d3-471e-9dc1-da1058fe3050 (api:48) 2019-04-13 21:26:04,144+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '1294894592', 'apparentsize': '12884901888'} from=internal, task_id=cbd4082f-ece4-47ba-bf9e-7cc091fa23ab (api:54) 2019-04-13 21:26:04,144+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '7263196672', 'apparentsize': '21474836480'} from=internal, task_id=ae9be6f4-cc62-4efc-a6ce-3e57827be64b (api:54) 2019-04-13 21:26:04,146+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '3447292416', 'apparentsize': '16106127360'} from=internal, task_id=8b1d10ca-fc80-4fd5-b72f-cce621921640 (api:54) 2019-04-13 21:26:04,146+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2879759360', 'apparentsize': '8589934592'} from=internal, task_id=3b31ff4e-42d3-471e-9dc1-da1058fe3050 (api:54) 2019-04-13 21:26:06,789+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:26:06,818+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:26:06,818+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:26:06,819+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:26:07,122+0200 WARN (qgapoller/5) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:26:08,994+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=95976673-6375-4201-b101-98a4e6dbc698 (api:48) 2019-04-13 21:26:08,998+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=95976673-6375-4201-b101-98a4e6dbc698 (api:54) 2019-04-13 21:26:08,998+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:26:09,015+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=c3b83071-43ec-461b-a5a7-346d16fb0455 (api:48) 2019-04-13 21:26:09,022+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707082113024', 'isoprefix': '', 'alerts': [], 'disktotal': '3898179846144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707082113024', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719830700032', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707082113024', 'isoprefix': '', 'alerts': [], 'disktotal': '4364470059008', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=c3b83071-43ec-461b-a5a7-346d16fb0455 (api:54) 2019-04-13 21:26:09,024+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:26:15,086+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:26:15,164+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=96c74e0d-5d08-4247-9350-f91bdf3a8faf (api:48) 2019-04-13 21:26:15,165+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000601599', 'lastCheck': '0.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000692089', 'lastCheck': '0.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000732285', 'lastCheck': '0.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000431255', 'lastCheck': '1.0', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=96c74e0d-5d08-4247-9350-f91bdf3a8faf (api:54) 2019-04-13 21:26:15,165+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=0b231758-216e-4d38-80a9-34547d1662e9 (api:48) 2019-04-13 21:26:15,166+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=0b231758-216e-4d38-80a9-34547d1662e9 (api:54) 2019-04-13 21:26:15,182+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '28.64', 'nodeIndex': 1, 'cpuSys': '1.98', 'cpuIdle': '69.38'}, '21': {'cpuUser': '20.98', 'nodeIndex': 1, 'cpuSys': '2.38', 'cpuIdle': '76.64'}, '22': {'cpuUser': '21.38', 'nodeIndex': 1, 'cpuSys': '2.38', 'cpuIdle': '76.24'}, '23': {'cpuUser': '21.97', 'nodeIndex': 1, 'cpuSys': '2.51', 'cpuIdle': '75.52'}, '1': {'cpuUser': '50.28', 'nodeIndex': 0, 'cpuSys': '3.63', 'cpuIdle': '46.09'}, '0': {'cpuUser': '43.22', 'nodeIndex': 0, 'cpuSys': '1.98', 'cpuIdle': '54.80'}, '3': {'cpuUser': '58.53', 'nodeIndex': 0, 'cpuSys': '1.39', 'cpuIdle': '40.08'}, '2': {'cpuUser': '49.29', 'nodeIndex': 0, 'cpuSys': '1.58', 'cpuIdle': '49.13'}, '5': {'cpuUser': '56.28', 'nodeIndex': 0, 'cpuSys': '1.52', 'cpuIdle': '42.20'}, '4': {'cpuUser': '55.36', 'nodeIndex': 0, 'cpuSys': '1.45', 'cpuIdle': '43.19'}, '7': {'cpuUser': '29.96', 'nodeIndex': 1, 'cpuSys': '3.70', 'cpuIdle': '66.34'}, '6': {'cpuUser': '36.36', 'nodeIndex': 1, 'cpuSys': '3.04', 'cpuIdle': '60.60'}, '9': {'cpuUser': '31.01', 'nodeIndex': 1, 'cpuSys': '3.23', 'cpuIdle': '65.76'}, '8': {'cpuUser': '31.34', 'nodeIndex': 1, 'cpuSys': '3.70', 'cpuIdle': '64.96'}, '11': {'cpuUser': '31.01', 'nodeIndex': 1, 'cpuSys': '3.43', 'cpuIdle': '65.56'}, '10': {'cpuUser': '32.60', 'nodeIndex': 1, 'cpuSys': '3.23', 'cpuIdle': '64.17'}, '13': {'cpuUser': '27.58', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '70.37'}, '12': {'cpuUser': '41.90', 'nodeIndex': 0, 'cpuSys': '1.32', 'cpuIdle': '56.78'}, '15': {'cpuUser': '19.07', 'nodeIndex': 0, 'cpuSys': '3.30', 'cpuIdle': '77.63'}, '14': {'cpuUser': '37.08', 'nodeIndex': 0, 'cpuSys': '1.98', 'cpuIdle': '60.94'}, '17': {'cpuUser': '20.59', 'nodeIndex': 0, 'cpuSys': '2.11', 'cpuIdle': '77.30'}, '16': {'cpuUser': '22.43', 'nodeIndex': 0, 'cpuSys': '2.18', 'cpuIdle': '75.39'}, '19': {'cpuUser': '19.00', 'nodeIndex': 1, 'cpuSys': '2.97', 'cpuIdle': '78.03'}, '18': {'cpuUser': '22.57', 'nodeIndex': 1, 'cpuSys': '2.11', 'cpuIdle': '75.32'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69573'}, '0': {'memPercent': 21, 'memFree': '78156'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000601599', 'lastCheck': '0.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000732285', 'lastCheck': '0.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000431255', 'lastCheck': '1.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000692089', 'lastCheck': '0.9', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168852612', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '5091489596', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590454040740', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '2705607855965', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433868844432', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '259066086646', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239866936', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '366094158', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122793950', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121315447', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423714963218', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '253647145849', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215273246', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680375173', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '29546859539', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652305910', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '29289346438', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255835479206', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '7106486207', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121690000', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75327490867', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '61889781513', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122248930', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418932083', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '1012774609', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121501377', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183575.163277, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183575.163277, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183575.163277, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183575.163277, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121271408', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458748015491', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183575.163277, 'rx': '2576966971725', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322329579114', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183575.163277, 'rx': '1322329579114', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183575.163277, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940835123', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '2700656959045', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571513205617', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '4950896920', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329704028383', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183575.163277, 'rx': '980208619779', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183575.163277, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183575.163277, 'rx': '45359154', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183575.163277, 'rx': '39838394', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1728323.25', 'cpuLoad': '8.01', 'cpuSys': '2.46', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.43', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150385, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:26:15 GMT', 'cpuUser': '33.68', 'memFree': 150129, 'cpuIdle': '63.86', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.25'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:26:15,188+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.10 seconds (__init__:312) 2019-04-13 21:26:15,361+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=c3c51a17-b6e0-48f1-a47c-cfbd671490a5 (api:48) 2019-04-13 21:26:15,362+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000601599', 'lastCheck': '1.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000692089', 'lastCheck': '1.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000732285', 'lastCheck': '1.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000431255', 'lastCheck': '1.2', 'valid': True}} from=internal, task_id=c3c51a17-b6e0-48f1-a47c-cfbd671490a5 (api:54) 2019-04-13 21:26:17,107+0200 WARN (qgapoller/9) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:26:18,018+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:26:18,040+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:26:18,041+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:26:18,042+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.02 seconds (__init__:312) 2019-04-13 21:26:19,072+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=b057719f-9fe8-4197-8bcd-a129f5a5359b (api:48) 2019-04-13 21:26:19,076+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=b057719f-9fe8-4197-8bcd-a129f5a5359b (api:54) 2019-04-13 21:26:19,077+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:26:19,094+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=48e70f3e-23db-4ff0-bba7-94460080d883 (api:48) 2019-04-13 21:26:19,102+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707082113024', 'isoprefix': '', 'alerts': [], 'disktotal': '3898179846144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707082113024', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719830700032', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707082113024', 'isoprefix': '', 'alerts': [], 'disktotal': '4364470059008', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=48e70f3e-23db-4ff0-bba7-94460080d883 (api:54) 2019-04-13 21:26:19,103+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:26:21,827+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:26:21,867+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:26:21,867+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:26:21,868+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:26:23,741+0200 INFO (itmap/0) [IOProcessClient] (/onode-1.discovery.intern:_mnt_store) Starting client (__init__:308) 2019-04-13 21:26:23,770+0200 INFO (ioprocess/28888) [IOProcess] (/onode-1.discovery.intern:_mnt_store) Starting ioprocess (__init__:434) 2019-04-13 21:26:23,771+0200 INFO (itmap/1) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_vms) Starting client (__init__:308) 2019-04-13 21:26:23,798+0200 INFO (itmap/2) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_isos) Starting client (__init__:308) 2019-04-13 21:26:23,807+0200 INFO (ioprocess/28905) [IOProcess] (/nas.discovery.intern:_mnt_d2600_vms) Starting ioprocess (__init__:434) 2019-04-13 21:26:23,833+0200 INFO (itmap/3) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting client (__init__:308) 2019-04-13 21:26:23,842+0200 INFO (ioprocess/28915) [IOProcess] (/nas.discovery.intern:_mnt_d2600_isos) Starting ioprocess (__init__:434) 2019-04-13 21:26:23,868+0200 INFO (itmap/4) [IOProcessClient] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting client (__init__:308) 2019-04-13 21:26:23,876+0200 INFO (ioprocess/28924) [IOProcess] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting ioprocess (__init__:434) 2019-04-13 21:26:23,909+0200 INFO (ioprocess/28933) [IOProcess] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting ioprocess (__init__:434) 2019-04-13 21:26:23,912+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:26:23,912+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:26:23,913+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:26:27,106+0200 WARN (qgapoller/10) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:26:29,150+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=0f942638-6cc5-4c34-9efb-5ee73fb23ece (api:48) 2019-04-13 21:26:29,155+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=0f942638-6cc5-4c34-9efb-5ee73fb23ece (api:54) 2019-04-13 21:26:29,156+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:26:29,276+0200 INFO (jsonrpc/0) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=57618b07-1c75-47b9-a7ce-64ab904bca0c (api:48) 2019-04-13 21:26:29,284+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707074772992', 'isoprefix': '', 'alerts': [], 'disktotal': '3898172506112', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707074772992', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719823360000', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707074772992', 'isoprefix': '', 'alerts': [], 'disktotal': '4364463767552', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=57618b07-1c75-47b9-a7ce-64ab904bca0c (api:54) 2019-04-13 21:26:29,285+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:26:30,153+0200 INFO (jsonrpc/6) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:26:30,238+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=8052995e-9249-45aa-9516-9d618a67d909 (api:48) 2019-04-13 21:26:30,239+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000702504', 'lastCheck': '5.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000883898', 'lastCheck': '6.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000651782', 'lastCheck': '5.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000397336', 'lastCheck': '2.8', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=8052995e-9249-45aa-9516-9d618a67d909 (api:54) 2019-04-13 21:26:30,240+0200 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=e9bbff9a-a51b-482d-af8b-5d24176b9b13 (api:48) 2019-04-13 21:26:30,240+0200 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=e9bbff9a-a51b-482d-af8b-5d24176b9b13 (api:54) 2019-04-13 21:26:30,258+0200 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '16.31', 'nodeIndex': 1, 'cpuSys': '2.24', 'cpuIdle': '81.45'}, '21': {'cpuUser': '13.81', 'nodeIndex': 1, 'cpuSys': '3.02', 'cpuIdle': '83.17'}, '22': {'cpuUser': '17.16', 'nodeIndex': 1, 'cpuSys': '2.63', 'cpuIdle': '80.21'}, '23': {'cpuUser': '8.15', 'nodeIndex': 1, 'cpuSys': '3.22', 'cpuIdle': '88.63'}, '1': {'cpuUser': '37.88', 'nodeIndex': 0, 'cpuSys': '4.41', 'cpuIdle': '57.71'}, '0': {'cpuUser': '46.16', 'nodeIndex': 0, 'cpuSys': '1.84', 'cpuIdle': '52.00'}, '3': {'cpuUser': '46.62', 'nodeIndex': 0, 'cpuSys': '1.45', 'cpuIdle': '51.93'}, '2': {'cpuUser': '49.52', 'nodeIndex': 0, 'cpuSys': '1.51', 'cpuIdle': '48.97'}, '5': {'cpuUser': '46.49', 'nodeIndex': 0, 'cpuSys': '1.64', 'cpuIdle': '51.87'}, '4': {'cpuUser': '48.60', 'nodeIndex': 0, 'cpuSys': '1.38', 'cpuIdle': '50.02'}, '7': {'cpuUser': '25.05', 'nodeIndex': 1, 'cpuSys': '4.21', 'cpuIdle': '70.74'}, '6': {'cpuUser': '24.40', 'nodeIndex': 1, 'cpuSys': '3.49', 'cpuIdle': '72.11'}, '9': {'cpuUser': '26.76', 'nodeIndex': 1, 'cpuSys': '3.75', 'cpuIdle': '69.49'}, '8': {'cpuUser': '22.75', 'nodeIndex': 1, 'cpuSys': '4.21', 'cpuIdle': '73.04'}, '11': {'cpuUser': '25.51', 'nodeIndex': 1, 'cpuSys': '4.41', 'cpuIdle': '70.08'}, '10': {'cpuUser': '26.37', 'nodeIndex': 1, 'cpuSys': '4.08', 'cpuIdle': '69.55'}, '13': {'cpuUser': '36.82', 'nodeIndex': 0, 'cpuSys': '1.32', 'cpuIdle': '61.86'}, '12': {'cpuUser': '23.28', 'nodeIndex': 0, 'cpuSys': '1.64', 'cpuIdle': '75.08'}, '15': {'cpuUser': '36.43', 'nodeIndex': 0, 'cpuSys': '1.18', 'cpuIdle': '62.39'}, '14': {'cpuUser': '29.53', 'nodeIndex': 0, 'cpuSys': '1.64', 'cpuIdle': '68.83'}, '17': {'cpuUser': '25.25', 'nodeIndex': 0, 'cpuSys': '1.78', 'cpuIdle': '72.97'}, '16': {'cpuUser': '38.53', 'nodeIndex': 0, 'cpuSys': '1.05', 'cpuIdle': '60.42'}, '19': {'cpuUser': '14.01', 'nodeIndex': 1, 'cpuSys': '2.56', 'cpuIdle': '83.43'}, '18': {'cpuUser': '24.73', 'nodeIndex': 1, 'cpuSys': '1.91', 'cpuIdle': '73.36'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69586'}, '0': {'memPercent': 21, 'memFree': '78141'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000702504', 'lastCheck': '5.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000651782', 'lastCheck': '5.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000397336', 'lastCheck': '2.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000883898', 'lastCheck': '6.0', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168852896', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '5091489596', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590454680370', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '2705609004071', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433872738996', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '259068705216', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239867220', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '366094158', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122794234', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121315731', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423718869376', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '253649374283', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215273530', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680375457', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '29546859539', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652306030', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '29289346602', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255838147640', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '7110145305', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121690284', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75329849651', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '61896898212', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122249214', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418932861', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '1012774933', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121501661', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183590.236951, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183590.236951, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183590.236951, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183590.236951, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121271692', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458748659883', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183590.236951, 'rx': '2576968067675', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322343095409', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183590.236951, 'rx': '1322343095409', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183590.236951, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940835209', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '2700658107023', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571513845161', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '4950897048', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329717451703', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183590.236951, 'rx': '980219162971', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183590.236951, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183590.236951, 'rx': '45359544', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183590.236951, 'rx': '39838762', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40566', 'ksmPages': 100, 'elapsedTime': '1728338.32', 'cpuLoad': '8.01', 'cpuSys': '2.53', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.75', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150382, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:26:30 GMT', 'cpuUser': '29.59', 'memFree': 150126, 'cpuIdle': '67.88', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.58'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:26:30,265+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.12 seconds (__init__:312) 2019-04-13 21:26:30,444+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=30fb5d8e-f734-4080-a373-6a30d8b9a0bb (api:48) 2019-04-13 21:26:30,445+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000702504', 'lastCheck': '6.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000883898', 'lastCheck': '6.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000651782', 'lastCheck': '6.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000397336', 'lastCheck': '3.0', 'valid': True}} from=internal, task_id=30fb5d8e-f734-4080-a373-6a30d8b9a0bb (api:54) 2019-04-13 21:26:33,067+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:26:33,103+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:26:33,104+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:26:33,105+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:26:36,877+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:26:36,909+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:26:36,910+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:26:36,912+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:26:37,105+0200 WARN (qgapoller/5) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:26:39,340+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=6b9a917f-42d7-4922-8d0e-28fc820ddc23 (api:48) 2019-04-13 21:26:39,345+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=6b9a917f-42d7-4922-8d0e-28fc820ddc23 (api:54) 2019-04-13 21:26:39,345+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:26:39,358+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=dc67646f-6f52-4f95-ae0b-737af5a9cba3 (api:48) 2019-04-13 21:26:39,368+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707074772992', 'isoprefix': '', 'alerts': [], 'disktotal': '3898172506112', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707074772992', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719823360000', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707074772992', 'isoprefix': '', 'alerts': [], 'disktotal': '4364463767552', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=dc67646f-6f52-4f95-ae0b-737af5a9cba3 (api:54) 2019-04-13 21:26:39,370+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:26:45,226+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:26:45,293+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=b8369f1c-085c-4cf8-9050-e05f6a6deede (api:48) 2019-04-13 21:26:45,294+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000634236', 'lastCheck': '1.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000650276', 'lastCheck': '1.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000637033', 'lastCheck': '1.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000371082', 'lastCheck': '1.1', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=b8369f1c-085c-4cf8-9050-e05f6a6deede (api:54) 2019-04-13 21:26:45,295+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=16496571-be99-48af-94d6-bd673e53d0c9 (api:48) 2019-04-13 21:26:45,295+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=16496571-be99-48af-94d6-bd673e53d0c9 (api:54) 2019-04-13 21:26:45,309+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '19.41', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '77.87'}, '21': {'cpuUser': '24.97', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '72.71'}, '22': {'cpuUser': '21.06', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '76.62'}, '23': {'cpuUser': '22.45', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '75.10'}, '1': {'cpuUser': '44.51', 'nodeIndex': 0, 'cpuSys': '4.31', 'cpuIdle': '51.18'}, '0': {'cpuUser': '42.86', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '55.09'}, '3': {'cpuUser': '47.69', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '50.59'}, '2': {'cpuUser': '49.48', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '48.86'}, '5': {'cpuUser': '51.40', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '47.01'}, '4': {'cpuUser': '49.74', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '48.34'}, '7': {'cpuUser': '31.93', 'nodeIndex': 1, 'cpuSys': '3.51', 'cpuIdle': '64.56'}, '6': {'cpuUser': '34.38', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '62.31'}, '9': {'cpuUser': '39.28', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '57.61'}, '8': {'cpuUser': '36.23', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '60.66'}, '11': {'cpuUser': '34.51', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '62.31'}, '10': {'cpuUser': '33.65', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '63.04'}, '13': {'cpuUser': '24.64', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '73.24'}, '12': {'cpuUser': '35.64', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '62.77'}, '15': {'cpuUser': '37.62', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '60.79'}, '14': {'cpuUser': '35.90', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '62.38'}, '17': {'cpuUser': '30.47', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '67.28'}, '16': {'cpuUser': '27.29', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '71.05'}, '19': {'cpuUser': '26.89', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '70.92'}, '18': {'cpuUser': '27.29', 'nodeIndex': 1, 'cpuSys': '1.99', 'cpuIdle': '70.72'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69586'}, '0': {'memPercent': 21, 'memFree': '78139'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000634236', 'lastCheck': '1.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000637033', 'lastCheck': '1.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000371082', 'lastCheck': '1.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000650276', 'lastCheck': '1.0', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168853323', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '5091489998', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590455083930', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '2705610132684', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433876753948', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '259070936491', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239867311', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '366094158', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122794325', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121315822', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423722903600', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '253651943823', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215273621', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680375548', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '29546859539', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652306121', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '29289346602', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255840416969', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '7112094200', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121690375', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75331619793', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '61901118708', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122249305', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418933281', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '1012774933', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121501752', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183605.292651, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183605.292651, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183605.292651, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183605.292651, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121271783', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458749051977', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183605.292651, 'rx': '2576969145252', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322353465920', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183605.292651, 'rx': '1322353465920', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183605.292651, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940835667', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '2700659234938', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571514248263', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '4950897746', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329726502196', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183605.292651, 'rx': '980229221913', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183605.292651, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183605.292651, 'rx': '45360011', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183605.292651, 'rx': '39839084', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40568', 'ksmPages': 100, 'elapsedTime': '1728353.39', 'cpuLoad': '8.01', 'cpuSys': '2.41', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.25', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150381, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:26:45 GMT', 'cpuUser': '34.55', 'memFree': 150125, 'cpuIdle': '63.03', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.66'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:26:45,315+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:26:45,523+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=a419b105-1659-4f66-8ba2-098871ba3bac (api:48) 2019-04-13 21:26:45,524+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000634236', 'lastCheck': '1.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000650276', 'lastCheck': '1.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000637033', 'lastCheck': '1.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000371082', 'lastCheck': '1.3', 'valid': True}} from=internal, task_id=a419b105-1659-4f66-8ba2-098871ba3bac (api:54) 2019-04-13 21:26:47,107+0200 WARN (qgapoller/10) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:26:48,147+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:26:48,185+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:26:48,185+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:26:48,186+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:26:49,414+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=4c650103-e37f-48de-add3-9ad661266bfe (api:48) 2019-04-13 21:26:49,419+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=4c650103-e37f-48de-add3-9ad661266bfe (api:54) 2019-04-13 21:26:49,419+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:26:49,532+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=43c4d1be-588f-4b3b-a1d3-12d9a4efed57 (api:48) 2019-04-13 21:26:49,540+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707070578688', 'isoprefix': '', 'alerts': [], 'disktotal': '3898168311808', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707070578688', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719819165696', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707070578688', 'isoprefix': '', 'alerts': [], 'disktotal': '4364459573248', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=43c4d1be-588f-4b3b-a1d3-12d9a4efed57 (api:54) 2019-04-13 21:26:49,541+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:26:51,919+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:26:51,948+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:26:51,948+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:26:51,949+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:26:59,594+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=539778ef-53aa-4775-a975-a9ea65741de2 (api:48) 2019-04-13 21:26:59,599+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=539778ef-53aa-4775-a975-a9ea65741de2 (api:54) 2019-04-13 21:26:59,600+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:26:59,612+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=bea5f2fa-e031-466b-9bba-9dcb237f6888 (api:48) 2019-04-13 21:26:59,623+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707066384384', 'isoprefix': '', 'alerts': [], 'disktotal': '3898163068928', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707066384384', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719814971392', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707066384384', 'isoprefix': '', 'alerts': [], 'disktotal': '4364454330368', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=bea5f2fa-e031-466b-9bba-9dcb237f6888 (api:54) 2019-04-13 21:26:59,624+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.02 seconds (__init__:312) 2019-04-13 21:27:00,269+0200 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:27:00,348+0200 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=bb4cf0da-2f2e-40e8-a976-dd1fdf51636a (api:48) 2019-04-13 21:27:00,349+0200 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000699652', 'lastCheck': '6.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000811902', 'lastCheck': '6.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000540874', 'lastCheck': '6.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000457255', 'lastCheck': '2.9', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=bb4cf0da-2f2e-40e8-a976-dd1fdf51636a (api:54) 2019-04-13 21:27:00,350+0200 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=588a5415-58f9-4aec-b866-9d9a89eb3e2e (api:48) 2019-04-13 21:27:00,350+0200 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=588a5415-58f9-4aec-b866-9d9a89eb3e2e (api:54) 2019-04-13 21:27:00,364+0200 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '16.25', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '81.43'}, '21': {'cpuUser': '22.68', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '75.13'}, '22': {'cpuUser': '16.85', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '80.83'}, '23': {'cpuUser': '19.97', 'nodeIndex': 1, 'cpuSys': '1.92', 'cpuIdle': '78.11'}, '1': {'cpuUser': '37.61', 'nodeIndex': 0, 'cpuSys': '4.05', 'cpuIdle': '58.34'}, '0': {'cpuUser': '38.01', 'nodeIndex': 0, 'cpuSys': '2.06', 'cpuIdle': '59.93'}, '3': {'cpuUser': '45.30', 'nodeIndex': 0, 'cpuSys': '2.06', 'cpuIdle': '52.64'}, '2': {'cpuUser': '44.84', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '53.30'}, '5': {'cpuUser': '51.41', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '46.73'}, '4': {'cpuUser': '43.51', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '54.77'}, '7': {'cpuUser': '31.57', 'nodeIndex': 1, 'cpuSys': '3.52', 'cpuIdle': '64.91'}, '6': {'cpuUser': '31.77', 'nodeIndex': 1, 'cpuSys': '3.52', 'cpuIdle': '64.71'}, '9': {'cpuUser': '34.49', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '62.33'}, '8': {'cpuUser': '31.84', 'nodeIndex': 1, 'cpuSys': '3.32', 'cpuIdle': '64.84'}, '11': {'cpuUser': '29.85', 'nodeIndex': 1, 'cpuSys': '3.12', 'cpuIdle': '67.03'}, '10': {'cpuUser': '29.38', 'nodeIndex': 1, 'cpuSys': '3.32', 'cpuIdle': '67.30'}, '13': {'cpuUser': '30.84', 'nodeIndex': 0, 'cpuSys': '1.53', 'cpuIdle': '67.63'}, '12': {'cpuUser': '24.61', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '73.27'}, '15': {'cpuUser': '14.06', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '84.22'}, '14': {'cpuUser': '17.31', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '80.57'}, '17': {'cpuUser': '11.34', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '86.34'}, '16': {'cpuUser': '20.63', 'nodeIndex': 0, 'cpuSys': '2.26', 'cpuIdle': '77.11'}, '19': {'cpuUser': '18.24', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '79.57'}, '18': {'cpuUser': '22.29', 'nodeIndex': 1, 'cpuSys': '1.72', 'cpuIdle': '75.99'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69626'}, '0': {'memPercent': 21, 'memFree': '78098'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000699652', 'lastCheck': '6.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000540874', 'lastCheck': '6.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000457255', 'lastCheck': '2.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000811902', 'lastCheck': '6.1', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168853496', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '5091489998', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590455646648', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '2705610588619', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433880147578', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '259073246206', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239867484', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '366094158', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122794498', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121315995', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423726560562', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '253654002003', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215273794', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680375721', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '29546859539', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652306212', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '29289346684', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255842722220', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '7113760225', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121690548', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75334133935', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '61905869994', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122249478', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418933761', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '1012774933', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121501925', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183620.347538, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183620.347538, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183620.347538, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183620.347538, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121271956', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458749572187', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183620.347538, 'rx': '2576969581459', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322366978347', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183620.347538, 'rx': '1322366978347', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183620.347538, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940835667', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '2700659690873', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571514810981', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '4950897746', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329735281955', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183620.347538, 'rx': '980239171439', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183620.347538, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183620.347538, 'rx': '45360478', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183620.347538, 'rx': '39839452', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40566', 'ksmPages': 100, 'elapsedTime': '1728368.43', 'cpuLoad': '8.01', 'cpuSys': '2.43', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '2.98', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150383, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:27:00 GMT', 'cpuUser': '28.53', 'memFree': 150127, 'cpuIdle': '69.05', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.13'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:27:00,371+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.10 seconds (__init__:312) 2019-04-13 21:27:00,606+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=8d26ac0c-27f2-4e22-9f35-81e9b5d336c2 (api:48) 2019-04-13 21:27:00,606+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000699652', 'lastCheck': '6.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000811902', 'lastCheck': '6.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000540874', 'lastCheck': '6.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000457255', 'lastCheck': '3.1', 'valid': True}} from=internal, task_id=8d26ac0c-27f2-4e22-9f35-81e9b5d336c2 (api:54) 2019-04-13 21:27:03,227+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:27:03,261+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:27:03,262+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:27:03,263+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:27:04,107+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID=u'ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', options=None) from=internal, task_id=f2ced6b3-c4d5-48fa-9eb3-eb1892a914a2 (api:48) 2019-04-13 21:27:04,108+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', options=None) from=internal, task_id=1d05e8c7-85f3-4e37-97c0-aee04fe6c5c9 (api:48) 2019-04-13 21:27:04,109+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', options=None) from=internal, task_id=9dedec9f-c004-49be-bb62-bc11fb59a927 (api:48) 2019-04-13 21:27:04,111+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '4579684864', 'apparentsize': '16106127360'} from=internal, task_id=1d05e8c7-85f3-4e37-97c0-aee04fe6c5c9 (api:54) 2019-04-13 21:27:04,112+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID='ef6741d1-380a-4736-a924-f88ed68e1270', options=None) from=internal, task_id=f2484a59-311b-4a1e-a794-1ec412beb52f (api:48) 2019-04-13 21:27:04,114+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '53760909312', 'apparentsize': '139586437120'} from=internal, task_id=f2ced6b3-c4d5-48fa-9eb3-eb1892a914a2 (api:54) 2019-04-13 21:27:04,115+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID=u'67a4dd71-69ff-49a8-b8a4-c7101c75e105', options=None) from=internal, task_id=4e6b8401-2697-418c-a181-3a40bb65b489 (api:48) 2019-04-13 21:27:04,116+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '5697701376', 'apparentsize': '16106127360'} from=internal, task_id=9dedec9f-c004-49be-bb62-bc11fb59a927 (api:54) 2019-04-13 21:27:04,117+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', options=None) from=internal, task_id=9238090a-9da0-41a5-9b16-3aa72ec5fe58 (api:48) 2019-04-13 21:27:04,118+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '6477467136', 'apparentsize': '21474836480'} from=internal, task_id=f2484a59-311b-4a1e-a794-1ec412beb52f (api:54) 2019-04-13 21:27:04,119+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID=u'2b136ed7-8b9a-45c1-9edc-64ee96c87c43', options=None) from=internal, task_id=8c876c51-7284-4d29-8795-d2c74593750c (api:48) 2019-04-13 21:27:04,121+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID=u'44b3ceca-f0db-43da-af29-aa681df146ac', options=None) from=internal, task_id=ec5619bd-5c99-4f4c-bab4-6910ec526399 (api:48) 2019-04-13 21:27:04,122+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2524562432', 'apparentsize': '12884901888'} from=internal, task_id=9238090a-9da0-41a5-9b16-3aa72ec5fe58 (api:54) 2019-04-13 21:27:04,123+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=8105bf3c-ece4-4c46-bf0b-14bb28c234a1 (api:48) 2019-04-13 21:27:04,124+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '55059109888', 'apparentsize': '171798691840'} from=internal, task_id=4e6b8401-2697-418c-a181-3a40bb65b489 (api:54) 2019-04-13 21:27:04,124+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID='123e2cc8-8f24-4ddd-b401-68393535f555', options=None) from=internal, task_id=2552d098-94d9-4603-b538-3f42c00e7543 (api:48) 2019-04-13 21:27:04,126+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2024405504', 'apparentsize': '10737418240'} from=internal, task_id=8c876c51-7284-4d29-8795-d2c74593750c (api:54) 2019-04-13 21:27:04,126+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID='3f6c31c6-ec38-4979-a617-55537123ee61', options=None) from=internal, task_id=0e187121-f403-4a76-b6e6-33ccb12d9aaf (api:48) 2019-04-13 21:27:04,128+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '11196678144', 'apparentsize': '21474836480'} from=internal, task_id=ec5619bd-5c99-4f4c-bab4-6910ec526399 (api:54) 2019-04-13 21:27:04,129+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', options=None) from=internal, task_id=d7b07707-9478-41e1-8d01-19e185028b18 (api:48) 2019-04-13 21:27:04,130+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2116486656', 'apparentsize': '3948150784'} from=internal, task_id=8105bf3c-ece4-4c46-bf0b-14bb28c234a1 (api:54) 2019-04-13 21:27:04,131+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID=u'a1d583c1-e135-4e3f-b6fa-0f29a5261366', options=None) from=internal, task_id=5581850a-e8ef-447a-9f36-153987d1ad5e (api:48) 2019-04-13 21:27:04,131+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2283003904', 'apparentsize': '16106127360'} from=internal, task_id=2552d098-94d9-4603-b538-3f42c00e7543 (api:54) 2019-04-13 21:27:04,132+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', options=None) from=internal, task_id=3b44a650-1aab-4784-a9ee-68107602f7f1 (api:48) 2019-04-13 21:27:04,135+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2109410304', 'apparentsize': '8589934592'} from=internal, task_id=d7b07707-9478-41e1-8d01-19e185028b18 (api:54) 2019-04-13 21:27:04,135+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID='a884b4ce-381f-4653-be14-a31d15181f48', options=None) from=internal, task_id=d6d7301b-d815-426a-be63-bda2ef92628a (api:48) 2019-04-13 21:27:04,136+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '4998711296', 'apparentsize': '16106127360'} from=internal, task_id=0e187121-f403-4a76-b6e6-33ccb12d9aaf (api:54) 2019-04-13 21:27:04,136+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', options=None) from=internal, task_id=9e81d825-0d42-4c0f-91ef-30d129561b63 (api:48) 2019-04-13 21:27:04,139+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '1294894592', 'apparentsize': '12884901888'} from=internal, task_id=5581850a-e8ef-447a-9f36-153987d1ad5e (api:54) 2019-04-13 21:27:04,139+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '7263185920', 'apparentsize': '21474836480'} from=internal, task_id=3b44a650-1aab-4784-a9ee-68107602f7f1 (api:54) 2019-04-13 21:27:04,141+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '3447292416', 'apparentsize': '16106127360'} from=internal, task_id=d6d7301b-d815-426a-be63-bda2ef92628a (api:54) 2019-04-13 21:27:04,141+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2879759360', 'apparentsize': '8589934592'} from=internal, task_id=9e81d825-0d42-4c0f-91ef-30d129561b63 (api:54) 2019-04-13 21:27:06,956+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:27:06,990+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:27:06,991+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:27:06,992+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:27:09,683+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=cb55e929-6275-441d-9d31-422db89a0df1 (api:48) 2019-04-13 21:27:09,687+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=cb55e929-6275-441d-9d31-422db89a0df1 (api:54) 2019-04-13 21:27:09,688+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:27:09,699+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=5963c290-c526-4467-a02a-fe114a7f024d (api:48) 2019-04-13 21:27:09,707+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707066384384', 'isoprefix': '', 'alerts': [], 'disktotal': '3898163068928', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707066384384', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719814971392', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707066384384', 'isoprefix': '', 'alerts': [], 'disktotal': '4364454330368', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=5963c290-c526-4467-a02a-fe114a7f024d (api:54) 2019-04-13 21:27:09,708+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:27:15,316+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:27:15,387+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=cc86e6e3-d133-408f-95e7-ec3d36c5deda (api:48) 2019-04-13 21:27:15,388+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00081489', 'lastCheck': '1.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000728904', 'lastCheck': '1.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000672933', 'lastCheck': '1.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000314286', 'lastCheck': '1.2', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=cc86e6e3-d133-408f-95e7-ec3d36c5deda (api:54) 2019-04-13 21:27:15,388+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=305d1684-5c3f-4eed-9cb2-238d01172746 (api:48) 2019-04-13 21:27:15,389+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=305d1684-5c3f-4eed-9cb2-238d01172746 (api:54) 2019-04-13 21:27:15,404+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '16.31', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '81.17'}, '21': {'cpuUser': '23.67', 'nodeIndex': 1, 'cpuSys': '2.05', 'cpuIdle': '74.28'}, '22': {'cpuUser': '18.56', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '79.19'}, '23': {'cpuUser': '16.77', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '81.04'}, '1': {'cpuUser': '39.04', 'nodeIndex': 0, 'cpuSys': '4.04', 'cpuIdle': '56.92'}, '0': {'cpuUser': '35.93', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '61.88'}, '3': {'cpuUser': '41.83', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '56.18'}, '2': {'cpuUser': '41.23', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '56.85'}, '5': {'cpuUser': '40.50', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '57.64'}, '4': {'cpuUser': '50.18', 'nodeIndex': 0, 'cpuSys': '1.33', 'cpuIdle': '48.49'}, '7': {'cpuUser': '28.90', 'nodeIndex': 1, 'cpuSys': '3.78', 'cpuIdle': '67.32'}, '6': {'cpuUser': '30.36', 'nodeIndex': 1, 'cpuSys': '3.71', 'cpuIdle': '65.93'}, '9': {'cpuUser': '25.06', 'nodeIndex': 1, 'cpuSys': '3.58', 'cpuIdle': '71.36'}, '8': {'cpuUser': '30.69', 'nodeIndex': 1, 'cpuSys': '3.91', 'cpuIdle': '65.40'}, '11': {'cpuUser': '28.37', 'nodeIndex': 1, 'cpuSys': '4.04', 'cpuIdle': '67.59'}, '10': {'cpuUser': '27.84', 'nodeIndex': 1, 'cpuSys': '3.84', 'cpuIdle': '68.32'}, '13': {'cpuUser': '21.61', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '76.34'}, '12': {'cpuUser': '31.42', 'nodeIndex': 0, 'cpuSys': '1.39', 'cpuIdle': '67.19'}, '15': {'cpuUser': '27.84', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '70.57'}, '14': {'cpuUser': '20.62', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '77.39'}, '17': {'cpuUser': '28.44', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '69.70'}, '16': {'cpuUser': '30.89', 'nodeIndex': 0, 'cpuSys': '1.06', 'cpuIdle': '68.05'}, '19': {'cpuUser': '19.03', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '78.52'}, '18': {'cpuUser': '12.73', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '84.82'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69500'}, '0': {'memPercent': 21, 'memFree': '78226'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00081489', 'lastCheck': '1.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000672933', 'lastCheck': '1.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000314286', 'lastCheck': '1.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000728904', 'lastCheck': '1.1', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168853729', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '5091489998', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590455999365', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '2705611773905', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433883418606', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '259075296568', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239868053', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '366094556', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122794731', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121316228', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423729846710', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '253656384961', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215274027', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680375954', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '29546859539', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652309960', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '29289350251', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255844845900', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '7117434475', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121690781', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75336574987', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '61909045747', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122249711', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418934251', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '1012774933', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121502158', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183635.386004, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183635.386004, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183635.386004, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183635.386004, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121272189', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458749899383', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183635.386004, 'rx': '2576970709822', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322376293374', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183635.386004, 'rx': '1322376293374', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183635.386004, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940838214', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '2700660872218', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571515161151', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '4950901687', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329744765327', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183635.386004, 'rx': '980248770911', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183635.386004, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183635.386004, 'rx': '45360945', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183635.386004, 'rx': '39839830', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40566', 'ksmPages': 100, 'elapsedTime': '1728383.48', 'cpuLoad': '7.88', 'cpuSys': '2.50', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.25', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150383, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:27:15 GMT', 'cpuUser': '28.66', 'memFree': 150127, 'cpuIdle': '68.84', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.39'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:27:15,411+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:27:15,682+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=e4c686a7-0db2-4221-884f-a955f310205b (api:48) 2019-04-13 21:27:15,683+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00081489', 'lastCheck': '1.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000728904', 'lastCheck': '1.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000672933', 'lastCheck': '1.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000314286', 'lastCheck': '1.5', 'valid': True}} from=internal, task_id=e4c686a7-0db2-4221-884f-a955f310205b (api:54) 2019-04-13 21:27:18,310+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:27:18,345+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:27:18,346+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:27:18,347+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:27:19,749+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=70e2c5c5-917b-476c-bc30-a898c3480732 (api:48) 2019-04-13 21:27:19,754+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=70e2c5c5-917b-476c-bc30-a898c3480732 (api:54) 2019-04-13 21:27:19,755+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:27:19,767+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=3cf6a25f-657f-4eb3-b211-5d7be2b87dea (api:48) 2019-04-13 21:27:19,775+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707063238656', 'isoprefix': '', 'alerts': [], 'disktotal': '3898159923200', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707063238656', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719811825664', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707063238656', 'isoprefix': '', 'alerts': [], 'disktotal': '4364451184640', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=3cf6a25f-657f-4eb3-b211-5d7be2b87dea (api:54) 2019-04-13 21:27:19,776+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:27:21,999+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:27:22,028+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:27:22,029+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:27:22,029+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:27:23,942+0200 INFO (monitor/42bb677) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_isos) Closing client (__init__:584) 2019-04-13 21:27:23,942+0200 INFO (monitor/42bb677) [IOProcessClient] (/glusterSD/onode-2.discovery.intern:vol-node-2) Closing client (__init__:584) 2019-04-13 21:27:23,942+0200 INFO (monitor/42bb677) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_vms) Closing client (__init__:584) 2019-04-13 21:27:23,942+0200 INFO (monitor/42bb677) [IOProcessClient] (/onode-1.discovery.intern:_mnt_store) Closing client (__init__:584) 2019-04-13 21:27:23,943+0200 INFO (monitor/42bb677) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Closing client (__init__:584) 2019-04-13 21:27:29,822+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=2dda1b8a-96e7-466d-8040-dd5188d5750f (api:48) 2019-04-13 21:27:29,826+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=2dda1b8a-96e7-466d-8040-dd5188d5750f (api:54) 2019-04-13 21:27:29,827+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:27:29,838+0200 INFO (jsonrpc/0) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=47cf2cd0-f92c-499f-a983-9c0546221359 (api:48) 2019-04-13 21:27:29,846+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707057995776', 'isoprefix': '', 'alerts': [], 'disktotal': '3898155728896', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707057995776', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719806582784', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707057995776', 'isoprefix': '', 'alerts': [], 'disktotal': '4364446990336', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=47cf2cd0-f92c-499f-a983-9c0546221359 (api:54) 2019-04-13 21:27:29,847+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:27:30,358+0200 INFO (jsonrpc/6) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:27:30,424+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=4d8b2768-ab15-43a3-9f92-f005b75fb7a3 (api:48) 2019-04-13 21:27:30,424+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000658185', 'lastCheck': '6.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000668144', 'lastCheck': '6.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000586311', 'lastCheck': '6.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000433547', 'lastCheck': '2.9', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=4d8b2768-ab15-43a3-9f92-f005b75fb7a3 (api:54) 2019-04-13 21:27:30,425+0200 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=87b4f1b6-faff-4450-af5e-51b9666c0678 (api:48) 2019-04-13 21:27:30,425+0200 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=87b4f1b6-faff-4450-af5e-51b9666c0678 (api:54) 2019-04-13 21:27:30,439+0200 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '19.70', 'nodeIndex': 1, 'cpuSys': '1.99', 'cpuIdle': '78.31'}, '21': {'cpuUser': '24.34', 'nodeIndex': 1, 'cpuSys': '2.06', 'cpuIdle': '73.60'}, '22': {'cpuUser': '20.10', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '77.58'}, '23': {'cpuUser': '23.35', 'nodeIndex': 1, 'cpuSys': '1.72', 'cpuIdle': '74.93'}, '1': {'cpuUser': '39.00', 'nodeIndex': 0, 'cpuSys': '4.18', 'cpuIdle': '56.82'}, '0': {'cpuUser': '47.49', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '50.59'}, '3': {'cpuUser': '44.71', 'nodeIndex': 0, 'cpuSys': '2.26', 'cpuIdle': '53.03'}, '2': {'cpuUser': '40.20', 'nodeIndex': 0, 'cpuSys': '2.06', 'cpuIdle': '57.74'}, '5': {'cpuUser': '43.64', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '54.44'}, '4': {'cpuUser': '42.12', 'nodeIndex': 0, 'cpuSys': '2.39', 'cpuIdle': '55.49'}, '7': {'cpuUser': '30.91', 'nodeIndex': 1, 'cpuSys': '3.52', 'cpuIdle': '65.57'}, '6': {'cpuUser': '29.78', 'nodeIndex': 1, 'cpuSys': '3.25', 'cpuIdle': '66.97'}, '9': {'cpuUser': '28.32', 'nodeIndex': 1, 'cpuSys': '3.45', 'cpuIdle': '68.23'}, '8': {'cpuUser': '32.57', 'nodeIndex': 1, 'cpuSys': '3.45', 'cpuIdle': '63.98'}, '11': {'cpuUser': '28.19', 'nodeIndex': 1, 'cpuSys': '3.38', 'cpuIdle': '68.43'}, '10': {'cpuUser': '31.37', 'nodeIndex': 1, 'cpuSys': '3.45', 'cpuIdle': '65.18'}, '13': {'cpuUser': '15.06', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '82.75'}, '12': {'cpuUser': '35.88', 'nodeIndex': 0, 'cpuSys': '1.26', 'cpuIdle': '62.86'}, '15': {'cpuUser': '15.52', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '82.36'}, '14': {'cpuUser': '27.26', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '70.55'}, '17': {'cpuUser': '27.86', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '70.28'}, '16': {'cpuUser': '26.33', 'nodeIndex': 0, 'cpuSys': '1.26', 'cpuIdle': '72.41'}, '19': {'cpuUser': '17.38', 'nodeIndex': 1, 'cpuSys': '2.59', 'cpuIdle': '80.03'}, '18': {'cpuUser': '15.39', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '81.96'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69517'}, '0': {'memPercent': 21, 'memFree': '78208'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000658185', 'lastCheck': '6.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000586311', 'lastCheck': '6.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000433547', 'lastCheck': '2.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000668144', 'lastCheck': '6.2', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168854069', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '5091489998', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590456345317', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '2705612955789', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433887863882', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '259077853863', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239868393', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '366094556', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122795071', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121316568', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423733873676', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '253658450952', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215274367', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680376294', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '29546859539', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652318506', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '29289358113', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255847406609', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '7119004686', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121691121', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75338600553', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '61913793121', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122250051', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418934959', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '1012775191', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121502498', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183650.42288, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183650.42288, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183650.42288, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183650.42288, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121272529', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458750227339', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183650.42288, 'rx': '2576971831464', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322388777651', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183650.42288, 'rx': '1322388777651', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183650.42288, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940846388', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '2700662045278', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571515498929', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '4950910511', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329753439853', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183650.42288, 'rx': '980259501007', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183650.42288, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183650.42288, 'rx': '45361335', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183650.42288, 'rx': '39840198', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40566', 'ksmPages': 100, 'elapsedTime': '1728398.52', 'cpuLoad': '7.95', 'cpuSys': '2.47', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '2.98', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150389, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:27:30 GMT', 'cpuUser': '29.43', 'memFree': 150133, 'cpuIdle': '68.10', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.13'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:27:30,445+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:27:30,761+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=51dc984e-b3e5-4ce0-9a45-516894864990 (api:48) 2019-04-13 21:27:30,762+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000658185', 'lastCheck': '6.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000668144', 'lastCheck': '6.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000586311', 'lastCheck': '6.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000433547', 'lastCheck': '3.3', 'valid': True}} from=internal, task_id=51dc984e-b3e5-4ce0-9a45-516894864990 (api:54) 2019-04-13 21:27:33,390+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:27:33,422+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:27:33,422+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:27:33,423+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:27:37,037+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:27:37,071+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:27:37,072+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:27:37,073+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:27:39,893+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=ac6c49fc-0394-4b44-9a2c-47d3997fcae1 (api:48) 2019-04-13 21:27:39,899+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=ac6c49fc-0394-4b44-9a2c-47d3997fcae1 (api:54) 2019-04-13 21:27:39,899+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:27:39,913+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=e0f45a49-c3b4-4e1b-b033-ec6e0b675632 (api:48) 2019-04-13 21:27:39,923+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707057995776', 'isoprefix': '', 'alerts': [], 'disktotal': '3898155728896', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707057995776', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719806582784', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707057995776', 'isoprefix': '', 'alerts': [], 'disktotal': '4364446990336', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=e0f45a49-c3b4-4e1b-b033-ec6e0b675632 (api:54) 2019-04-13 21:27:39,925+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.02 seconds (__init__:312) 2019-04-13 21:27:45,424+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:27:45,488+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=125403fc-6d09-4566-90e5-9b41af434fb8 (api:48) 2019-04-13 21:27:45,488+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000466981', 'lastCheck': '1.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000702899', 'lastCheck': '1.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000562149', 'lastCheck': '1.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000398947', 'lastCheck': '1.3', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=125403fc-6d09-4566-90e5-9b41af434fb8 (api:54) 2019-04-13 21:27:45,489+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=d77282a7-36d8-4ddd-855c-22fc4452d2cd (api:48) 2019-04-13 21:27:45,489+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=d77282a7-36d8-4ddd-855c-22fc4452d2cd (api:54) 2019-04-13 21:27:45,503+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '20.89', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '76.86'}, '21': {'cpuUser': '17.84', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '79.71'}, '22': {'cpuUser': '20.89', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '76.92'}, '23': {'cpuUser': '24.87', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '72.81'}, '1': {'cpuUser': '43.37', 'nodeIndex': 0, 'cpuSys': '4.18', 'cpuIdle': '52.45'}, '0': {'cpuUser': '41.05', 'nodeIndex': 0, 'cpuSys': '2.52', 'cpuIdle': '56.43'}, '3': {'cpuUser': '44.37', 'nodeIndex': 0, 'cpuSys': '2.39', 'cpuIdle': '53.24'}, '2': {'cpuUser': '33.23', 'nodeIndex': 0, 'cpuSys': '2.06', 'cpuIdle': '64.71'}, '5': {'cpuUser': '48.15', 'nodeIndex': 0, 'cpuSys': '1.46', 'cpuIdle': '50.39'}, '4': {'cpuUser': '45.16', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '53.05'}, '7': {'cpuUser': '32.83', 'nodeIndex': 1, 'cpuSys': '3.58', 'cpuIdle': '63.59'}, '6': {'cpuUser': '34.35', 'nodeIndex': 1, 'cpuSys': '3.51', 'cpuIdle': '62.14'}, '9': {'cpuUser': '32.43', 'nodeIndex': 1, 'cpuSys': '3.12', 'cpuIdle': '64.45'}, '8': {'cpuUser': '33.56', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '63.26'}, '11': {'cpuUser': '28.72', 'nodeIndex': 1, 'cpuSys': '3.32', 'cpuIdle': '67.96'}, '10': {'cpuUser': '33.09', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '63.93'}, '13': {'cpuUser': '13.86', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '84.35'}, '12': {'cpuUser': '18.30', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '79.84'}, '15': {'cpuUser': '10.08', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '87.47'}, '14': {'cpuUser': '37.40', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '60.61'}, '17': {'cpuUser': '9.75', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '88.66'}, '16': {'cpuUser': '21.09', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '77.19'}, '19': {'cpuUser': '19.03', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '78.72'}, '18': {'cpuUser': '18.64', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '78.71'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '69807'}, '0': {'memPercent': 21, 'memFree': '77923'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000466981', 'lastCheck': '1.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000562149', 'lastCheck': '1.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000398947', 'lastCheck': '1.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000702899', 'lastCheck': '1.2', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168854820', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '5091490400', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590457012019', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '2705614100576', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433892313510', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '259080228921', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239868808', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '366094556', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122795486', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121316983', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423738344790', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '253661024450', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215274782', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680376709', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '29546859539', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652333015', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '29289372177', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255849801033', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '7121083428', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121691536', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75340092103', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '61921954546', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122250466', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418935379', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '1012775191', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121502913', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183665.48677, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183665.48677, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183665.48677, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183665.48677, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121272944', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458750858921', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183665.48677, 'rx': '2576972912571', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322400170770', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183665.48677, 'rx': '1322400170770', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183665.48677, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940858932', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '2700663176929', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571516153087', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '4950923647', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329763456403', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183665.48677, 'rx': '980270196335', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183665.48677, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183665.48677, 'rx': '45361725', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183665.48677, 'rx': '39840520', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1728413.59', 'cpuLoad': '7.91', 'cpuSys': '2.49', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '2.72', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150388, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:27:45 GMT', 'cpuUser': '28.46', 'memFree': 150132, 'cpuIdle': '69.05', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.19'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:27:45,509+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:27:45,853+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=fe3437ad-3588-4f20-89f2-ed27be4cb3e9 (api:48) 2019-04-13 21:27:45,854+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000466981', 'lastCheck': '1.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000702899', 'lastCheck': '1.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000562149', 'lastCheck': '1.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000398947', 'lastCheck': '1.7', 'valid': True}} from=internal, task_id=fe3437ad-3588-4f20-89f2-ed27be4cb3e9 (api:54) 2019-04-13 21:27:48,471+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:27:48,514+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:27:48,514+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:27:48,516+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.05 seconds (__init__:312) 2019-04-13 21:27:49,976+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=88f24a34-b5cf-4760-941c-ebfaf7676e18 (api:48) 2019-04-13 21:27:49,980+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=88f24a34-b5cf-4760-941c-ebfaf7676e18 (api:54) 2019-04-13 21:27:49,980+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:27:50,111+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=ab8cfee9-4ce1-48de-91bd-76dca90b8670 (api:48) 2019-04-13 21:27:50,119+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707052752896', 'isoprefix': '', 'alerts': [], 'disktotal': '3898150486016', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707052752896', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719801339904', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707052752896', 'isoprefix': '', 'alerts': [], 'disktotal': '4364441747456', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=ab8cfee9-4ce1-48de-91bd-76dca90b8670 (api:54) 2019-04-13 21:27:50,120+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:27:52,079+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:27:52,114+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:27:52,114+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:27:52,115+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:28:00,164+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=e3e16b2b-3985-46ca-9fc4-c374caf02c5d (api:48) 2019-04-13 21:28:00,172+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=e3e16b2b-3985-46ca-9fc4-c374caf02c5d (api:54) 2019-04-13 21:28:00,172+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:28:00,184+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=c9b43485-e83a-4585-91ba-31455e43eb58 (api:48) 2019-04-13 21:28:00,192+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707049607168', 'isoprefix': '', 'alerts': [], 'disktotal': '3898147340288', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707049607168', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719798194176', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707049607168', 'isoprefix': '', 'alerts': [], 'disktotal': '4364438601728', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=c9b43485-e83a-4585-91ba-31455e43eb58 (api:54) 2019-04-13 21:28:00,193+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:28:00,488+0200 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:28:00,561+0200 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=5d71f90f-d601-4459-9d59-596c665fa660 (api:48) 2019-04-13 21:28:00,561+0200 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000675453', 'lastCheck': '6.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000648239', 'lastCheck': '6.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000543036', 'lastCheck': '6.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000401068', 'lastCheck': '3.1', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=5d71f90f-d601-4459-9d59-596c665fa660 (api:54) 2019-04-13 21:28:00,562+0200 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=d25a8504-2fd9-4637-a2ea-cfad7098ead1 (api:48) 2019-04-13 21:28:00,562+0200 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=d25a8504-2fd9-4637-a2ea-cfad7098ead1 (api:54) 2019-04-13 21:28:00,576+0200 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '23.61', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '73.67'}, '21': {'cpuUser': '11.54', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '85.61'}, '22': {'cpuUser': '25.87', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '71.68'}, '23': {'cpuUser': '20.36', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '76.99'}, '1': {'cpuUser': '29.58', 'nodeIndex': 0, 'cpuSys': '5.04', 'cpuIdle': '65.38'}, '0': {'cpuUser': '35.55', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '62.13'}, '3': {'cpuUser': '39.99', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '57.56'}, '2': {'cpuUser': '43.84', 'nodeIndex': 0, 'cpuSys': '2.39', 'cpuIdle': '53.77'}, '5': {'cpuUser': '41.85', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '55.96'}, '4': {'cpuUser': '42.98', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '54.57'}, '7': {'cpuUser': '25.20', 'nodeIndex': 1, 'cpuSys': '4.18', 'cpuIdle': '70.62'}, '6': {'cpuUser': '33.82', 'nodeIndex': 1, 'cpuSys': '3.65', 'cpuIdle': '62.53'}, '9': {'cpuUser': '32.30', 'nodeIndex': 1, 'cpuSys': '3.58', 'cpuIdle': '64.12'}, '8': {'cpuUser': '26.79', 'nodeIndex': 1, 'cpuSys': '4.18', 'cpuIdle': '69.03'}, '11': {'cpuUser': '26.66', 'nodeIndex': 1, 'cpuSys': '3.52', 'cpuIdle': '69.82'}, '10': {'cpuUser': '32.96', 'nodeIndex': 1, 'cpuSys': '3.38', 'cpuIdle': '63.66'}, '13': {'cpuUser': '26.33', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '71.88'}, '12': {'cpuUser': '27.39', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '70.89'}, '15': {'cpuUser': '21.69', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '76.65'}, '14': {'cpuUser': '17.91', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '80.43'}, '17': {'cpuUser': '12.40', 'nodeIndex': 0, 'cpuSys': '2.06', 'cpuIdle': '85.54'}, '16': {'cpuUser': '11.08', 'nodeIndex': 0, 'cpuSys': '2.06', 'cpuIdle': '86.86'}, '19': {'cpuUser': '18.50', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '79.18'}, '18': {'cpuUser': '18.97', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '78.51'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69561'}, '0': {'memPercent': 21, 'memFree': '78171'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000675453', 'lastCheck': '6.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000543036', 'lastCheck': '6.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000401068', 'lastCheck': '3.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000648239', 'lastCheck': '6.3', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168854820', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '5091490400', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590457512400', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '2705614554932', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433896391408', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '259082909906', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239868808', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '366094556', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122795486', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121316983', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423742809452', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '253663484345', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215274782', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680376709', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '29546859539', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652333199', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '29289372340', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255852525707', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '7124803738', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121691536', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75343337415', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '61924874848', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122250466', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418935859', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '1012775191', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121502913', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183680.559851, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183680.559851, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183680.559851, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183680.559851, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121272944', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458751336555', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183680.559851, 'rx': '2576973345369', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322412651177', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183680.559851, 'rx': '1322412651177', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183680.559851, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940861425', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '2700663628741', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571516650975', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '4950926191', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329776015439', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183680.559851, 'rx': '980282662843', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183680.559851, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183680.559851, 'rx': '45362115', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183680.559851, 'rx': '39840888', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1728428.65', 'cpuLoad': '7.94', 'cpuSys': '2.74', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '2.98', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150390, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:28:00 GMT', 'cpuUser': '26.97', 'memFree': 150134, 'cpuIdle': '70.30', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.06'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:28:00,582+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:28:00,952+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=235fc7ce-5779-40e2-a5e8-dc95ebb887ea (api:48) 2019-04-13 21:28:00,952+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000675453', 'lastCheck': '6.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000648239', 'lastCheck': '6.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000543036', 'lastCheck': '6.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000401068', 'lastCheck': '3.4', 'valid': True}} from=internal, task_id=235fc7ce-5779-40e2-a5e8-dc95ebb887ea (api:54) 2019-04-13 21:28:03,551+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:28:03,586+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:28:03,587+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:28:03,587+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:28:04,115+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID=u'ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', options=None) from=internal, task_id=5d4eeb2d-1a41-4a2b-b1a5-39c376c60a33 (api:48) 2019-04-13 21:28:04,116+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', options=None) from=internal, task_id=7b97dc9e-6de7-4e6e-95d9-db32dcd3b431 (api:48) 2019-04-13 21:28:04,118+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', options=None) from=internal, task_id=3e517c1c-0180-4760-b19c-7050f28ba397 (api:48) 2019-04-13 21:28:04,121+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '4579684864', 'apparentsize': '16106127360'} from=internal, task_id=7b97dc9e-6de7-4e6e-95d9-db32dcd3b431 (api:54) 2019-04-13 21:28:04,121+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID='ef6741d1-380a-4736-a924-f88ed68e1270', options=None) from=internal, task_id=47072196-2ba2-4f2a-9f87-dfb6e85f4674 (api:48) 2019-04-13 21:28:04,123+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '5697701376', 'apparentsize': '16106127360'} from=internal, task_id=3e517c1c-0180-4760-b19c-7050f28ba397 (api:54) 2019-04-13 21:28:04,124+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID=u'67a4dd71-69ff-49a8-b8a4-c7101c75e105', options=None) from=internal, task_id=f545a7da-86b1-4fc0-a61e-91625856811b (api:48) 2019-04-13 21:28:04,125+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '53760898560', 'apparentsize': '139586437120'} from=internal, task_id=5d4eeb2d-1a41-4a2b-b1a5-39c376c60a33 (api:54) 2019-04-13 21:28:04,125+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', options=None) from=internal, task_id=abea4b10-bbbf-431e-b86c-11049a0da9bd (api:48) 2019-04-13 21:28:04,129+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '6477467136', 'apparentsize': '21474836480'} from=internal, task_id=47072196-2ba2-4f2a-9f87-dfb6e85f4674 (api:54) 2019-04-13 21:28:04,130+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID=u'2b136ed7-8b9a-45c1-9edc-64ee96c87c43', options=None) from=internal, task_id=8edda7fc-6064-44f3-b8cb-2428225e256e (api:48) 2019-04-13 21:28:04,131+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID=u'44b3ceca-f0db-43da-af29-aa681df146ac', options=None) from=internal, task_id=11722fcf-9517-49af-9f65-69ba47a7a025 (api:48) 2019-04-13 21:28:04,133+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '55059099136', 'apparentsize': '171798691840'} from=internal, task_id=f545a7da-86b1-4fc0-a61e-91625856811b (api:54) 2019-04-13 21:28:04,133+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=29fce7e4-8530-4a10-920e-4900a1f14395 (api:48) 2019-04-13 21:28:04,134+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2524562432', 'apparentsize': '12884901888'} from=internal, task_id=abea4b10-bbbf-431e-b86c-11049a0da9bd (api:54) 2019-04-13 21:28:04,134+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID='123e2cc8-8f24-4ddd-b401-68393535f555', options=None) from=internal, task_id=58df50d6-c1e6-4d95-aac2-e6e07a972ecc (api:48) 2019-04-13 21:28:04,137+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '11196678144', 'apparentsize': '21474836480'} from=internal, task_id=11722fcf-9517-49af-9f65-69ba47a7a025 (api:54) 2019-04-13 21:28:04,138+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID='3f6c31c6-ec38-4979-a617-55537123ee61', options=None) from=internal, task_id=e0730f83-e4f2-4417-9c18-2788c183ca3a (api:48) 2019-04-13 21:28:04,139+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2024405504', 'apparentsize': '10737418240'} from=internal, task_id=8edda7fc-6064-44f3-b8cb-2428225e256e (api:54) 2019-04-13 21:28:04,139+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', options=None) from=internal, task_id=352d0e75-74d8-4024-8de3-c6b25d20492a (api:48) 2019-04-13 21:28:04,141+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2116486656', 'apparentsize': '3948150784'} from=internal, task_id=29fce7e4-8530-4a10-920e-4900a1f14395 (api:54) 2019-04-13 21:28:04,142+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID=u'a1d583c1-e135-4e3f-b6fa-0f29a5261366', options=None) from=internal, task_id=f9e32e99-9a2e-4400-9e42-025be8329781 (api:48) 2019-04-13 21:28:04,142+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2282998272', 'apparentsize': '16106127360'} from=internal, task_id=58df50d6-c1e6-4d95-aac2-e6e07a972ecc (api:54) 2019-04-13 21:28:04,143+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', options=None) from=internal, task_id=c04ac9a1-f66b-4b17-98d6-e671c8dd7344 (api:48) 2019-04-13 21:28:04,146+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '4998711296', 'apparentsize': '16106127360'} from=internal, task_id=e0730f83-e4f2-4417-9c18-2788c183ca3a (api:54) 2019-04-13 21:28:04,147+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID='a884b4ce-381f-4653-be14-a31d15181f48', options=None) from=internal, task_id=c993f9e4-443d-4e70-b871-7639d956a884 (api:48) 2019-04-13 21:28:04,150+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '1294900224', 'apparentsize': '12884901888'} from=internal, task_id=f9e32e99-9a2e-4400-9e42-025be8329781 (api:54) 2019-04-13 21:28:04,150+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', options=None) from=internal, task_id=9ad12606-9e71-4109-9d5a-1782f72e9911 (api:48) 2019-04-13 21:28:04,151+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2109410304', 'apparentsize': '8589934592'} from=internal, task_id=352d0e75-74d8-4024-8de3-c6b25d20492a (api:54) 2019-04-13 21:28:04,152+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '7263185920', 'apparentsize': '21474836480'} from=internal, task_id=c04ac9a1-f66b-4b17-98d6-e671c8dd7344 (api:54) 2019-04-13 21:28:04,153+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '3447292416', 'apparentsize': '16106127360'} from=internal, task_id=c993f9e4-443d-4e70-b871-7639d956a884 (api:54) 2019-04-13 21:28:04,154+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2879770112', 'apparentsize': '8589934592'} from=internal, task_id=9ad12606-9e71-4109-9d5a-1782f72e9911 (api:54) 2019-04-13 21:28:07,122+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:28:07,153+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:28:07,153+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:28:07,154+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:28:10,243+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=7ee200c6-1865-41fc-be37-3ae646087f29 (api:48) 2019-04-13 21:28:10,248+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=7ee200c6-1865-41fc-be37-3ae646087f29 (api:54) 2019-04-13 21:28:10,248+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:28:10,261+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=092da96f-6e14-4299-ab5e-90dd5ce49e04 (api:48) 2019-04-13 21:28:10,268+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707047510016', 'isoprefix': '', 'alerts': [], 'disktotal': '3898144194560', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707047510016', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719796097024', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707047510016', 'isoprefix': '', 'alerts': [], 'disktotal': '4364435456000', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=092da96f-6e14-4299-ab5e-90dd5ce49e04 (api:54) 2019-04-13 21:28:10,271+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:28:15,536+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:28:15,605+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=87663d7b-b560-40c7-a23f-60beeab365fd (api:48) 2019-04-13 21:28:15,605+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000691906', 'lastCheck': '1.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000794416', 'lastCheck': '1.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000700563', 'lastCheck': '1.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000414735', 'lastCheck': '1.4', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=87663d7b-b560-40c7-a23f-60beeab365fd (api:54) 2019-04-13 21:28:15,606+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=dbd7b4a0-86de-4222-8a46-bf32fe31158e (api:48) 2019-04-13 21:28:15,606+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=dbd7b4a0-86de-4222-8a46-bf32fe31158e (api:54) 2019-04-13 21:28:15,620+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '18.48', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '79.00'}, '21': {'cpuUser': '19.01', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '78.67'}, '22': {'cpuUser': '12.32', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '85.10'}, '23': {'cpuUser': '28.35', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '69.53'}, '1': {'cpuUser': '33.72', 'nodeIndex': 0, 'cpuSys': '4.57', 'cpuIdle': '61.71'}, '0': {'cpuUser': '48.23', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '49.52'}, '3': {'cpuUser': '43.86', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '54.15'}, '2': {'cpuUser': '42.80', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '55.35'}, '5': {'cpuUser': '54.12', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '44.09'}, '4': {'cpuUser': '49.75', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '48.20'}, '7': {'cpuUser': '25.04', 'nodeIndex': 1, 'cpuSys': '3.64', 'cpuIdle': '71.32'}, '6': {'cpuUser': '29.21', 'nodeIndex': 1, 'cpuSys': '3.58', 'cpuIdle': '67.21'}, '9': {'cpuUser': '29.08', 'nodeIndex': 1, 'cpuSys': '3.64', 'cpuIdle': '67.28'}, '8': {'cpuUser': '29.61', 'nodeIndex': 1, 'cpuSys': '3.84', 'cpuIdle': '66.55'}, '11': {'cpuUser': '28.88', 'nodeIndex': 1, 'cpuSys': '3.25', 'cpuIdle': '67.87'}, '10': {'cpuUser': '27.43', 'nodeIndex': 1, 'cpuSys': '4.17', 'cpuIdle': '68.40'}, '13': {'cpuUser': '25.51', 'nodeIndex': 0, 'cpuSys': '2.65', 'cpuIdle': '71.84'}, '12': {'cpuUser': '27.16', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '70.99'}, '15': {'cpuUser': '30.94', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '67.07'}, '14': {'cpuUser': '40.34', 'nodeIndex': 0, 'cpuSys': '1.32', 'cpuIdle': '58.34'}, '17': {'cpuUser': '20.87', 'nodeIndex': 0, 'cpuSys': '1.39', 'cpuIdle': '77.74'}, '16': {'cpuUser': '27.96', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '70.32'}, '19': {'cpuUser': '23.32', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '74.49'}, '18': {'cpuUser': '24.51', 'nodeIndex': 1, 'cpuSys': '2.05', 'cpuIdle': '73.44'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69027'}, '0': {'memPercent': 20, 'memFree': '78706'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000691906', 'lastCheck': '1.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000700563', 'lastCheck': '1.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000414735', 'lastCheck': '1.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000794416', 'lastCheck': '1.3', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168854820', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '5091490400', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590458940288', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '2705615719182', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433900405822', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '259085471694', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239869144', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '366094954', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122795486', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '15226102', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121316983', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423746598452', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '253666127948', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215274782', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680376817', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '29546859636', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6652333296', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '29289372448', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255855112262', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '7126996172', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121691536', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75346523193', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '61928583763', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122250466', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418936279', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '1012775191', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121502913', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183695.603708, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183695.603708, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183695.603708, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183695.603708, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121272944', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458752753207', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183695.603708, 'rx': '2576974453233', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322421966204', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183695.603708, 'rx': '1322421966204', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183695.603708, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940861553', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '2700664792511', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571518078735', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '4950926671', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329784874928', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183695.603708, 'rx': '980294042743', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183695.603708, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183695.603708, 'rx': '45362505', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183695.603708, 'rx': '39841210', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1728443.70', 'cpuLoad': '7.88', 'cpuSys': '2.56', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '2.85', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150387, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:28:15 GMT', 'cpuUser': '30.85', 'memFree': 150131, 'cpuIdle': '66.59', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.52'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:28:15,626+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:28:16,048+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=a1f529bb-a5c4-4052-bd16-b1a014d05214 (api:48) 2019-04-13 21:28:16,049+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000691906', 'lastCheck': '1.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000794416', 'lastCheck': '1.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000700563', 'lastCheck': '1.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000414735', 'lastCheck': '1.9', 'valid': True}} from=internal, task_id=a1f529bb-a5c4-4052-bd16-b1a014d05214 (api:54) 2019-04-13 21:28:18,627+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:28:18,654+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:28:18,654+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:28:18,655+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:28:20,319+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=3400908d-241b-4614-af94-0d11820cf2e1 (api:48) 2019-04-13 21:28:20,324+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=3400908d-241b-4614-af94-0d11820cf2e1 (api:54) 2019-04-13 21:28:20,325+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:28:20,340+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=1c56875b-a06e-4b07-9e51-ed3030acd6b4 (api:48) 2019-04-13 21:28:20,350+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707047510016', 'isoprefix': '', 'alerts': [], 'disktotal': '3898144194560', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707047510016', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719796097024', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707047510016', 'isoprefix': '', 'alerts': [], 'disktotal': '4364435456000', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=1c56875b-a06e-4b07-9e51-ed3030acd6b4 (api:54) 2019-04-13 21:28:20,352+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:28:22,161+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:28:22,192+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:28:22,192+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:28:22,193+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:28:30,398+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=082d1215-b752-4bba-9196-b8b7da528c21 (api:48) 2019-04-13 21:28:30,403+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=082d1215-b752-4bba-9196-b8b7da528c21 (api:54) 2019-04-13 21:28:30,403+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:28:30,415+0200 INFO (jsonrpc/0) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=dc98a2b0-69a3-49af-985e-ce045acf0801 (api:48) 2019-04-13 21:28:30,423+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707043315712', 'isoprefix': '', 'alerts': [], 'disktotal': '3898141048832', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707043315712', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719791902720', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707043315712', 'isoprefix': '', 'alerts': [], 'disktotal': '4364432310272', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=dc98a2b0-69a3-49af-985e-ce045acf0801 (api:54) 2019-04-13 21:28:30,424+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:28:30,581+0200 INFO (jsonrpc/6) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:28:30,651+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=fed831ff-c531-4fbb-a023-ba0f0e6c9400 (api:48) 2019-04-13 21:28:30,652+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000692298', 'lastCheck': '6.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000539844', 'lastCheck': '6.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000566686', 'lastCheck': '6.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000416003', 'lastCheck': '3.1', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=fed831ff-c531-4fbb-a023-ba0f0e6c9400 (api:54) 2019-04-13 21:28:30,653+0200 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=fd028637-ff45-401a-bb4c-db317bf8a238 (api:48) 2019-04-13 21:28:30,653+0200 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=fd028637-ff45-401a-bb4c-db317bf8a238 (api:54) 2019-04-13 21:28:30,666+0200 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '21.39', 'nodeIndex': 1, 'cpuSys': '2.91', 'cpuIdle': '75.70'}, '21': {'cpuUser': '17.55', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '79.60'}, '22': {'cpuUser': '21.39', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '76.16'}, '23': {'cpuUser': '21.66', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '75.82'}, '1': {'cpuUser': '52.58', 'nodeIndex': 0, 'cpuSys': '4.17', 'cpuIdle': '43.25'}, '0': {'cpuUser': '59.34', 'nodeIndex': 0, 'cpuSys': '1.39', 'cpuIdle': '39.27'}, '3': {'cpuUser': '47.48', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '50.67'}, '2': {'cpuUser': '54.57', 'nodeIndex': 0, 'cpuSys': '1.32', 'cpuIdle': '44.11'}, '5': {'cpuUser': '59.07', 'nodeIndex': 0, 'cpuSys': '1.19', 'cpuIdle': '39.74'}, '4': {'cpuUser': '62.78', 'nodeIndex': 0, 'cpuSys': '1.46', 'cpuIdle': '35.76'}, '7': {'cpuUser': '32.32', 'nodeIndex': 1, 'cpuSys': '3.51', 'cpuIdle': '64.17'}, '6': {'cpuUser': '33.05', 'nodeIndex': 1, 'cpuSys': '3.77', 'cpuIdle': '63.18'}, '9': {'cpuUser': '34.83', 'nodeIndex': 1, 'cpuSys': '3.91', 'cpuIdle': '61.26'}, '8': {'cpuUser': '29.54', 'nodeIndex': 1, 'cpuSys': '4.04', 'cpuIdle': '66.42'}, '11': {'cpuUser': '30.20', 'nodeIndex': 1, 'cpuSys': '3.71', 'cpuIdle': '66.09'}, '10': {'cpuUser': '30.66', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '66.03'}, '13': {'cpuUser': '28.01', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '70.00'}, '12': {'cpuUser': '36.69', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '61.59'}, '15': {'cpuUser': '44.50', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '53.91'}, '14': {'cpuUser': '55.69', 'nodeIndex': 0, 'cpuSys': '1.26', 'cpuIdle': '43.05'}, '17': {'cpuUser': '40.00', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '58.34'}, '16': {'cpuUser': '34.17', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '63.98'}, '19': {'cpuUser': '17.81', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '79.21'}, '18': {'cpuUser': '23.51', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '74.30'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '68941'}, '0': {'memPercent': 20, 'memFree': '78789'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000692298', 'lastCheck': '6.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000566686', 'lastCheck': '6.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000416003', 'lastCheck': '3.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000539844', 'lastCheck': '6.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168855644', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '5091491024', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590459960508', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '2705630785511', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433904542984', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '259088363333', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239869428', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '366094954', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122795980', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121317267', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423750880260', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '253668764840', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215275066', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680501097', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '29547812501', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653284363', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '29289492383', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255858035135', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '7129764916', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121691820', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75349161379', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '61932551095', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122250750', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418936987', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '1012775449', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121503197', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183710.650544, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183710.650544, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183710.650544, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183710.650544, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121273228', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '221708', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458753688219', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183710.650544, 'rx': '2576988861539', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322435480475', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183710.650544, 'rx': '1322435480475', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183710.650544, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940862561', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '2700679857852', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571519097947', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '4950927659', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329794072408', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183710.650544, 'rx': '980305436313', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183710.650544, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183710.650544, 'rx': '45362895', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183710.650544, 'rx': '39841578', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1728458.74', 'cpuLoad': '7.89', 'cpuSys': '2.48', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.38', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150388, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:28:30 GMT', 'cpuUser': '37.03', 'memFree': 150132, 'cpuIdle': '60.49', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.26'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:28:30,673+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:28:31,142+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=d6502fdc-1842-40d9-80e7-50221e0f60a8 (api:48) 2019-04-13 21:28:31,143+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000692298', 'lastCheck': '6.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000539844', 'lastCheck': '6.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000566686', 'lastCheck': '6.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000416003', 'lastCheck': '3.6', 'valid': True}} from=internal, task_id=d6502fdc-1842-40d9-80e7-50221e0f60a8 (api:54) 2019-04-13 21:28:33,694+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:28:33,736+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:28:33,737+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:28:33,738+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:28:37,200+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:28:37,235+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:28:37,236+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:28:37,237+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:28:40,468+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=1f812ad6-2359-4932-98f6-8bb4b7a8ad76 (api:48) 2019-04-13 21:28:40,474+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=1f812ad6-2359-4932-98f6-8bb4b7a8ad76 (api:54) 2019-04-13 21:28:40,474+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:28:40,581+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=c2ea86cc-e23a-4755-aa57-0214497e2ac1 (api:48) 2019-04-13 21:28:40,590+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707039121408', 'isoprefix': '', 'alerts': [], 'disktotal': '3898136854528', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707039121408', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719787708416', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707039121408', 'isoprefix': '', 'alerts': [], 'disktotal': '4364428115968', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=c2ea86cc-e23a-4755-aa57-0214497e2ac1 (api:54) 2019-04-13 21:28:40,591+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:28:45,644+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:28:45,714+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=121ad591-9916-43f3-8770-cc091f56347c (api:48) 2019-04-13 21:28:45,715+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000659562', 'lastCheck': '1.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000646689', 'lastCheck': '1.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000631114', 'lastCheck': '1.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000497607', 'lastCheck': '1.5', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=121ad591-9916-43f3-8770-cc091f56347c (api:54) 2019-04-13 21:28:45,715+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=8fe2b7f1-4cec-4fae-b6c0-0d6badcc0b20 (api:48) 2019-04-13 21:28:45,716+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=8fe2b7f1-4cec-4fae-b6c0-0d6badcc0b20 (api:54) 2019-04-13 21:28:45,731+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '30.47', 'nodeIndex': 1, 'cpuSys': '2.38', 'cpuIdle': '67.15'}, '21': {'cpuUser': '29.87', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '67.35'}, '22': {'cpuUser': '20.53', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '76.62'}, '23': {'cpuUser': '23.25', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '73.97'}, '1': {'cpuUser': '48.15', 'nodeIndex': 0, 'cpuSys': '4.44', 'cpuIdle': '47.41'}, '0': {'cpuUser': '45.63', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '52.25'}, '3': {'cpuUser': '65.50', 'nodeIndex': 0, 'cpuSys': '1.46', 'cpuIdle': '33.04'}, '2': {'cpuUser': '51.06', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '46.69'}, '5': {'cpuUser': '58.55', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '39.73'}, '4': {'cpuUser': '55.90', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '42.38'}, '7': {'cpuUser': '45.57', 'nodeIndex': 1, 'cpuSys': '3.64', 'cpuIdle': '50.79'}, '6': {'cpuUser': '44.51', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '52.44'}, '9': {'cpuUser': '45.76', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '51.06'}, '8': {'cpuUser': '45.17', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '51.78'}, '11': {'cpuUser': '48.35', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '48.60'}, '10': {'cpuUser': '48.94', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '47.88'}, '13': {'cpuUser': '29.01', 'nodeIndex': 0, 'cpuSys': '2.85', 'cpuIdle': '68.14'}, '12': {'cpuUser': '37.29', 'nodeIndex': 0, 'cpuSys': '2.58', 'cpuIdle': '60.13'}, '15': {'cpuUser': '34.11', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '64.10'}, '14': {'cpuUser': '46.96', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '50.79'}, '17': {'cpuUser': '44.44', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '53.44'}, '16': {'cpuUser': '40.13', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '58.21'}, '19': {'cpuUser': '23.18', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '73.51'}, '18': {'cpuUser': '29.60', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '67.95'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69231'}, '0': {'memPercent': 21, 'memFree': '78500'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000659562', 'lastCheck': '1.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000631114', 'lastCheck': '1.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000497607', 'lastCheck': '1.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000646689', 'lastCheck': '1.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168855764', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '5091491108', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590461122468', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '2705632696233', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433908372230', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '259091146678', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239869428', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '366094954', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122795980', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121317267', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423754736012', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '253671717405', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215275066', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680501157', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '29547812543', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653284483', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '29289492467', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255860881516', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '7133506684', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121691820', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75350670829', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '61935455296', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122250750', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418937407', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '1012775449', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121503197', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183725.713251, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183725.713251, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183725.713251, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183725.713251, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121273438', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458754811011', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183725.713251, 'rx': '2576990687015', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322445850310', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183725.713251, 'rx': '1322445850310', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183725.713251, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940862847', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '2700681768190', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571520218373', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '4950928043', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329804407490', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183725.713251, 'rx': '980315130967', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183725.713251, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183725.713251, 'rx': '45363415', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183725.713251, 'rx': '39841900', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1728473.81', 'cpuLoad': '7.94', 'cpuSys': '2.61', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.05', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150389, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:28:45 GMT', 'cpuUser': '41.33', 'memFree': 150133, 'cpuIdle': '56.06', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.26'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:28:45,738+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:28:46,231+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=67db892b-08d0-4327-b159-37cc14c5b864 (api:48) 2019-04-13 21:28:46,231+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000659562', 'lastCheck': '1.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000646689', 'lastCheck': '2.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000631114', 'lastCheck': '1.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000497607', 'lastCheck': '2.0', 'valid': True}} from=internal, task_id=67db892b-08d0-4327-b159-37cc14c5b864 (api:54) 2019-04-13 21:28:48,774+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:28:48,804+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:28:48,804+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:28:48,805+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:28:50,643+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=e3efeaf4-6ac2-4532-850e-9402f34608b2 (api:48) 2019-04-13 21:28:50,649+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=e3efeaf4-6ac2-4532-850e-9402f34608b2 (api:54) 2019-04-13 21:28:50,650+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:28:50,773+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=b0e50f9f-438e-475c-a39a-309631492d13 (api:48) 2019-04-13 21:28:50,780+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707037024256', 'isoprefix': '', 'alerts': [], 'disktotal': '3898133708800', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707037024256', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719785611264', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707037024256', 'isoprefix': '', 'alerts': [], 'disktotal': '4364424970240', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=b0e50f9f-438e-475c-a39a-309631492d13 (api:54) 2019-04-13 21:28:50,782+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:28:52,244+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:28:52,276+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:28:52,277+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:28:52,277+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:28:57,536+0200 INFO (itmap/0) [IOProcessClient] (/onode-1.discovery.intern:_mnt_store) Starting client (__init__:308) 2019-04-13 21:28:57,589+0200 INFO (itmap/1) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_vms) Starting client (__init__:308) 2019-04-13 21:28:57,600+0200 INFO (ioprocess/29261) [IOProcess] (/onode-1.discovery.intern:_mnt_store) Starting ioprocess (__init__:434) 2019-04-13 21:28:57,623+0200 INFO (itmap/2) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_isos) Starting client (__init__:308) 2019-04-13 21:28:57,633+0200 INFO (ioprocess/29269) [IOProcess] (/nas.discovery.intern:_mnt_d2600_vms) Starting ioprocess (__init__:434) 2019-04-13 21:28:57,653+0200 INFO (itmap/3) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting client (__init__:308) 2019-04-13 21:28:57,674+0200 INFO (ioprocess/29277) [IOProcess] (/nas.discovery.intern:_mnt_d2600_isos) Starting ioprocess (__init__:434) 2019-04-13 21:28:57,679+0200 INFO (itmap/4) [IOProcessClient] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting client (__init__:308) 2019-04-13 21:28:57,688+0200 INFO (ioprocess/29280) [IOProcess] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting ioprocess (__init__:434) 2019-04-13 21:28:57,707+0200 INFO (ioprocess/29291) [IOProcess] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting ioprocess (__init__:434) 2019-04-13 21:28:57,717+0200 INFO (monitor/6a4cde7) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:29:00,687+0200 INFO (jsonrpc/4) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:29:00,767+0200 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=b9e9c812-2345-4e1c-a787-657e6cb31a89 (api:48) 2019-04-13 21:29:00,768+0200 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000532764', 'lastCheck': '6.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00068173', 'lastCheck': '6.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000427935', 'lastCheck': '6.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000503368', 'lastCheck': '3.0', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=b9e9c812-2345-4e1c-a787-657e6cb31a89 (api:54) 2019-04-13 21:29:00,769+0200 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=34352c2f-b4ce-4254-975e-14d64a554abb (api:48) 2019-04-13 21:29:00,769+0200 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=34352c2f-b4ce-4254-975e-14d64a554abb (api:54) 2019-04-13 21:29:00,789+0200 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '31.68', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '66.13'}, '21': {'cpuUser': '30.49', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '67.26'}, '22': {'cpuUser': '25.19', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '72.36'}, '23': {'cpuUser': '25.12', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '72.36'}, '1': {'cpuUser': '42.69', 'nodeIndex': 0, 'cpuSys': '4.51', 'cpuIdle': '52.80'}, '0': {'cpuUser': '42.22', 'nodeIndex': 0, 'cpuSys': '3.05', 'cpuIdle': '54.73'}, '3': {'cpuUser': '41.56', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '55.99'}, '2': {'cpuUser': '32.15', 'nodeIndex': 0, 'cpuSys': '2.65', 'cpuIdle': '65.20'}, '5': {'cpuUser': '43.35', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '54.53'}, '4': {'cpuUser': '50.38', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '47.30'}, '7': {'cpuUser': '39.44', 'nodeIndex': 1, 'cpuSys': '3.25', 'cpuIdle': '57.31'}, '6': {'cpuUser': '44.68', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '52.54'}, '9': {'cpuUser': '43.35', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '53.87'}, '8': {'cpuUser': '42.42', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '54.80'}, '11': {'cpuUser': '45.34', 'nodeIndex': 1, 'cpuSys': '2.59', 'cpuIdle': '52.07'}, '10': {'cpuUser': '46.60', 'nodeIndex': 1, 'cpuSys': '2.59', 'cpuIdle': '50.81'}, '13': {'cpuUser': '20.02', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '77.66'}, '12': {'cpuUser': '15.84', 'nodeIndex': 0, 'cpuSys': '2.59', 'cpuIdle': '81.57'}, '15': {'cpuUser': '19.22', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '78.99'}, '14': {'cpuUser': '32.48', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '65.80'}, '17': {'cpuUser': '26.65', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '71.49'}, '16': {'cpuUser': '19.22', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '78.66'}, '19': {'cpuUser': '23.73', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '73.49'}, '18': {'cpuUser': '25.52', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '71.83'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69628'}, '0': {'memPercent': 21, 'memFree': '78103'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000532764', 'lastCheck': '6.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000427935', 'lastCheck': '6.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000503368', 'lastCheck': '3.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00068173', 'lastCheck': '6.5', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168856089', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '5091491108', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590462079464', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '2705633162300', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433911890650', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '259093637736', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239869753', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '366094954', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122796305', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121317592', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423758524076', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '253674395231', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215275391', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680501482', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '29547812543', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653284726', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '29289492549', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255863390854', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '7135613705', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121692145', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75353249361', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '61939684114', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122251075', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418937887', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '1012775449', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121503522', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183740.764941, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183740.764941, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183740.764941, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183740.764941, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121273763', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458755726773', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183740.764941, 'rx': '2576991128888', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322455172921', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183740.764941, 'rx': '1322455172921', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183740.764941, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940862847', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '2700682234039', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571521216617', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '4950928261', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329813710955', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183740.764941, 'rx': '980325296441', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183740.764941, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183740.764941, 'rx': '45363805', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183740.764941, 'rx': '39842268', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1728488.85', 'cpuLoad': '7.93', 'cpuSys': '2.55', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.05', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150369, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:29:00 GMT', 'cpuUser': '33.72', 'memFree': 150113, 'cpuIdle': '63.73', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.06'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:29:00,797+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.11 seconds (__init__:312) 2019-04-13 21:29:00,831+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=c52e6d8e-cef0-4403-80f0-d857bfef660e (api:48) 2019-04-13 21:29:00,835+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=c52e6d8e-cef0-4403-80f0-d857bfef660e (api:54) 2019-04-13 21:29:00,836+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:29:00,849+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=e158a3d6-2fce-4f30-aa3a-78dedb84133d (api:48) 2019-04-13 21:29:00,857+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707037024256', 'isoprefix': '', 'alerts': [], 'disktotal': '3898133708800', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707037024256', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719785611264', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707037024256', 'isoprefix': '', 'alerts': [], 'disktotal': '4364424970240', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=e158a3d6-2fce-4f30-aa3a-78dedb84133d (api:54) 2019-04-13 21:29:00,858+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:29:01,310+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=6e7f6d3e-b97b-492c-8649-9eb0788a93c5 (api:48) 2019-04-13 21:29:01,311+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000532764', 'lastCheck': '7.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00068173', 'lastCheck': '7.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000427935', 'lastCheck': '7.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000503368', 'lastCheck': '3.6', 'valid': True}} from=internal, task_id=6e7f6d3e-b97b-492c-8649-9eb0788a93c5 (api:54) 2019-04-13 21:29:03,854+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:29:03,895+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:29:03,896+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:29:03,897+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.05 seconds (__init__:312) 2019-04-13 21:29:04,109+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID=u'ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', options=None) from=internal, task_id=43669b15-0840-4ea0-a2e0-4b6ff3c052e1 (api:48) 2019-04-13 21:29:04,111+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', options=None) from=internal, task_id=5abaef28-7758-48ec-bd73-353b3c5375ec (api:48) 2019-04-13 21:29:04,112+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', options=None) from=internal, task_id=e535a52a-52eb-42b4-9284-d43f307c821c (api:48) 2019-04-13 21:29:04,116+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '5697701376', 'apparentsize': '16106127360'} from=internal, task_id=e535a52a-52eb-42b4-9284-d43f307c821c (api:54) 2019-04-13 21:29:04,116+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID='ef6741d1-380a-4736-a924-f88ed68e1270', options=None) from=internal, task_id=e15b8fd9-e1d2-4df9-bbaa-3f0c7898a477 (api:48) 2019-04-13 21:29:04,117+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '4579684864', 'apparentsize': '16106127360'} from=internal, task_id=5abaef28-7758-48ec-bd73-353b3c5375ec (api:54) 2019-04-13 21:29:04,118+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID=u'67a4dd71-69ff-49a8-b8a4-c7101c75e105', options=None) from=internal, task_id=edf05c84-a051-4f0a-8896-5f6d4f38e648 (api:48) 2019-04-13 21:29:04,120+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '53760882176', 'apparentsize': '139586437120'} from=internal, task_id=43669b15-0840-4ea0-a2e0-4b6ff3c052e1 (api:54) 2019-04-13 21:29:04,120+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', options=None) from=internal, task_id=cda3cba8-e027-41c3-a0c3-a9ceaef21c55 (api:48) 2019-04-13 21:29:04,123+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID=u'2b136ed7-8b9a-45c1-9edc-64ee96c87c43', options=None) from=internal, task_id=2c5daef1-4317-415b-8681-6df7403e01fb (api:48) 2019-04-13 21:29:04,125+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '55059093504', 'apparentsize': '171798691840'} from=internal, task_id=edf05c84-a051-4f0a-8896-5f6d4f38e648 (api:54) 2019-04-13 21:29:04,125+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID=u'44b3ceca-f0db-43da-af29-aa681df146ac', options=None) from=internal, task_id=49456342-d82a-43b5-bc49-8e8785fb2a5c (api:48) 2019-04-13 21:29:04,126+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '6477462016', 'apparentsize': '21474836480'} from=internal, task_id=e15b8fd9-e1d2-4df9-bbaa-3f0c7898a477 (api:54) 2019-04-13 21:29:04,126+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=4d91cac1-00cc-41ef-aa13-f5e4f4176a82 (api:48) 2019-04-13 21:29:04,129+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2024405504', 'apparentsize': '10737418240'} from=internal, task_id=2c5daef1-4317-415b-8681-6df7403e01fb (api:54) 2019-04-13 21:29:04,129+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID='123e2cc8-8f24-4ddd-b401-68393535f555', options=None) from=internal, task_id=7f11f26f-ba85-4d6b-843c-a5a6c1bd6fc2 (api:48) 2019-04-13 21:29:04,132+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2524562432', 'apparentsize': '12884901888'} from=internal, task_id=cda3cba8-e027-41c3-a0c3-a9ceaef21c55 (api:54) 2019-04-13 21:29:04,132+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID='3f6c31c6-ec38-4979-a617-55537123ee61', options=None) from=internal, task_id=2ff98c78-0c33-42e3-9989-25324e848946 (api:48) 2019-04-13 21:29:04,133+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2116486656', 'apparentsize': '3948150784'} from=internal, task_id=4d91cac1-00cc-41ef-aa13-f5e4f4176a82 (api:54) 2019-04-13 21:29:04,133+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', options=None) from=internal, task_id=28adb033-15c5-42d0-9047-de64f551cda9 (api:48) 2019-04-13 21:29:04,133+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '11196673024', 'apparentsize': '21474836480'} from=internal, task_id=49456342-d82a-43b5-bc49-8e8785fb2a5c (api:54) 2019-04-13 21:29:04,134+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID=u'a1d583c1-e135-4e3f-b6fa-0f29a5261366', options=None) from=internal, task_id=1e33cb77-58cd-4be9-9b88-c029cdeb9237 (api:48) 2019-04-13 21:29:04,138+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2283003904', 'apparentsize': '16106127360'} from=internal, task_id=7f11f26f-ba85-4d6b-843c-a5a6c1bd6fc2 (api:54) 2019-04-13 21:29:04,138+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', options=None) from=internal, task_id=9adb4f04-5095-4029-8f9c-9d1489c9076a (api:48) 2019-04-13 21:29:04,141+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '4998711296', 'apparentsize': '16106127360'} from=internal, task_id=2ff98c78-0c33-42e3-9989-25324e848946 (api:54) 2019-04-13 21:29:04,141+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID='a884b4ce-381f-4653-be14-a31d15181f48', options=None) from=internal, task_id=ff5ad2be-275f-47a6-bb50-2cd08d3942a9 (api:48) 2019-04-13 21:29:04,142+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2109410304', 'apparentsize': '8589934592'} from=internal, task_id=28adb033-15c5-42d0-9047-de64f551cda9 (api:54) 2019-04-13 21:29:04,142+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', options=None) from=internal, task_id=9a1cc136-8b94-40f9-9edf-b5242f48d2b8 (api:48) 2019-04-13 21:29:04,144+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '1294900224', 'apparentsize': '12884901888'} from=internal, task_id=1e33cb77-58cd-4be9-9b88-c029cdeb9237 (api:54) 2019-04-13 21:29:04,146+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '7263185920', 'apparentsize': '21474836480'} from=internal, task_id=9adb4f04-5095-4029-8f9c-9d1489c9076a (api:54) 2019-04-13 21:29:04,147+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '3447292416', 'apparentsize': '16106127360'} from=internal, task_id=ff5ad2be-275f-47a6-bb50-2cd08d3942a9 (api:54) 2019-04-13 21:29:04,147+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2879770112', 'apparentsize': '8589934592'} from=internal, task_id=9a1cc136-8b94-40f9-9edf-b5242f48d2b8 (api:54) 2019-04-13 21:29:07,284+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:29:07,314+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:29:07,314+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:29:07,315+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:29:10,906+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=d31ee5b4-7e13-4545-adee-c6ecc443b0ed (api:48) 2019-04-13 21:29:10,911+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=d31ee5b4-7e13-4545-adee-c6ecc443b0ed (api:54) 2019-04-13 21:29:10,912+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:29:11,051+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=05ba2337-2fdb-4f95-9785-01520440f110 (api:48) 2019-04-13 21:29:11,071+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707032829952', 'isoprefix': '', 'alerts': [], 'disktotal': '3898129514496', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707032829952', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719781416960', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707032829952', 'isoprefix': '', 'alerts': [], 'disktotal': '4364420775936', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=05ba2337-2fdb-4f95-9785-01520440f110 (api:54) 2019-04-13 21:29:11,073+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.02 seconds (__init__:312) 2019-04-13 21:29:15,759+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:29:15,829+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=7ac561f5-b2ef-4abf-8603-3c26398fda7e (api:48) 2019-04-13 21:29:15,830+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000850156', 'lastCheck': '1.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000763624', 'lastCheck': '1.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000694621', 'lastCheck': '1.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000371124', 'lastCheck': '1.6', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=7ac561f5-b2ef-4abf-8603-3c26398fda7e (api:54) 2019-04-13 21:29:15,831+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=e3568d37-97d7-486e-8026-c8901b6ed44b (api:48) 2019-04-13 21:29:15,831+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=e3568d37-97d7-486e-8026-c8901b6ed44b (api:54) 2019-04-13 21:29:15,845+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '36.05', 'nodeIndex': 1, 'cpuSys': '1.99', 'cpuIdle': '61.96'}, '21': {'cpuUser': '14.91', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '82.04'}, '22': {'cpuUser': '23.86', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '73.56'}, '23': {'cpuUser': '25.71', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '71.97'}, '1': {'cpuUser': '47.45', 'nodeIndex': 0, 'cpuSys': '4.77', 'cpuIdle': '47.78'}, '0': {'cpuUser': '46.99', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '50.69'}, '3': {'cpuUser': '52.42', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '45.39'}, '2': {'cpuUser': '41.95', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '55.60'}, '5': {'cpuUser': '51.03', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '47.25'}, '4': {'cpuUser': '61.11', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '37.30'}, '7': {'cpuUser': '41.22', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '55.60'}, '6': {'cpuUser': '36.65', 'nodeIndex': 1, 'cpuSys': '3.38', 'cpuIdle': '59.97'}, '9': {'cpuUser': '45.99', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '50.96'}, '8': {'cpuUser': '38.51', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '58.91'}, '11': {'cpuUser': '39.90', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '57.32'}, '10': {'cpuUser': '43.48', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '54.00'}, '13': {'cpuUser': '34.66', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '62.89'}, '12': {'cpuUser': '39.57', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '58.11'}, '15': {'cpuUser': '33.73', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '64.22'}, '14': {'cpuUser': '38.04', 'nodeIndex': 0, 'cpuSys': '2.52', 'cpuIdle': '59.44'}, '17': {'cpuUser': '34.73', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '63.02'}, '16': {'cpuUser': '23.53', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '74.15'}, '19': {'cpuUser': '22.20', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '75.35'}, '18': {'cpuUser': '27.50', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '70.31'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '69842'}, '0': {'memPercent': 21, 'memFree': '77880'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000850156', 'lastCheck': '1.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000694621', 'lastCheck': '1.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000371124', 'lastCheck': '1.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000763624', 'lastCheck': '1.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168858294', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '5091491108', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590462396727', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '2705634287990', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433915532024', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '259095906668', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239872618', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '366095654', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122798510', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121319797', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423762118748', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '253676581436', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215277596', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680503444', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '29547812786', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653286849', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '29289492631', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255865702043', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '7139171225', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121694350', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75355637789', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '61944174178', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122253280', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418938307', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '1012775449', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121505727', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183755.828489, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183755.828489, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183755.828489, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183755.828489, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121275968', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458756031321', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183755.828489, 'rx': '2576992205867', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322468683290', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183755.828489, 'rx': '1322468683290', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183755.828489, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940863308', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '2700683358925', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571521533419', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '4950929065', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329823751584', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183755.828489, 'rx': '980335404171', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183755.828489, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183755.828489, 'rx': '45364195', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183755.828489, 'rx': '39842590', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40570', 'ksmPages': 100, 'elapsedTime': '1728503.92', 'cpuLoad': '7.85', 'cpuSys': '2.54', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.58', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150384, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:29:15 GMT', 'cpuUser': '37.56', 'memFree': 150128, 'cpuIdle': '59.90', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.72'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:29:15,851+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:29:16,412+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=a1a67a3e-140d-4b6d-b37f-7ee590576d32 (api:48) 2019-04-13 21:29:16,413+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000850156', 'lastCheck': '2.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000763624', 'lastCheck': '2.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000694621', 'lastCheck': '2.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000371124', 'lastCheck': '2.2', 'valid': True}} from=internal, task_id=a1a67a3e-140d-4b6d-b37f-7ee590576d32 (api:54) 2019-04-13 21:29:18,935+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:29:18,975+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:29:18,976+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:29:18,977+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:29:21,143+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=e9a6ef4d-2672-48f1-bd98-0a690b56dee4 (api:48) 2019-04-13 21:29:21,148+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=e9a6ef4d-2672-48f1-bd98-0a690b56dee4 (api:54) 2019-04-13 21:29:21,148+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:29:21,280+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=7b1b630c-cd5e-4de5-b62a-c88ef500fd5f (api:48) 2019-04-13 21:29:21,313+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707032829952', 'isoprefix': '', 'alerts': [], 'disktotal': '3898129514496', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707032829952', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719781416960', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707032829952', 'isoprefix': '', 'alerts': [], 'disktotal': '4364420775936', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=7b1b630c-cd5e-4de5-b62a-c88ef500fd5f (api:54) 2019-04-13 21:29:21,315+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.04 seconds (__init__:312) 2019-04-13 21:29:22,323+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:29:22,352+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:29:22,353+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:29:22,353+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:29:30,810+0200 INFO (jsonrpc/1) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:29:30,899+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=8fb2ae8f-a6f7-4c05-a0ad-8662699e67f2 (api:48) 2019-04-13 21:29:30,900+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000710187', 'lastCheck': '6.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000812358', 'lastCheck': '6.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000582584', 'lastCheck': '6.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000443438', 'lastCheck': '3.2', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=8fb2ae8f-a6f7-4c05-a0ad-8662699e67f2 (api:54) 2019-04-13 21:29:30,900+0200 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=4c3209a3-a167-4816-b08b-828c1d32b965 (api:48) 2019-04-13 21:29:30,901+0200 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=4c3209a3-a167-4816-b08b-828c1d32b965 (api:54) 2019-04-13 21:29:30,915+0200 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '19.75', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '77.53'}, '21': {'cpuUser': '18.23', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '79.45'}, '22': {'cpuUser': '22.87', 'nodeIndex': 1, 'cpuSys': '2.05', 'cpuIdle': '75.08'}, '23': {'cpuUser': '20.02', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '77.73'}, '1': {'cpuUser': '53.83', 'nodeIndex': 0, 'cpuSys': '4.04', 'cpuIdle': '42.13'}, '0': {'cpuUser': '53.16', 'nodeIndex': 0, 'cpuSys': '1.52', 'cpuIdle': '45.32'}, '3': {'cpuUser': '61.32', 'nodeIndex': 0, 'cpuSys': '1.46', 'cpuIdle': '37.22'}, '2': {'cpuUser': '48.59', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '49.49'}, '5': {'cpuUser': '47.40', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '50.81'}, '4': {'cpuUser': '47.99', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '50.22'}, '7': {'cpuUser': '25.26', 'nodeIndex': 1, 'cpuSys': '3.25', 'cpuIdle': '71.49'}, '6': {'cpuUser': '33.01', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '64.21'}, '9': {'cpuUser': '30.69', 'nodeIndex': 1, 'cpuSys': '3.12', 'cpuIdle': '66.19'}, '8': {'cpuUser': '32.22', 'nodeIndex': 1, 'cpuSys': '3.25', 'cpuIdle': '64.53'}, '11': {'cpuUser': '33.34', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '63.81'}, '10': {'cpuUser': '32.08', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '64.87'}, '13': {'cpuUser': '19.22', 'nodeIndex': 0, 'cpuSys': '2.65', 'cpuIdle': '78.13'}, '12': {'cpuUser': '30.56', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '67.65'}, '15': {'cpuUser': '23.20', 'nodeIndex': 0, 'cpuSys': '2.72', 'cpuIdle': '74.08'}, '14': {'cpuUser': '30.09', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '67.86'}, '17': {'cpuUser': '35.40', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '62.94'}, '16': {'cpuUser': '28.90', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '69.24'}, '19': {'cpuUser': '26.52', 'nodeIndex': 1, 'cpuSys': '1.86', 'cpuIdle': '71.62'}, '18': {'cpuUser': '21.81', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '76.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '69826'}, '0': {'memPercent': 21, 'memFree': '77900'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000710187', 'lastCheck': '6.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000582584', 'lastCheck': '6.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000443438', 'lastCheck': '3.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000812358', 'lastCheck': '6.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168859093', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '5091491426', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590462717387', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '2705635406993', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433919174442', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '259098386257', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239873201', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '366095654', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122799093', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121320380', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423765780182', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '253679100824', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215278179', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680504087', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '29547812894', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653287432', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '29289492631', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255868207151', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '7141195038', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121694933', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75357963639', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '61948398304', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122253863', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418939015', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '1012775707', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121506310', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183770.898087, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183770.898087, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183770.898087, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183770.898087, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121276551', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458756340415', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183770.898087, 'rx': '2576993276156', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322479053865', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183770.898087, 'rx': '1322479053865', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183770.898087, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940863638', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '2700684477244', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571521853749', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '4950929749', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329833349549', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183770.898087, 'rx': '980345253354', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183770.898087, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183770.898087, 'rx': '45364585', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183770.898087, 'rx': '39842958', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40566', 'ksmPages': 100, 'elapsedTime': '1728518.97', 'cpuLoad': '7.91', 'cpuSys': '2.38', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.18', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150384, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:29:30 GMT', 'cpuUser': '33.14', 'memFree': 150128, 'cpuIdle': '64.48', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.33'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:29:30,921+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.11 seconds (__init__:312) 2019-04-13 21:29:31,374+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=2857596a-7fa4-43a8-886b-04640eba93e6 (api:48) 2019-04-13 21:29:31,379+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=2857596a-7fa4-43a8-886b-04640eba93e6 (api:54) 2019-04-13 21:29:31,379+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:29:31,392+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=8333e87b-8df3-495d-9220-4ce9854dd558 (api:48) 2019-04-13 21:29:31,400+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707027587072', 'isoprefix': '', 'alerts': [], 'disktotal': '3898125320192', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707027587072', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719776174080', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707027587072', 'isoprefix': '', 'alerts': [], 'disktotal': '4364416581632', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=8333e87b-8df3-495d-9220-4ce9854dd558 (api:54) 2019-04-13 21:29:31,402+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:29:31,522+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=d1127157-8027-40f3-b2a9-6c503f6948e8 (api:48) 2019-04-13 21:29:31,524+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000710187', 'lastCheck': '7.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000812358', 'lastCheck': '7.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000582584', 'lastCheck': '7.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000443438', 'lastCheck': '3.8', 'valid': True}} from=internal, task_id=d1127157-8027-40f3-b2a9-6c503f6948e8 (api:54) 2019-04-13 21:29:34,014+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:29:34,063+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:29:34,064+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:29:34,065+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.05 seconds (__init__:312) 2019-04-13 21:29:37,362+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:29:37,402+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:29:37,403+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:29:37,404+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.05 seconds (__init__:312) 2019-04-13 21:29:41,443+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=d4d21b76-8681-4a77-ae76-4ae41f8a795b (api:48) 2019-04-13 21:29:41,448+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=d4d21b76-8681-4a77-ae76-4ae41f8a795b (api:54) 2019-04-13 21:29:41,449+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:29:41,464+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=64f36cdb-1f52-41b7-ada8-5e853b69f369 (api:48) 2019-04-13 21:29:41,474+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707023392768', 'isoprefix': '', 'alerts': [], 'disktotal': '3898121125888', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707023392768', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719771979776', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707023392768', 'isoprefix': '', 'alerts': [], 'disktotal': '4364412387328', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=64f36cdb-1f52-41b7-ada8-5e853b69f369 (api:54) 2019-04-13 21:29:41,476+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.02 seconds (__init__:312) 2019-04-13 21:29:45,856+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:29:45,925+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=02ec34af-9d40-4e7c-b8a7-273e5d63f67e (api:48) 2019-04-13 21:29:45,925+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000669525', 'lastCheck': '1.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000698852', 'lastCheck': '1.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000630638', 'lastCheck': '1.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000392636', 'lastCheck': '1.7', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=02ec34af-9d40-4e7c-b8a7-273e5d63f67e (api:54) 2019-04-13 21:29:45,926+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=efa429a6-dc72-4625-9098-a7485250002c (api:48) 2019-04-13 21:29:45,927+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=efa429a6-dc72-4625-9098-a7485250002c (api:54) 2019-04-13 21:29:45,941+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '16.83', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '80.52'}, '21': {'cpuUser': '23.58', 'nodeIndex': 1, 'cpuSys': '1.66', 'cpuIdle': '74.76'}, '22': {'cpuUser': '22.26', 'nodeIndex': 1, 'cpuSys': '1.85', 'cpuIdle': '75.89'}, '23': {'cpuUser': '21.93', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '75.75'}, '1': {'cpuUser': '39.29', 'nodeIndex': 0, 'cpuSys': '3.91', 'cpuIdle': '56.80'}, '0': {'cpuUser': '36.17', 'nodeIndex': 0, 'cpuSys': '2.72', 'cpuIdle': '61.11'}, '3': {'cpuUser': '44.78', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '52.97'}, '2': {'cpuUser': '47.37', 'nodeIndex': 0, 'cpuSys': '2.38', 'cpuIdle': '50.25'}, '5': {'cpuUser': '47.24', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '50.71'}, '4': {'cpuUser': '41.47', 'nodeIndex': 0, 'cpuSys': '2.52', 'cpuIdle': '56.01'}, '7': {'cpuUser': '38.76', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '58.06'}, '6': {'cpuUser': '38.62', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '58.40'}, '9': {'cpuUser': '40.08', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '57.34'}, '8': {'cpuUser': '38.09', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '58.86'}, '11': {'cpuUser': '31.80', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '64.89'}, '10': {'cpuUser': '40.28', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '57.40'}, '13': {'cpuUser': '29.48', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '68.20'}, '12': {'cpuUser': '23.98', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '73.97'}, '15': {'cpuUser': '19.15', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '79.06'}, '14': {'cpuUser': '12.39', 'nodeIndex': 0, 'cpuSys': '2.58', 'cpuIdle': '85.03'}, '17': {'cpuUser': '17.89', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '79.79'}, '16': {'cpuUser': '34.91', 'nodeIndex': 0, 'cpuSys': '1.32', 'cpuIdle': '63.77'}, '19': {'cpuUser': '19.74', 'nodeIndex': 1, 'cpuSys': '2.05', 'cpuIdle': '78.21'}, '18': {'cpuUser': '15.77', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '81.91'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '70289'}, '0': {'memPercent': 22, 'memFree': '77437'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000669525', 'lastCheck': '1.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000630638', 'lastCheck': '1.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000392636', 'lastCheck': '1.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000698852', 'lastCheck': '1.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168859628', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '5091491510', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590463428553', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '2705635441592', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433922818984', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '259100772398', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239873616', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '366095654', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122799508', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121320795', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423769377290', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '253681568887', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215278594', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680504861', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '29547813143', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653287848', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '29289493034', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255870610093', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '7143190033', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121695348', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75361165545', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '61952891368', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122254278', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418940179', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '1012776443', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121506725', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183785.923935, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183785.923935, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183785.923935, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183785.923935, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121276966', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458757023379', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183785.923935, 'rx': '2576994356853', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322492565212', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183785.923935, 'rx': '1322492565212', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183785.923935, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940863852', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '2700684511459', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571522564701', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '4950930133', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329842585331', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183785.923935, 'rx': '980356255970', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183785.923935, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183785.923935, 'rx': '45364975', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183785.923935, 'rx': '39843280', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40566', 'ksmPages': 100, 'elapsedTime': '1728534.02', 'cpuLoad': '7.80', 'cpuSys': '2.43', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.31', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150383, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:29:45 GMT', 'cpuUser': '30.91', 'memFree': 150127, 'cpuIdle': '66.66', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '0.99'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:29:45,947+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:29:46,613+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=6792a58e-59ef-43c9-92c7-133e6f31f984 (api:48) 2019-04-13 21:29:46,614+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000669525', 'lastCheck': '2.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000698852', 'lastCheck': '2.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000630638', 'lastCheck': '2.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000392636', 'lastCheck': '2.4', 'valid': True}} from=internal, task_id=6792a58e-59ef-43c9-92c7-133e6f31f984 (api:54) 2019-04-13 21:29:49,079+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:29:49,120+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:29:49,120+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:29:49,121+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:29:51,521+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=e5f8fe9a-b8c1-4091-acb8-3b35706846e9 (api:48) 2019-04-13 21:29:51,527+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=e5f8fe9a-b8c1-4091-acb8-3b35706846e9 (api:54) 2019-04-13 21:29:51,527+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:29:51,542+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=8baec2a3-6126-4455-bf6d-4f6c6055d360 (api:48) 2019-04-13 21:29:51,550+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707019198464', 'isoprefix': '', 'alerts': [], 'disktotal': '3898116931584', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707019198464', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719767785472', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707019198464', 'isoprefix': '', 'alerts': [], 'disktotal': '4364408193024', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=8baec2a3-6126-4455-bf6d-4f6c6055d360 (api:54) 2019-04-13 21:29:51,551+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:29:52,411+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:29:52,441+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:29:52,442+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:29:52,442+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:29:57,755+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_isos) Closing client (__init__:584) 2019-04-13 21:29:57,755+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/glusterSD/onode-2.discovery.intern:vol-node-2) Closing client (__init__:584) 2019-04-13 21:29:57,755+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_vms) Closing client (__init__:584) 2019-04-13 21:29:57,755+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/onode-1.discovery.intern:_mnt_store) Closing client (__init__:584) 2019-04-13 21:29:57,755+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Closing client (__init__:584) 2019-04-13 21:30:00,928+0200 INFO (jsonrpc/4) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:30:01,004+0200 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=6a068e78-0eb0-4d4a-a414-59283624f04e (api:48) 2019-04-13 21:30:01,005+0200 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000681956', 'lastCheck': '6.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000692223', 'lastCheck': '6.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000549857', 'lastCheck': '6.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000421785', 'lastCheck': '3.2', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=6a068e78-0eb0-4d4a-a414-59283624f04e (api:54) 2019-04-13 21:30:01,006+0200 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=4e55b9ae-e8ce-4d01-84f5-06c77f5199a3 (api:48) 2019-04-13 21:30:01,006+0200 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=4e55b9ae-e8ce-4d01-84f5-06c77f5199a3 (api:54) 2019-04-13 21:30:01,020+0200 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '18.71', 'nodeIndex': 1, 'cpuSys': '1.98', 'cpuIdle': '79.31'}, '21': {'cpuUser': '22.22', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '75.53'}, '22': {'cpuUser': '25.46', 'nodeIndex': 1, 'cpuSys': '1.72', 'cpuIdle': '72.82'}, '23': {'cpuUser': '18.65', 'nodeIndex': 1, 'cpuSys': '1.98', 'cpuIdle': '79.37'}, '1': {'cpuUser': '45.16', 'nodeIndex': 0, 'cpuSys': '4.36', 'cpuIdle': '50.48'}, '0': {'cpuUser': '54.95', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '43.46'}, '3': {'cpuUser': '52.56', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '45.52'}, '2': {'cpuUser': '55.14', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '42.94'}, '5': {'cpuUser': '50.18', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '47.70'}, '4': {'cpuUser': '53.23', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '44.92'}, '7': {'cpuUser': '32.66', 'nodeIndex': 1, 'cpuSys': '3.37', 'cpuIdle': '63.97'}, '6': {'cpuUser': '32.93', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '64.09'}, '9': {'cpuUser': '27.77', 'nodeIndex': 1, 'cpuSys': '3.44', 'cpuIdle': '68.79'}, '8': {'cpuUser': '32.07', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '64.62'}, '11': {'cpuUser': '31.94', 'nodeIndex': 1, 'cpuSys': '3.24', 'cpuIdle': '64.82'}, '10': {'cpuUser': '35.24', 'nodeIndex': 1, 'cpuSys': '2.51', 'cpuIdle': '62.25'}, '13': {'cpuUser': '23.74', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '74.34'}, '12': {'cpuUser': '23.47', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '74.41'}, '15': {'cpuUser': '23.94', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '73.94'}, '14': {'cpuUser': '22.68', 'nodeIndex': 0, 'cpuSys': '2.38', 'cpuIdle': '74.94'}, '17': {'cpuUser': '30.22', 'nodeIndex': 0, 'cpuSys': '1.98', 'cpuIdle': '67.80'}, '16': {'cpuUser': '19.11', 'nodeIndex': 0, 'cpuSys': '2.31', 'cpuIdle': '78.58'}, '19': {'cpuUser': '17.59', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '79.83'}, '18': {'cpuUser': '22.15', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '75.60'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '70264'}, '0': {'memPercent': 22, 'memFree': '77462'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000681956', 'lastCheck': '6.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000549857', 'lastCheck': '6.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000421785', 'lastCheck': '3.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000692223', 'lastCheck': '6.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168859628', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '5091491510', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590464320477', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '2705637368570', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433926401940', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '259103302940', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239873616', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '366095654', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122799508', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121320795', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423773034024', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '253683950414', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215278594', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680504861', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '29547813143', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653287848', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '29289493034', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255873196356', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '7146908787', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121695348', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75362881845', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '61957636279', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122254278', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418940659', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '1012776443', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121506725', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183801.003577, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183801.003577, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183801.003577, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183801.003577, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121276966', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458757883353', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183801.003577, 'rx': '2576995146717', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322501880735', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183801.003577, 'rx': '1322501880735', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183801.003577, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940863852', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '2700686438219', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571523456625', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '4950930351', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329853593059', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183801.003577, 'rx': '980365543940', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183801.003577, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183801.003577, 'rx': '45365365', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183801.003577, 'rx': '39843648', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40566', 'ksmPages': 100, 'elapsedTime': '1728549.09', 'cpuLoad': '7.95', 'cpuSys': '2.43', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.04', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150389, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:30:01 GMT', 'cpuUser': '32.16', 'memFree': 150133, 'cpuIdle': '65.41', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.52'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:30:01,025+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.10 seconds (__init__:312) 2019-04-13 21:30:01,604+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=69739fc7-ea74-4c65-8b75-469c4e5ef135 (api:48) 2019-04-13 21:30:01,611+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=69739fc7-ea74-4c65-8b75-469c4e5ef135 (api:54) 2019-04-13 21:30:01,612+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:30:01,626+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=31f21876-c993-46f2-9dcd-f112367ed708 (api:48) 2019-04-13 21:30:01,681+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707019198464', 'isoprefix': '', 'alerts': [], 'disktotal': '3898116931584', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707019198464', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719767785472', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707019198464', 'isoprefix': '', 'alerts': [], 'disktotal': '4364408193024', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=31f21876-c993-46f2-9dcd-f112367ed708 (api:54) 2019-04-13 21:30:01,682+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.05 seconds (__init__:312) 2019-04-13 21:30:01,744+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=59347bdd-a1d4-4659-b8e8-cc50bb7a1f97 (api:48) 2019-04-13 21:30:01,744+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000681956', 'lastCheck': '7.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000692223', 'lastCheck': '7.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000549857', 'lastCheck': '7.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000421785', 'lastCheck': '4.0', 'valid': True}} from=internal, task_id=59347bdd-a1d4-4659-b8e8-cc50bb7a1f97 (api:54) 2019-04-13 21:30:04,114+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID=u'ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', options=None) from=internal, task_id=1ad1bb52-c81b-4b01-9be9-ca30f2ad9635 (api:48) 2019-04-13 21:30:04,116+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', options=None) from=internal, task_id=0441a69b-7b30-4796-bee9-bdcb7ce7c394 (api:48) 2019-04-13 21:30:04,119+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', options=None) from=internal, task_id=b6d6a50f-31be-4df1-a873-8939c300ef80 (api:48) 2019-04-13 21:30:04,124+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '5697701376', 'apparentsize': '16106127360'} from=internal, task_id=b6d6a50f-31be-4df1-a873-8939c300ef80 (api:54) 2019-04-13 21:30:04,125+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID='ef6741d1-380a-4736-a924-f88ed68e1270', options=None) from=internal, task_id=9aafe7e7-ebc1-443d-bf84-a868bdd027ef (api:48) 2019-04-13 21:30:04,127+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '4579684864', 'apparentsize': '16106127360'} from=internal, task_id=0441a69b-7b30-4796-bee9-bdcb7ce7c394 (api:54) 2019-04-13 21:30:04,128+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID=u'67a4dd71-69ff-49a8-b8a4-c7101c75e105', options=None) from=internal, task_id=61fcf250-269a-465c-ba90-c9e0281e1946 (api:48) 2019-04-13 21:30:04,129+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '53760865792', 'apparentsize': '139586437120'} from=internal, task_id=1ad1bb52-c81b-4b01-9be9-ca30f2ad9635 (api:54) 2019-04-13 21:30:04,130+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', options=None) from=internal, task_id=c6d910b1-861e-4190-bad1-0f67962a2ce4 (api:48) 2019-04-13 21:30:04,138+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID=u'2b136ed7-8b9a-45c1-9edc-64ee96c87c43', options=None) from=internal, task_id=680a98ec-e28c-4699-a91a-97e788edd4ce (api:48) 2019-04-13 21:30:04,139+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '55059099136', 'apparentsize': '171798691840'} from=internal, task_id=61fcf250-269a-465c-ba90-c9e0281e1946 (api:54) 2019-04-13 21:30:04,140+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID=u'44b3ceca-f0db-43da-af29-aa681df146ac', options=None) from=internal, task_id=2d7bd1a2-378c-42b2-aaa7-fa66d74a1d9b (api:48) 2019-04-13 21:30:04,141+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '6477462016', 'apparentsize': '21474836480'} from=internal, task_id=9aafe7e7-ebc1-443d-bf84-a868bdd027ef (api:54) 2019-04-13 21:30:04,141+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=ac1a0f1f-4071-4ea9-aecd-0a8b5f1589ea (api:48) 2019-04-13 21:30:04,145+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:30:04,166+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2524562432', 'apparentsize': '12884901888'} from=internal, task_id=c6d910b1-861e-4190-bad1-0f67962a2ce4 (api:54) 2019-04-13 21:30:04,169+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID='123e2cc8-8f24-4ddd-b401-68393535f555', options=None) from=internal, task_id=c3ec86a4-4fc1-4335-bb92-2b82200bc530 (api:48) 2019-04-13 21:30:04,188+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2024405504', 'apparentsize': '10737418240'} from=internal, task_id=680a98ec-e28c-4699-a91a-97e788edd4ce (api:54) 2019-04-13 21:30:04,191+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID='3f6c31c6-ec38-4979-a617-55537123ee61', options=None) from=internal, task_id=e0d5a932-952e-46e8-9d9b-3d416c023096 (api:48) 2019-04-13 21:30:04,192+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '11196667392', 'apparentsize': '21474836480'} from=internal, task_id=2d7bd1a2-378c-42b2-aaa7-fa66d74a1d9b (api:54) 2019-04-13 21:30:04,193+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', options=None) from=internal, task_id=37f30126-00f5-4424-8ae9-a0e843731509 (api:48) 2019-04-13 21:30:04,194+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2116486656', 'apparentsize': '3948150784'} from=internal, task_id=ac1a0f1f-4071-4ea9-aecd-0a8b5f1589ea (api:54) 2019-04-13 21:30:04,195+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID=u'a1d583c1-e135-4e3f-b6fa-0f29a5261366', options=None) from=internal, task_id=921ab08f-6469-4635-9635-6d6a1546449d (api:48) 2019-04-13 21:30:04,201+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2283003904', 'apparentsize': '16106127360'} from=internal, task_id=c3ec86a4-4fc1-4335-bb92-2b82200bc530 (api:54) 2019-04-13 21:30:04,202+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', options=None) from=internal, task_id=c144bc0c-bd24-4ed7-b34f-e0dbfedb6a0a (api:48) 2019-04-13 21:30:04,218+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2109410304', 'apparentsize': '8589934592'} from=internal, task_id=37f30126-00f5-4424-8ae9-a0e843731509 (api:54) 2019-04-13 21:30:04,221+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID='a884b4ce-381f-4653-be14-a31d15181f48', options=None) from=internal, task_id=b5ed45d3-186c-4a16-ada2-ed26d153fd17 (api:48) 2019-04-13 21:30:04,222+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '4998711296', 'apparentsize': '16106127360'} from=internal, task_id=e0d5a932-952e-46e8-9d9b-3d416c023096 (api:54) 2019-04-13 21:30:04,223+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', options=None) from=internal, task_id=0cc6e0fb-ece9-4979-89dc-473fb6f81c58 (api:48) 2019-04-13 21:30:04,224+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '1294894592', 'apparentsize': '12884901888'} from=internal, task_id=921ab08f-6469-4635-9635-6d6a1546449d (api:54) 2019-04-13 21:30:04,228+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '7263185920', 'apparentsize': '21474836480'} from=internal, task_id=c144bc0c-bd24-4ed7-b34f-e0dbfedb6a0a (api:54) 2019-04-13 21:30:04,231+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '3447292416', 'apparentsize': '16106127360'} from=internal, task_id=b5ed45d3-186c-4a16-ada2-ed26d153fd17 (api:54) 2019-04-13 21:30:04,232+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2879759360', 'apparentsize': '8589934592'} from=internal, task_id=0cc6e0fb-ece9-4979-89dc-473fb6f81c58 (api:54) 2019-04-13 21:30:04,259+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:30:04,260+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:30:04,262+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.11 seconds (__init__:312) 2019-04-13 21:30:07,449+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:30:07,488+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:30:07,488+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:30:07,490+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:30:11,729+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=f824d856-3d42-4323-a1d1-b521056b4ae4 (api:48) 2019-04-13 21:30:11,735+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=f824d856-3d42-4323-a1d1-b521056b4ae4 (api:54) 2019-04-13 21:30:11,735+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:30:11,752+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=498ecaf0-0213-4688-969a-3e2b06a99f28 (api:48) 2019-04-13 21:30:11,760+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707015004160', 'isoprefix': '', 'alerts': [], 'disktotal': '3898111688704', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707015004160', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719762542592', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707015004160', 'isoprefix': '', 'alerts': [], 'disktotal': '4364402950144', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=498ecaf0-0213-4688-969a-3e2b06a99f28 (api:54) 2019-04-13 21:30:11,761+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:30:15,979+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:30:16,062+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=58fd2683-d87d-400f-aff2-3cbdeb1ac0c4 (api:48) 2019-04-13 21:30:16,063+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00070356', 'lastCheck': '1.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000685971', 'lastCheck': '1.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000650832', 'lastCheck': '1.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000434831', 'lastCheck': '1.9', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=58fd2683-d87d-400f-aff2-3cbdeb1ac0c4 (api:54) 2019-04-13 21:30:16,064+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=e51880db-7bf4-4022-9603-18baedae670c (api:48) 2019-04-13 21:30:16,065+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=e51880db-7bf4-4022-9603-18baedae670c (api:54) 2019-04-13 21:30:16,088+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '22.67', 'nodeIndex': 1, 'cpuSys': '2.39', 'cpuIdle': '74.94'}, '21': {'cpuUser': '15.71', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '81.31'}, '22': {'cpuUser': '26.19', 'nodeIndex': 1, 'cpuSys': '2.06', 'cpuIdle': '71.75'}, '23': {'cpuUser': '29.24', 'nodeIndex': 1, 'cpuSys': '1.79', 'cpuIdle': '68.97'}, '1': {'cpuUser': '35.67', 'nodeIndex': 0, 'cpuSys': '4.11', 'cpuIdle': '60.22'}, '0': {'cpuUser': '37.66', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '60.09'}, '3': {'cpuUser': '45.21', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '52.67'}, '2': {'cpuUser': '45.35', 'nodeIndex': 0, 'cpuSys': '2.39', 'cpuIdle': '52.26'}, '5': {'cpuUser': '42.89', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '54.99'}, '4': {'cpuUser': '36.59', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '60.96'}, '7': {'cpuUser': '32.75', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '64.20'}, '6': {'cpuUser': '37.46', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '59.89'}, '9': {'cpuUser': '35.60', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '61.35'}, '8': {'cpuUser': '36.46', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '60.76'}, '11': {'cpuUser': '29.30', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '67.65'}, '10': {'cpuUser': '35.67', 'nodeIndex': 1, 'cpuSys': '3.51', 'cpuIdle': '60.82'}, '13': {'cpuUser': '26.78', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '71.23'}, '12': {'cpuUser': '26.52', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '71.62'}, '15': {'cpuUser': '25.13', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '72.88'}, '14': {'cpuUser': '5.90', 'nodeIndex': 0, 'cpuSys': '2.85', 'cpuIdle': '91.25'}, '17': {'cpuUser': '15.84', 'nodeIndex': 0, 'cpuSys': '2.52', 'cpuIdle': '81.64'}, '16': {'cpuUser': '24.93', 'nodeIndex': 0, 'cpuSys': '1.33', 'cpuIdle': '73.74'}, '19': {'cpuUser': '21.68', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '75.67'}, '18': {'cpuUser': '25.46', 'nodeIndex': 1, 'cpuSys': '2.06', 'cpuIdle': '72.48'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '70100'}, '0': {'memPercent': 22, 'memFree': '77630'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00070356', 'lastCheck': '1.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000650832', 'lastCheck': '1.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000434831', 'lastCheck': '1.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000685971', 'lastCheck': '1.8', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168859719', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '5091491510', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590465223551', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '2705638531972', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433930522288', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '259105605935', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239874043', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '366096052', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122799599', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121320886', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423776754938', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '253686511889', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215278685', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680505018', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '29547813198', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653292204', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '29289494262', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255875512518', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '7148984058', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121695439', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75365517861', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '61961864389', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122254369', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418941139', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '1012776443', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121506816', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183816.060076, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183816.060076, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183816.060076, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183816.060076, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121277057', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458758829249', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183816.060076, 'rx': '2576996253456', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322514361166', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183816.060076, 'rx': '1322514361166', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183816.060076, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940863980', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '2700687601141', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571524359571', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '4950930831', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329862922761', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183816.060076, 'rx': '980376371913', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183816.060076, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183816.060076, 'rx': '45365755', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183816.060076, 'rx': '39844016', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1728564.14', 'cpuLoad': '7.97', 'cpuSys': '2.50', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '2.92', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150389, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:30:16 GMT', 'cpuUser': '29.86', 'memFree': 150133, 'cpuIdle': '67.64', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.19'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:30:16,101+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.12 seconds (__init__:312) 2019-04-13 21:30:16,832+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=23dacbdc-5c92-4e97-ae6c-fd0794fa1ef8 (api:48) 2019-04-13 21:30:16,833+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00070356', 'lastCheck': '2.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000685971', 'lastCheck': '2.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000650832', 'lastCheck': '2.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000434831', 'lastCheck': '2.6', 'valid': True}} from=internal, task_id=23dacbdc-5c92-4e97-ae6c-fd0794fa1ef8 (api:54) 2019-04-13 21:30:19,336+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:30:19,359+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:30:19,359+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:30:19,361+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.02 seconds (__init__:312) 2019-04-13 21:30:21,802+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=0dc17679-3ed6-4ecf-b2c7-7366989481de (api:48) 2019-04-13 21:30:21,807+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=0dc17679-3ed6-4ecf-b2c7-7366989481de (api:54) 2019-04-13 21:30:21,808+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:30:21,822+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=1b9a3e18-b372-4f4a-904b-ed74321ebecc (api:48) 2019-04-13 21:30:21,830+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707009761280', 'isoprefix': '', 'alerts': [], 'disktotal': '3898107494400', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707009761280', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719758348288', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707009761280', 'isoprefix': '', 'alerts': [], 'disktotal': '4364398755840', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=1b9a3e18-b372-4f4a-904b-ed74321ebecc (api:54) 2019-04-13 21:30:21,831+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:30:22,497+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:30:22,533+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:30:22,534+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:30:22,535+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:30:31,025+0200 INFO (jsonrpc/1) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:30:31,096+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=bf383822-2183-403a-b894-ff3cbe923219 (api:48) 2019-04-13 21:30:31,096+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000662361', 'lastCheck': '6.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000702129', 'lastCheck': '6.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000618801', 'lastCheck': '6.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000222684', 'lastCheck': '3.3', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=bf383822-2183-403a-b894-ff3cbe923219 (api:54) 2019-04-13 21:30:31,097+0200 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=c2aea2fb-0b7a-493d-b946-9240f01d2e0f (api:48) 2019-04-13 21:30:31,098+0200 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=c2aea2fb-0b7a-493d-b946-9240f01d2e0f (api:54) 2019-04-13 21:30:31,112+0200 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '22.48', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '75.27'}, '21': {'cpuUser': '28.70', 'nodeIndex': 1, 'cpuSys': '2.05', 'cpuIdle': '69.25'}, '22': {'cpuUser': '20.37', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '77.38'}, '23': {'cpuUser': '19.64', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '77.65'}, '1': {'cpuUser': '48.47', 'nodeIndex': 0, 'cpuSys': '4.50', 'cpuIdle': '47.03'}, '0': {'cpuUser': '44.64', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '53.64'}, '3': {'cpuUser': '61.43', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '36.98'}, '2': {'cpuUser': '51.98', 'nodeIndex': 0, 'cpuSys': '2.18', 'cpuIdle': '45.84'}, '5': {'cpuUser': '50.19', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '47.96'}, '4': {'cpuUser': '49.66', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '48.49'}, '7': {'cpuUser': '33.39', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '63.30'}, '6': {'cpuUser': '35.58', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '61.71'}, '9': {'cpuUser': '33.92', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '62.97'}, '8': {'cpuUser': '38.09', 'nodeIndex': 1, 'cpuSys': '3.04', 'cpuIdle': '58.87'}, '11': {'cpuUser': '37.43', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '59.79'}, '10': {'cpuUser': '34.78', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '62.44'}, '13': {'cpuUser': '26.78', 'nodeIndex': 0, 'cpuSys': '2.51', 'cpuIdle': '70.71'}, '12': {'cpuUser': '38.55', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '59.60'}, '15': {'cpuUser': '19.51', 'nodeIndex': 0, 'cpuSys': '2.38', 'cpuIdle': '78.11'}, '14': {'cpuUser': '28.70', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '69.05'}, '17': {'cpuUser': '36.57', 'nodeIndex': 0, 'cpuSys': '1.98', 'cpuIdle': '61.45'}, '16': {'cpuUser': '34.92', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '62.96'}, '19': {'cpuUser': '26.32', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '71.43'}, '18': {'cpuUser': '25.79', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '72.09'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '69965'}, '0': {'memPercent': 21, 'memFree': '77768'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000662361', 'lastCheck': '6.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000618801', 'lastCheck': '6.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000222684', 'lastCheck': '3.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000702129', 'lastCheck': '6.8', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168860310', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '5091491828', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590465873715', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '2705639670059', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433934432018', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '259108432783', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239874418', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '366096052', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122799974', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121321261', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423781092006', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '253688923785', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215279060', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680505393', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '29547813198', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653292415', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '29289494426', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255878359910', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '7150773365', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121695814', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75368157975', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '61966355407', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122254744', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418941559', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '1012776443', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121507191', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183831.094826, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183831.094826, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183831.094826, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183831.094826, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121277432', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458759417571', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183831.094826, 'rx': '2576997340681', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322525760379', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183831.094826, 'rx': '1322525760379', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183831.094826, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940864396', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '2700688738658', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571525009319', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '4950931401', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329871880688', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183831.094826, 'rx': '980387688004', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183831.094826, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183831.094826, 'rx': '45366145', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183831.094826, 'rx': '39844338', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1728579.19', 'cpuLoad': '8.07', 'cpuSys': '2.42', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.64', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150389, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:30:31 GMT', 'cpuUser': '35.33', 'memFree': 150133, 'cpuIdle': '62.25', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.72'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:30:31,118+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:30:31,876+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=b03ec7e8-71b9-4c62-b7c4-9f63d7531ad7 (api:48) 2019-04-13 21:30:31,881+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=b03ec7e8-71b9-4c62-b7c4-9f63d7531ad7 (api:54) 2019-04-13 21:30:31,881+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:30:31,896+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=951e0f8b-381b-4ebb-a42c-7c5c4d17ae36 (api:48) 2019-04-13 21:30:31,906+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707009761280', 'isoprefix': '', 'alerts': [], 'disktotal': '3898107494400', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707009761280', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719758348288', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707009761280', 'isoprefix': '', 'alerts': [], 'disktotal': '4364398755840', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=951e0f8b-381b-4ebb-a42c-7c5c4d17ae36 (api:54) 2019-04-13 21:30:31,908+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:30:31,960+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=5d27dcd3-e522-4b36-abae-12beea1eb7a4 (api:48) 2019-04-13 21:30:31,960+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000662361', 'lastCheck': '7.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000702129', 'lastCheck': '7.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000618801', 'lastCheck': '7.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000222684', 'lastCheck': '4.2', 'valid': True}} from=internal, task_id=5d27dcd3-e522-4b36-abae-12beea1eb7a4 (api:54) 2019-04-13 21:30:34,375+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:30:34,404+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:30:34,404+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:30:34,406+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:30:37,542+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:30:37,571+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:30:37,572+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:30:37,573+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:30:41,960+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=bdbe1442-2481-4544-b745-eb9910a318b0 (api:48) 2019-04-13 21:30:41,965+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=bdbe1442-2481-4544-b745-eb9910a318b0 (api:54) 2019-04-13 21:30:41,965+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:30:42,080+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=b29de3a8-a5f5-418c-be38-ef875e34900f (api:48) 2019-04-13 21:30:42,088+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707005566976', 'isoprefix': '', 'alerts': [], 'disktotal': '3898103300096', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707005566976', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719754153984', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707005566976', 'isoprefix': '', 'alerts': [], 'disktotal': '4364394561536', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=b29de3a8-a5f5-418c-be38-ef875e34900f (api:54) 2019-04-13 21:30:42,089+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:30:46,089+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:30:46,153+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=86fefaf8-f407-4142-9ab0-5c4a152dea8a (api:48) 2019-04-13 21:30:46,153+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000680278', 'lastCheck': '1.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00045626', 'lastCheck': '1.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000513342', 'lastCheck': '1.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000423433', 'lastCheck': '2.0', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=86fefaf8-f407-4142-9ab0-5c4a152dea8a (api:54) 2019-04-13 21:30:46,154+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=287dea4e-703c-4611-a3fc-e92ed2098154 (api:48) 2019-04-13 21:30:46,154+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=287dea4e-703c-4611-a3fc-e92ed2098154 (api:54) 2019-04-13 21:30:46,169+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '25.64', 'nodeIndex': 1, 'cpuSys': '2.05', 'cpuIdle': '72.31'}, '21': {'cpuUser': '31.27', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '66.21'}, '22': {'cpuUser': '24.84', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '72.71'}, '23': {'cpuUser': '21.86', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '75.82'}, '1': {'cpuUser': '46.18', 'nodeIndex': 0, 'cpuSys': '3.64', 'cpuIdle': '50.18'}, '0': {'cpuUser': '43.06', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '54.69'}, '3': {'cpuUser': '44.98', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '52.83'}, '2': {'cpuUser': '50.35', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '47.60'}, '5': {'cpuUser': '53.46', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '44.95'}, '4': {'cpuUser': '53.73', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '44.42'}, '7': {'cpuUser': '38.03', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '58.86'}, '6': {'cpuUser': '34.52', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '62.63'}, '9': {'cpuUser': '37.10', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '59.59'}, '8': {'cpuUser': '35.44', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '61.51'}, '11': {'cpuUser': '41.87', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '55.55'}, '10': {'cpuUser': '39.22', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '58.13'}, '13': {'cpuUser': '34.52', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '63.43'}, '12': {'cpuUser': '36.04', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '61.77'}, '15': {'cpuUser': '38.49', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '59.79'}, '14': {'cpuUser': '31.73', 'nodeIndex': 0, 'cpuSys': '2.78', 'cpuIdle': '65.49'}, '17': {'cpuUser': '35.71', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '62.37'}, '16': {'cpuUser': '18.42', 'nodeIndex': 0, 'cpuSys': '2.78', 'cpuIdle': '78.80'}, '19': {'cpuUser': '23.19', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '74.23'}, '18': {'cpuUser': '26.90', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '70.98'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '69948'}, '0': {'memPercent': 21, 'memFree': '77784'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000680278', 'lastCheck': '1.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000513342', 'lastCheck': '1.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000423433', 'lastCheck': '2.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00045626', 'lastCheck': '1.9', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168860729', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '5091492004', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590466209331', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '2705639699553', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433938201736', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '259110296740', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239874601', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '366096052', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122800157', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121321444', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423784693698', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '253691591040', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215279243', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680505576', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '29547813198', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653292598', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '29289494426', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255880290545', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '7154726955', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121695997', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75370551995', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '61970583955', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122254927', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418942039', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '1012776443', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121507374', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183846.151724, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183846.151724, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183846.151724, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183846.151724, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121277615', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458759732725', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183846.151724, 'rx': '2576997364202', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322538243222', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183846.151724, 'rx': '1322538243222', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183846.151724, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940864524', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '2700688767680', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571525344807', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '4950931873', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329883242301', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183846.151724, 'rx': '980398621513', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183846.151724, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183846.151724, 'rx': '45366535', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183846.151724, 'rx': '39844706', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1728594.25', 'cpuLoad': '8.38', 'cpuSys': '2.44', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '2.91', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150389, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:30:46 GMT', 'cpuUser': '36.10', 'memFree': 150133, 'cpuIdle': '61.46', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.19'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:30:46,181+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:30:47,039+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=f6834dbe-165f-4653-9559-a8c0b66d6f35 (api:48) 2019-04-13 21:30:47,040+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000680278', 'lastCheck': '2.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00045626', 'lastCheck': '2.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000513342', 'lastCheck': '2.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000423433', 'lastCheck': '2.8', 'valid': True}} from=internal, task_id=f6834dbe-165f-4653-9559-a8c0b66d6f35 (api:54) 2019-04-13 21:30:49,456+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:30:49,484+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:30:49,484+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:30:49,485+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:30:52,144+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=466c3360-aa32-46b1-b1e5-db070c584be8 (api:48) 2019-04-13 21:30:52,149+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=466c3360-aa32-46b1-b1e5-db070c584be8 (api:54) 2019-04-13 21:30:52,149+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:30:52,162+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=591ab234-9ae8-479b-9099-ea77ff206d45 (api:48) 2019-04-13 21:30:52,171+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707005566976', 'isoprefix': '', 'alerts': [], 'disktotal': '3898103300096', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707005566976', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719754153984', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707005566976', 'isoprefix': '', 'alerts': [], 'disktotal': '4364394561536', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=591ab234-9ae8-479b-9099-ea77ff206d45 (api:54) 2019-04-13 21:30:52,173+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:30:52,579+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:30:52,608+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:30:52,609+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:30:52,610+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:30:57,110+0200 WARN (qgapoller/5) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:30:57,572+0200 WARN (qgapoller/6) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090050> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:30:57,807+0200 WARN (qgapoller/6) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc086ed8> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:31:01,132+0200 INFO (jsonrpc/4) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:31:01,210+0200 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=31316a2e-53d7-4eae-82c6-ea6651d0d5bb (api:48) 2019-04-13 21:31:01,211+0200 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000612422', 'lastCheck': '6.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000708255', 'lastCheck': '6.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.0005632', 'lastCheck': '6.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000352753', 'lastCheck': '3.4', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=31316a2e-53d7-4eae-82c6-ea6651d0d5bb (api:54) 2019-04-13 21:31:01,212+0200 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=fb8941e3-58c2-4127-bdb3-dc280a15e7bd (api:48) 2019-04-13 21:31:01,212+0200 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=fb8941e3-58c2-4127-bdb3-dc280a15e7bd (api:54) 2019-04-13 21:31:01,227+0200 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '14.68', 'nodeIndex': 1, 'cpuSys': '2.18', 'cpuIdle': '83.14'}, '21': {'cpuUser': '21.09', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '76.66'}, '22': {'cpuUser': '16.13', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '81.29'}, '23': {'cpuUser': '15.27', 'nodeIndex': 1, 'cpuSys': '2.38', 'cpuIdle': '82.35'}, '1': {'cpuUser': '54.01', 'nodeIndex': 0, 'cpuSys': '4.30', 'cpuIdle': '41.69'}, '0': {'cpuUser': '50.24', 'nodeIndex': 0, 'cpuSys': '1.98', 'cpuIdle': '47.78'}, '3': {'cpuUser': '56.79', 'nodeIndex': 0, 'cpuSys': '2.31', 'cpuIdle': '40.90'}, '2': {'cpuUser': '53.02', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '45.06'}, '5': {'cpuUser': '58.77', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '39.64'}, '4': {'cpuUser': '54.41', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '43.87'}, '7': {'cpuUser': '30.74', 'nodeIndex': 1, 'cpuSys': '2.84', 'cpuIdle': '66.42'}, '6': {'cpuUser': '30.28', 'nodeIndex': 1, 'cpuSys': '3.17', 'cpuIdle': '66.55'}, '9': {'cpuUser': '30.08', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '66.81'}, '8': {'cpuUser': '28.30', 'nodeIndex': 1, 'cpuSys': '3.44', 'cpuIdle': '68.26'}, '11': {'cpuUser': '30.48', 'nodeIndex': 1, 'cpuSys': '3.04', 'cpuIdle': '66.48'}, '10': {'cpuUser': '29.88', 'nodeIndex': 1, 'cpuSys': '3.11', 'cpuIdle': '67.01'}, '13': {'cpuUser': '27.57', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '70.51'}, '12': {'cpuUser': '30.41', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '67.34'}, '15': {'cpuUser': '11.64', 'nodeIndex': 0, 'cpuSys': '3.11', 'cpuIdle': '85.25'}, '14': {'cpuUser': '34.58', 'nodeIndex': 0, 'cpuSys': '1.98', 'cpuIdle': '63.44'}, '17': {'cpuUser': '21.88', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '75.87'}, '16': {'cpuUser': '25.39', 'nodeIndex': 0, 'cpuSys': '1.98', 'cpuIdle': '72.63'}, '19': {'cpuUser': '24.13', 'nodeIndex': 1, 'cpuSys': '1.85', 'cpuIdle': '74.02'}, '18': {'cpuUser': '29.15', 'nodeIndex': 1, 'cpuSys': '1.59', 'cpuIdle': '69.26'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '69970'}, '0': {'memPercent': 21, 'memFree': '77761'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000612422', 'lastCheck': '6.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.0005632', 'lastCheck': '6.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000352753', 'lastCheck': '3.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000708255', 'lastCheck': '6.9', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168861586', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '5091492276', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590466758733', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '2705641239628', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433941906990', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '259112571095', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239875730', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '366096052', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122801286', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121322573', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423788591996', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '253693550430', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215280372', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680506705', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '29547813198', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653293645', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '29289494508', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255882589023', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '7156303498', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121697126', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75372598147', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '61970601964', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122256056', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418942459', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '1012776443', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121508503', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183861.20953, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183861.20953, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183861.20953, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183861.20953, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121278744', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458760257353', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183861.20953, 'rx': '2576998837449', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322547560023', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183861.20953, 'rx': '1322547560023', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183861.20953, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940864524', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '2700690306750', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571525894209', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '4950932878', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329886759664', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183861.20953, 'rx': '980408488338', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183861.20953, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183861.20953, 'rx': '45366925', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183861.20953, 'rx': '39845028', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1728609.29', 'cpuLoad': '8.34', 'cpuSys': '2.46', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.11', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150391, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:31:01 GMT', 'cpuUser': '32.46', 'memFree': 150135, 'cpuIdle': '65.08', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.19'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:31:01,232+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.10 seconds (__init__:312) 2019-04-13 21:31:02,126+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=6f6c86ec-e820-4e69-ac7a-1630115805dd (api:48) 2019-04-13 21:31:02,127+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000612422', 'lastCheck': '7.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000708255', 'lastCheck': '7.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.0005632', 'lastCheck': '7.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000352753', 'lastCheck': '4.3', 'valid': True}} from=internal, task_id=6f6c86ec-e820-4e69-ac7a-1630115805dd (api:54) 2019-04-13 21:31:02,232+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=01d22c17-f61e-4429-9567-efeb5bb5f8bb (api:48) 2019-04-13 21:31:02,237+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=01d22c17-f61e-4429-9567-efeb5bb5f8bb (api:54) 2019-04-13 21:31:02,238+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:31:02,254+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=c78a062d-beb9-4a9c-b63d-59df06aa8c9c (api:48) 2019-04-13 21:31:02,261+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707001372672', 'isoprefix': '', 'alerts': [], 'disktotal': '3898099105792', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707001372672', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719749959680', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707001372672', 'isoprefix': '', 'alerts': [], 'disktotal': '4364390367232', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=c78a062d-beb9-4a9c-b63d-59df06aa8c9c (api:54) 2019-04-13 21:31:02,262+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:31:04,107+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID=u'ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', options=None) from=internal, task_id=e9322eb0-8a1a-4b80-967e-03bbb733eebe (api:48) 2019-04-13 21:31:04,109+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', options=None) from=internal, task_id=9eddbab1-c9a0-4261-9db4-aabd0b84e7aa (api:48) 2019-04-13 21:31:04,110+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', options=None) from=internal, task_id=ca7e07d2-6bf9-446e-b373-9ed2caceb283 (api:48) 2019-04-13 21:31:04,113+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '4579684864', 'apparentsize': '16106127360'} from=internal, task_id=9eddbab1-c9a0-4261-9db4-aabd0b84e7aa (api:54) 2019-04-13 21:31:04,114+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID='ef6741d1-380a-4736-a924-f88ed68e1270', options=None) from=internal, task_id=b4d30ba9-5bd1-472f-aa8c-11144bd633a6 (api:48) 2019-04-13 21:31:04,116+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '5697701376', 'apparentsize': '16106127360'} from=internal, task_id=ca7e07d2-6bf9-446e-b373-9ed2caceb283 (api:54) 2019-04-13 21:31:04,116+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID=u'67a4dd71-69ff-49a8-b8a4-c7101c75e105', options=None) from=internal, task_id=eae15d23-b19a-468d-b075-e6b8a8862c2b (api:48) 2019-04-13 21:31:04,117+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '53760860160', 'apparentsize': '139586437120'} from=internal, task_id=e9322eb0-8a1a-4b80-967e-03bbb733eebe (api:54) 2019-04-13 21:31:04,118+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', options=None) from=internal, task_id=2eb7f236-7834-4358-b464-644ac984ce08 (api:48) 2019-04-13 21:31:04,121+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID=u'2b136ed7-8b9a-45c1-9edc-64ee96c87c43', options=None) from=internal, task_id=d452feee-a2e7-42a8-b5ff-cb334d0f1ded (api:48) 2019-04-13 21:31:04,122+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '6477462016', 'apparentsize': '21474836480'} from=internal, task_id=b4d30ba9-5bd1-472f-aa8c-11144bd633a6 (api:54) 2019-04-13 21:31:04,123+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID=u'44b3ceca-f0db-43da-af29-aa681df146ac', options=None) from=internal, task_id=ad746272-fbe1-4de3-a160-6ede592da98c (api:48) 2019-04-13 21:31:04,125+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '55059093504', 'apparentsize': '171798691840'} from=internal, task_id=eae15d23-b19a-468d-b075-e6b8a8862c2b (api:54) 2019-04-13 21:31:04,126+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=fd3639f8-8255-4e82-9d75-4dce0873c86e (api:48) 2019-04-13 21:31:04,127+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2524562432', 'apparentsize': '12884901888'} from=internal, task_id=2eb7f236-7834-4358-b464-644ac984ce08 (api:54) 2019-04-13 21:31:04,127+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID='123e2cc8-8f24-4ddd-b401-68393535f555', options=None) from=internal, task_id=e6fce2c7-640b-42ac-b22a-9e71dea92385 (api:48) 2019-04-13 21:31:04,129+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2024405504', 'apparentsize': '10737418240'} from=internal, task_id=d452feee-a2e7-42a8-b5ff-cb334d0f1ded (api:54) 2019-04-13 21:31:04,130+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID='3f6c31c6-ec38-4979-a617-55537123ee61', options=None) from=internal, task_id=08459fbf-fbc3-420d-8c3a-c082082e308b (api:48) 2019-04-13 21:31:04,132+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2116486656', 'apparentsize': '3948150784'} from=internal, task_id=fd3639f8-8255-4e82-9d75-4dce0873c86e (api:54) 2019-04-13 21:31:04,133+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', options=None) from=internal, task_id=9ebbf2f5-7f9c-49bb-933e-cca553e71ffd (api:48) 2019-04-13 21:31:04,133+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '11196661760', 'apparentsize': '21474836480'} from=internal, task_id=ad746272-fbe1-4de3-a160-6ede592da98c (api:54) 2019-04-13 21:31:04,134+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID=u'a1d583c1-e135-4e3f-b6fa-0f29a5261366', options=None) from=internal, task_id=91f39a9e-ec56-40ca-8c0b-a38f04534f06 (api:48) 2019-04-13 21:31:04,135+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2283003904', 'apparentsize': '16106127360'} from=internal, task_id=e6fce2c7-640b-42ac-b22a-9e71dea92385 (api:54) 2019-04-13 21:31:04,135+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', options=None) from=internal, task_id=822f7212-1631-4b4b-8097-f6b655a6cab4 (api:48) 2019-04-13 21:31:04,138+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '4998711296', 'apparentsize': '16106127360'} from=internal, task_id=08459fbf-fbc3-420d-8c3a-c082082e308b (api:54) 2019-04-13 21:31:04,139+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID='a884b4ce-381f-4653-be14-a31d15181f48', options=None) from=internal, task_id=b160846c-da36-4551-b735-6b3ef4c43754 (api:48) 2019-04-13 21:31:04,142+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '7263185920', 'apparentsize': '21474836480'} from=internal, task_id=822f7212-1631-4b4b-8097-f6b655a6cab4 (api:54) 2019-04-13 21:31:04,142+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', options=None) from=internal, task_id=62e56e1f-9feb-4533-b231-dccbd3d41dd1 (api:48) 2019-04-13 21:31:04,143+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '1294894592', 'apparentsize': '12884901888'} from=internal, task_id=91f39a9e-ec56-40ca-8c0b-a38f04534f06 (api:54) 2019-04-13 21:31:04,143+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2109410304', 'apparentsize': '8589934592'} from=internal, task_id=9ebbf2f5-7f9c-49bb-933e-cca553e71ffd (api:54) 2019-04-13 21:31:04,145+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '3447286784', 'apparentsize': '16106127360'} from=internal, task_id=b160846c-da36-4551-b735-6b3ef4c43754 (api:54) 2019-04-13 21:31:04,146+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2879759360', 'apparentsize': '8589934592'} from=internal, task_id=62e56e1f-9feb-4533-b231-dccbd3d41dd1 (api:54) 2019-04-13 21:31:04,536+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:31:04,569+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:31:04,570+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:31:04,571+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:31:07,118+0200 WARN (qgapoller/10) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:31:07,617+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:31:07,652+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:31:07,653+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:31:07,654+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:31:12,316+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=c9b91531-2fc2-4e18-a72c-86dcdc24aef1 (api:48) 2019-04-13 21:31:12,321+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=c9b91531-2fc2-4e18-a72c-86dcdc24aef1 (api:54) 2019-04-13 21:31:12,322+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:31:12,336+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=ff88f509-2f3c-48b1-a318-bfdd99a6f509 (api:48) 2019-04-13 21:31:12,344+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3707001372672', 'isoprefix': '', 'alerts': [], 'disktotal': '3898099105792', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3707001372672', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719749959680', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3707001372672', 'isoprefix': '', 'alerts': [], 'disktotal': '4364390367232', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=ff88f509-2f3c-48b1-a318-bfdd99a6f509 (api:54) 2019-04-13 21:31:12,345+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:31:16,182+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:31:16,250+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=1e9a2b02-a37a-485b-91e8-c16ced99ddfb (api:48) 2019-04-13 21:31:16,251+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000447329', 'lastCheck': '1.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000531342', 'lastCheck': '2.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000471142', 'lastCheck': '1.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000380967', 'lastCheck': '2.1', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=1e9a2b02-a37a-485b-91e8-c16ced99ddfb (api:54) 2019-04-13 21:31:16,252+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=84868b50-28ed-4896-91cb-84ab3debda78 (api:48) 2019-04-13 21:31:16,252+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=84868b50-28ed-4896-91cb-84ab3debda78 (api:54) 2019-04-13 21:31:16,267+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '22.95', 'nodeIndex': 1, 'cpuSys': '1.66', 'cpuIdle': '75.39'}, '21': {'cpuUser': '14.46', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '83.02'}, '22': {'cpuUser': '16.71', 'nodeIndex': 1, 'cpuSys': '1.99', 'cpuIdle': '81.30'}, '23': {'cpuUser': '15.05', 'nodeIndex': 1, 'cpuSys': '1.99', 'cpuIdle': '82.96'}, '1': {'cpuUser': '43.77', 'nodeIndex': 0, 'cpuSys': '3.98', 'cpuIdle': '52.25'}, '0': {'cpuUser': '47.49', 'nodeIndex': 0, 'cpuSys': '2.06', 'cpuIdle': '50.45'}, '3': {'cpuUser': '30.04', 'nodeIndex': 0, 'cpuSys': '2.59', 'cpuIdle': '67.37'}, '2': {'cpuUser': '46.03', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '51.78'}, '5': {'cpuUser': '50.60', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '47.41'}, '4': {'cpuUser': '48.02', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '50.19'}, '7': {'cpuUser': '29.18', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '67.77'}, '6': {'cpuUser': '28.85', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '68.17'}, '9': {'cpuUser': '26.73', 'nodeIndex': 1, 'cpuSys': '3.25', 'cpuIdle': '70.02'}, '8': {'cpuUser': '28.32', 'nodeIndex': 1, 'cpuSys': '3.38', 'cpuIdle': '68.30'}, '11': {'cpuUser': '27.66', 'nodeIndex': 1, 'cpuSys': '3.12', 'cpuIdle': '69.22'}, '10': {'cpuUser': '28.98', 'nodeIndex': 1, 'cpuSys': '3.25', 'cpuIdle': '67.77'}, '13': {'cpuUser': '30.37', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '68.04'}, '12': {'cpuUser': '30.64', 'nodeIndex': 0, 'cpuSys': '1.53', 'cpuIdle': '67.83'}, '15': {'cpuUser': '33.56', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '64.25'}, '14': {'cpuUser': '26.46', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '71.35'}, '17': {'cpuUser': '15.65', 'nodeIndex': 0, 'cpuSys': '2.06', 'cpuIdle': '82.29'}, '16': {'cpuUser': '23.81', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '74.00'}, '19': {'cpuUser': '18.97', 'nodeIndex': 1, 'cpuSys': '2.06', 'cpuIdle': '78.97'}, '18': {'cpuUser': '18.24', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '79.31'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '69949'}, '0': {'memPercent': 21, 'memFree': '77784'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000447329', 'lastCheck': '1.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000471142', 'lastCheck': '1.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000380967', 'lastCheck': '2.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000531342', 'lastCheck': '2.0', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168861668', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '5091492276', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590467155649', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '2705642378309', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433945363708', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '259114968510', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239876148', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '366096450', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122801368', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121322655', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423792001720', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '253695885490', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215280454', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680506949', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '29547813360', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653295036', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '29289495596', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255885006168', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '7158132420', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121697208', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75374940215', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '61976664679', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122256138', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418942939', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '1012776443', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121508585', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183876.249203, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183876.249203, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183876.249203, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183876.249203, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121278826', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458760617039', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183876.249203, 'rx': '2576999924459', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322561070422', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183876.249203, 'rx': '1322561070422', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183876.249203, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940865374', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '2700691444951', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571526290275', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '4950933358', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329897435921', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183876.249203, 'rx': '980418273269', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183876.249203, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183876.249203, 'rx': '45367315', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183876.249203, 'rx': '39845396', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1728624.34', 'cpuLoad': '8.32', 'cpuSys': '2.41', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '3.38', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150390, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:31:16 GMT', 'cpuUser': '29.27', 'memFree': 150134, 'cpuIdle': '68.32', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.53'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:31:16,272+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:31:17,111+0200 WARN (qgapoller/6) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:31:17,208+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=6f588e16-df44-4c39-ab72-315ed3b65dd7 (api:48) 2019-04-13 21:31:17,209+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000447329', 'lastCheck': '2.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000531342', 'lastCheck': '2.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000471142', 'lastCheck': '2.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000380967', 'lastCheck': '3.0', 'valid': True}} from=internal, task_id=6f588e16-df44-4c39-ab72-315ed3b65dd7 (api:54) 2019-04-13 21:31:19,613+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:31:19,644+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:31:19,644+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:31:19,645+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:31:22,393+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=b872d734-3e88-48b1-b184-52fbe7ba5a4e (api:48) 2019-04-13 21:31:22,398+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=b872d734-3e88-48b1-b184-52fbe7ba5a4e (api:54) 2019-04-13 21:31:22,399+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:31:22,411+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=a85c5627-3746-4cbb-9924-3b9777d6dc31 (api:48) 2019-04-13 21:31:22,419+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706995081216', 'isoprefix': '', 'alerts': [], 'disktotal': '3898092814336', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706995081216', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719743668224', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706995081216', 'isoprefix': '', 'alerts': [], 'disktotal': '4364384075776', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=a85c5627-3746-4cbb-9924-3b9777d6dc31 (api:54) 2019-04-13 21:31:22,420+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:31:22,662+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:31:22,686+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:31:22,686+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:31:22,687+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:31:24,096+0200 INFO (itmap/0) [IOProcessClient] (/onode-1.discovery.intern:_mnt_store) Starting client (__init__:308) 2019-04-13 21:31:24,120+0200 INFO (itmap/1) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_vms) Starting client (__init__:308) 2019-04-13 21:31:24,125+0200 INFO (ioprocess/29617) [IOProcess] (/onode-1.discovery.intern:_mnt_store) Starting ioprocess (__init__:434) 2019-04-13 21:31:24,142+0200 INFO (itmap/2) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_isos) Starting client (__init__:308) 2019-04-13 21:31:24,151+0200 INFO (ioprocess/29633) [IOProcess] (/nas.discovery.intern:_mnt_d2600_vms) Starting ioprocess (__init__:434) 2019-04-13 21:31:24,174+0200 INFO (itmap/3) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting client (__init__:308) 2019-04-13 21:31:24,181+0200 INFO (ioprocess/29652) [IOProcess] (/nas.discovery.intern:_mnt_d2600_isos) Starting ioprocess (__init__:434) 2019-04-13 21:31:24,217+0200 INFO (itmap/4) [IOProcessClient] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting client (__init__:308) 2019-04-13 21:31:24,231+0200 INFO (ioprocess/29663) [IOProcess] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting ioprocess (__init__:434) 2019-04-13 21:31:24,279+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:31:24,280+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:31:24,285+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:31:24,286+0200 INFO (ioprocess/29678) [IOProcess] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting ioprocess (__init__:434) 2019-04-13 21:31:27,109+0200 WARN (qgapoller/10) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:31:31,227+0200 INFO (jsonrpc/1) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:31:31,295+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=d41b629e-414f-493f-bf4c-c9a6f4ea2753 (api:48) 2019-04-13 21:31:31,296+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000652739', 'lastCheck': '7.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000584246', 'lastCheck': '7.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00134206', 'lastCheck': '7.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000228169', 'lastCheck': '3.5', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=d41b629e-414f-493f-bf4c-c9a6f4ea2753 (api:54) 2019-04-13 21:31:31,296+0200 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=4dd5f57e-0277-45b6-a86d-9fa596a4491f (api:48) 2019-04-13 21:31:31,297+0200 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=4dd5f57e-0277-45b6-a86d-9fa596a4491f (api:54) 2019-04-13 21:31:31,311+0200 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '15.97', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '81.38'}, '21': {'cpuUser': '26.57', 'nodeIndex': 1, 'cpuSys': '1.79', 'cpuIdle': '71.64'}, '22': {'cpuUser': '17.83', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '79.65'}, '23': {'cpuUser': '25.98', 'nodeIndex': 1, 'cpuSys': '1.92', 'cpuIdle': '72.10'}, '1': {'cpuUser': '33.80', 'nodeIndex': 0, 'cpuSys': '4.71', 'cpuIdle': '61.49'}, '0': {'cpuUser': '40.76', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '56.99'}, '3': {'cpuUser': '44.47', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '53.61'}, '2': {'cpuUser': '35.12', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '62.89'}, '5': {'cpuUser': '44.93', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '53.02'}, '4': {'cpuUser': '40.82', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '57.32'}, '7': {'cpuUser': '29.89', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '66.80'}, '6': {'cpuUser': '26.91', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '69.78'}, '9': {'cpuUser': '33.27', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '63.42'}, '8': {'cpuUser': '30.88', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '65.81'}, '11': {'cpuUser': '27.04', 'nodeIndex': 1, 'cpuSys': '3.38', 'cpuIdle': '69.58'}, '10': {'cpuUser': '28.63', 'nodeIndex': 1, 'cpuSys': '3.51', 'cpuIdle': '67.86'}, '13': {'cpuUser': '28.56', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '69.58'}, '12': {'cpuUser': '28.03', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '70.25'}, '15': {'cpuUser': '20.61', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '77.27'}, '14': {'cpuUser': '31.28', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '66.93'}, '17': {'cpuUser': '29.42', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '68.92'}, '16': {'cpuUser': '26.91', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '71.43'}, '19': {'cpuUser': '19.68', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '77.74'}, '18': {'cpuUser': '25.91', 'nodeIndex': 1, 'cpuSys': '1.66', 'cpuIdle': '72.43'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '70007'}, '0': {'memPercent': 21, 'memFree': '77726'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000652739', 'lastCheck': '7.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00134206', 'lastCheck': '7.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000228169', 'lastCheck': '3.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000584246', 'lastCheck': '7.0', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168862131', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '5091492276', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590467617816', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '2705643510971', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433949005874', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '259117421385', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239876611', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '366096450', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122801831', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121323118', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423795472834', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '253698296717', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215280917', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680507496', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '29547813444', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653295340', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '29289495923', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255887517613', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '7161907820', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121697671', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75377296327', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '61976687629', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122256601', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418943657', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '1012776767', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121509048', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183891.294255, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183891.294255, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183891.294255, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183891.294255, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121279289', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458761057191', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183891.294255, 'rx': '2577001006712', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322571444269', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183891.294255, 'rx': '1322571444269', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183891.294255, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940865621', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '2700692577281', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571526752195', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '4950933690', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329903752873', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183891.294255, 'rx': '980428044285', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183891.294255, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183891.294255, 'rx': '45367705', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183891.294255, 'rx': '39845718', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1728639.39', 'cpuLoad': '8.28', 'cpuSys': '2.46', 'diskStats': {'/var/log': {'free': '47798'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47798'}}, 'cpuUserVdsmd': '3.25', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150379, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:31:31 GMT', 'cpuUser': '29.72', 'memFree': 150123, 'cpuIdle': '67.83', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.39'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:31:31,317+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:31:32,299+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=46ce6f85-387f-4b93-b2fb-0c351e5c1b24 (api:48) 2019-04-13 21:31:32,300+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000652739', 'lastCheck': '8.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000584246', 'lastCheck': '8.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00134206', 'lastCheck': '8.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000228169', 'lastCheck': '4.5', 'valid': True}} from=internal, task_id=46ce6f85-387f-4b93-b2fb-0c351e5c1b24 (api:54) 2019-04-13 21:31:32,469+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=116d10bd-a005-444f-a411-2c3d02af9a58 (api:48) 2019-04-13 21:31:32,474+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=116d10bd-a005-444f-a411-2c3d02af9a58 (api:54) 2019-04-13 21:31:32,474+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:31:32,486+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=4a2bb173-5f40-4ffe-ba73-c0a9f11c8829 (api:48) 2019-04-13 21:31:32,494+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706995081216', 'isoprefix': '', 'alerts': [], 'disktotal': '3898092814336', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706995081216', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719743668224', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706995081216', 'isoprefix': '', 'alerts': [], 'disktotal': '4364384075776', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=4a2bb173-5f40-4ffe-ba73-c0a9f11c8829 (api:54) 2019-04-13 21:31:32,495+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:31:34,679+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:31:34,721+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:31:34,721+0200 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:31:34,722+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:31:37,111+0200 WARN (qgapoller/9) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:31:37,695+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:31:37,735+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:31:37,736+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:31:37,737+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:31:42,539+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=9e20ad04-9aac-4e2e-b7ec-5997f5725df9 (api:48) 2019-04-13 21:31:42,544+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=9e20ad04-9aac-4e2e-b7ec-5997f5725df9 (api:54) 2019-04-13 21:31:42,544+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:31:42,557+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=9e00e9f8-3534-42e8-9a1f-988e53af277d (api:48) 2019-04-13 21:31:42,565+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706987741184', 'isoprefix': '', 'alerts': [], 'disktotal': '3898085474304', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706987741184', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719736328192', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706987741184', 'isoprefix': '', 'alerts': [], 'disktotal': '4364376735744', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=9e00e9f8-3534-42e8-9a1f-988e53af277d (api:54) 2019-04-13 21:31:42,566+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:31:46,291+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:31:46,360+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=7d556008-d488-416b-8817-f707242f9920 (api:48) 2019-04-13 21:31:46,361+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000700934', 'lastCheck': '2.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000479327', 'lastCheck': '2.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000627525', 'lastCheck': '2.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000364083', 'lastCheck': '2.2', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=7d556008-d488-416b-8817-f707242f9920 (api:54) 2019-04-13 21:31:46,362+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=69e4eb34-002b-4841-8904-bf20df512b84 (api:48) 2019-04-13 21:31:46,362+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=69e4eb34-002b-4841-8904-bf20df512b84 (api:54) 2019-04-13 21:31:46,377+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '13.60', 'nodeIndex': 1, 'cpuSys': '2.79', 'cpuIdle': '83.61'}, '21': {'cpuUser': '17.25', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '80.03'}, '22': {'cpuUser': '21.23', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '76.32'}, '23': {'cpuUser': '22.69', 'nodeIndex': 1, 'cpuSys': '2.06', 'cpuIdle': '75.25'}, '1': {'cpuUser': '33.63', 'nodeIndex': 0, 'cpuSys': '4.84', 'cpuIdle': '61.53'}, '0': {'cpuUser': '46.77', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '51.37'}, '3': {'cpuUser': '50.88', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '47.40'}, '2': {'cpuUser': '51.01', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '47.00'}, '5': {'cpuUser': '49.16', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '49.12'}, '4': {'cpuUser': '41.26', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '56.75'}, '7': {'cpuUser': '32.57', 'nodeIndex': 1, 'cpuSys': '3.45', 'cpuIdle': '63.98'}, '6': {'cpuUser': '36.35', 'nodeIndex': 1, 'cpuSys': '3.25', 'cpuIdle': '60.40'}, '9': {'cpuUser': '37.41', 'nodeIndex': 1, 'cpuSys': '3.25', 'cpuIdle': '59.34'}, '8': {'cpuUser': '34.56', 'nodeIndex': 1, 'cpuSys': '3.45', 'cpuIdle': '61.99'}, '11': {'cpuUser': '36.35', 'nodeIndex': 1, 'cpuSys': '2.59', 'cpuIdle': '61.06'}, '10': {'cpuUser': '36.09', 'nodeIndex': 1, 'cpuSys': '3.52', 'cpuIdle': '60.39'}, '13': {'cpuUser': '27.86', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '70.55'}, '12': {'cpuUser': '28.46', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '69.42'}, '15': {'cpuUser': '20.83', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '76.98'}, '14': {'cpuUser': '19.17', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '78.71'}, '17': {'cpuUser': '15.46', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '82.35'}, '16': {'cpuUser': '35.16', 'nodeIndex': 0, 'cpuSys': '1.39', 'cpuIdle': '63.45'}, '19': {'cpuUser': '20.83', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '76.85'}, '18': {'cpuUser': '21.89', 'nodeIndex': 1, 'cpuSys': '2.26', 'cpuIdle': '75.85'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '69910'}, '0': {'memPercent': 21, 'memFree': '77809'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000700934', 'lastCheck': '2.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000627525', 'lastCheck': '2.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000364083', 'lastCheck': '2.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000479327', 'lastCheck': '2.1', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168863002', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '5091492678', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590468199342', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '2705643543000', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433953020160', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '259119990520', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239877146', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '366096450', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122802366', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121323653', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '856919', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423799378314', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '253700719368', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215281452', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680508031', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '29547813444', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653295711', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '29289496087', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255890056981', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '7163914003', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121698206', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75379220773', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '61984326331', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122257136', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418944137', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '1012776767', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121509583', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183906.35948, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183906.35948, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183906.35948, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183906.35948, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121279824', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458761616287', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183906.35948, 'rx': '2577001034849', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322583924218', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183906.35948, 'rx': '1322583924218', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183906.35948, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940866165', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '2700692608574', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571527333177', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '4950934426', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329916264943', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183906.35948, 'rx': '980438105032', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183906.35948, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183906.35948, 'rx': '45368095', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183906.35948, 'rx': '39846086', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40570', 'ksmPages': 100, 'elapsedTime': '1728654.45', 'cpuLoad': '8.18', 'cpuSys': '2.49', 'diskStats': {'/var/log': {'free': '47798'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47798'}}, 'cpuUserVdsmd': '3.05', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150385, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:31:46 GMT', 'cpuUser': '31.26', 'memFree': 150129, 'cpuIdle': '66.25', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.39'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:31:46,383+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:31:47,121+0200 WARN (qgapoller/10) [virt.periodic.VmDispatcher] could not run at 0x7fb9bc090140> on ['28d7d5ff-2ca1-42c6-ae8d-065c721d3066', '611486ee-b72f-4acc-8925-4d7b6df9af1d'] (periodic:289) 2019-04-13 21:31:47,391+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=8882ef79-65b6-4eb4-82fc-68547e42e4c3 (api:48) 2019-04-13 21:31:47,391+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000700934', 'lastCheck': '3.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000479327', 'lastCheck': '3.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000627525', 'lastCheck': '3.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000364083', 'lastCheck': '3.2', 'valid': True}} from=internal, task_id=8882ef79-65b6-4eb4-82fc-68547e42e4c3 (api:54) 2019-04-13 21:31:49,758+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:31:49,791+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:31:49,792+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:31:49,793+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:31:52,618+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=6538f39e-43d3-472f-b5e2-c8d4c117ad99 (api:48) 2019-04-13 21:31:52,624+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=6538f39e-43d3-472f-b5e2-c8d4c117ad99 (api:54) 2019-04-13 21:31:52,625+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:31:52,744+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:31:52,769+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=9fb71e27-6e43-44c9-9040-663b5f94d241 (api:48) 2019-04-13 21:31:52,778+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706987741184', 'isoprefix': '', 'alerts': [], 'disktotal': '3898085474304', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706987741184', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719736328192', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706987741184', 'isoprefix': '', 'alerts': [], 'disktotal': '4364376735744', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=9fb71e27-6e43-44c9-9040-663b5f94d241 (api:54) 2019-04-13 21:31:52,780+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:31:52,792+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:31:52,793+0200 ERROR (jsonrpc/6) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:31:52,793+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:32:01,340+0200 INFO (jsonrpc/4) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:32:01,409+0200 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=225e427b-65f1-4ae8-abba-7df200e54ebd (api:48) 2019-04-13 21:32:01,409+0200 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000758077', 'lastCheck': '7.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000722786', 'lastCheck': '7.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000680119', 'lastCheck': '7.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000418863', 'lastCheck': '3.6', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=225e427b-65f1-4ae8-abba-7df200e54ebd (api:54) 2019-04-13 21:32:01,410+0200 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=7946d913-6103-428b-a25d-859e56aa6078 (api:48) 2019-04-13 21:32:01,410+0200 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=7946d913-6103-428b-a25d-859e56aa6078 (api:54) 2019-04-13 21:32:01,425+0200 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '24.50', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '73.31'}, '21': {'cpuUser': '13.77', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '83.45'}, '22': {'cpuUser': '21.79', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '75.36'}, '23': {'cpuUser': '23.71', 'nodeIndex': 1, 'cpuSys': '1.92', 'cpuIdle': '74.37'}, '1': {'cpuUser': '30.79', 'nodeIndex': 0, 'cpuSys': '4.77', 'cpuIdle': '64.44'}, '0': {'cpuUser': '43.64', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '54.17'}, '3': {'cpuUser': '35.49', 'nodeIndex': 0, 'cpuSys': '2.58', 'cpuIdle': '61.93'}, '2': {'cpuUser': '36.49', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '61.52'}, '5': {'cpuUser': '45.43', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '52.78'}, '4': {'cpuUser': '38.41', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '59.47'}, '7': {'cpuUser': '32.32', 'nodeIndex': 1, 'cpuSys': '3.58', 'cpuIdle': '64.10'}, '6': {'cpuUser': '33.77', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '62.92'}, '9': {'cpuUser': '36.55', 'nodeIndex': 1, 'cpuSys': '3.44', 'cpuIdle': '60.01'}, '8': {'cpuUser': '35.30', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '61.39'}, '11': {'cpuUser': '35.83', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '61.19'}, '10': {'cpuUser': '32.38', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '64.44'}, '13': {'cpuUser': '25.76', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '72.58'}, '12': {'cpuUser': '19.47', 'nodeIndex': 0, 'cpuSys': '2.78', 'cpuIdle': '77.75'}, '15': {'cpuUser': '25.43', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '72.58'}, '14': {'cpuUser': '35.03', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '63.12'}, '17': {'cpuUser': '24.77', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '73.44'}, '16': {'cpuUser': '33.97', 'nodeIndex': 0, 'cpuSys': '1.39', 'cpuIdle': '64.64'}, '19': {'cpuUser': '24.97', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '72.45'}, '18': {'cpuUser': '21.46', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '75.96'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '69816'}, '0': {'memPercent': 21, 'memFree': '77911'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000758077', 'lastCheck': '7.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000680119', 'lastCheck': '7.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000418863', 'lastCheck': '3.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000722786', 'lastCheck': '7.1', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168863002', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '5091492678', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590468676834', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '2705645079672', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433956414720', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '259122423356', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239877146', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '366096450', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122802366', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121323743', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '857009', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423803165430', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '253703399741', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215281452', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680508031', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '29547813444', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653295711', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '29289496087', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255892550509', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '7165822860', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121698206', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75381876181', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '61990656943', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122257136', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418944557', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '1012776767', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121509583', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '1041398', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183921.407738, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183921.407738, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183921.407738, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183921.407738, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121279824', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458762076595', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183921.407738, 'rx': '2577002504633', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322594268277', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183921.407738, 'rx': '1322594268277', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183921.407738, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940866259', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '2700694144934', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571527810575', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '4950934738', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329927534601', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183921.407738, 'rx': '980448540705', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183921.407738, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183921.407738, 'rx': '45368485', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183921.407738, 'rx': '39846408', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40566', 'ksmPages': 100, 'elapsedTime': '1728669.50', 'cpuLoad': '8.25', 'cpuSys': '2.56', 'diskStats': {'/var/log': {'free': '47798'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47798'}}, 'cpuUserVdsmd': '3.58', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150383, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:32:01 GMT', 'cpuUser': '30.46', 'memFree': 150127, 'cpuIdle': '66.97', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.13'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:32:01,431+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:32:02,493+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=2cb0f94c-694c-4920-9d45-ae50ee6b3742 (api:48) 2019-04-13 21:32:02,493+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000758077', 'lastCheck': '8.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000722786', 'lastCheck': '8.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000680119', 'lastCheck': '8.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000418863', 'lastCheck': '4.7', 'valid': True}} from=internal, task_id=2cb0f94c-694c-4920-9d45-ae50ee6b3742 (api:54) 2019-04-13 21:32:02,832+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=2fc349ac-7be7-40f9-8e90-c7d848433d21 (api:48) 2019-04-13 21:32:02,836+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=2fc349ac-7be7-40f9-8e90-c7d848433d21 (api:54) 2019-04-13 21:32:02,837+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:32:02,857+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=780f86c6-fc59-4cec-9cdb-74b9fc410bd3 (api:48) 2019-04-13 21:32:02,868+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706981449728', 'isoprefix': '', 'alerts': [], 'disktotal': '3898079182848', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706981449728', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719730036736', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706981449728', 'isoprefix': '', 'alerts': [], 'disktotal': '4364370444288', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=780f86c6-fc59-4cec-9cdb-74b9fc410bd3 (api:54) 2019-04-13 21:32:02,870+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:32:04,112+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID=u'ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', options=None) from=internal, task_id=3c8b6203-7114-47df-80ac-9992517128d2 (api:48) 2019-04-13 21:32:04,113+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', options=None) from=internal, task_id=3cb79f08-3dee-4213-91a4-488931a907a4 (api:48) 2019-04-13 21:32:04,114+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', options=None) from=internal, task_id=7cf89dc8-9754-4dfd-9f40-421fb2310fa9 (api:48) 2019-04-13 21:32:04,118+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '5697701376', 'apparentsize': '16106127360'} from=internal, task_id=7cf89dc8-9754-4dfd-9f40-421fb2310fa9 (api:54) 2019-04-13 21:32:04,119+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID='ef6741d1-380a-4736-a924-f88ed68e1270', options=None) from=internal, task_id=a5aa5e89-7380-4dcf-ad56-db7882b62641 (api:48) 2019-04-13 21:32:04,119+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '4579684864', 'apparentsize': '16106127360'} from=internal, task_id=3cb79f08-3dee-4213-91a4-488931a907a4 (api:54) 2019-04-13 21:32:04,120+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID=u'67a4dd71-69ff-49a8-b8a4-c7101c75e105', options=None) from=internal, task_id=82c0a08a-052f-4314-a944-cdc12c1fc97c (api:48) 2019-04-13 21:32:04,122+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '53760849408', 'apparentsize': '139586437120'} from=internal, task_id=3c8b6203-7114-47df-80ac-9992517128d2 (api:54) 2019-04-13 21:32:04,123+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', options=None) from=internal, task_id=23790d9a-408a-4dd0-9aba-0034fea5f413 (api:48) 2019-04-13 21:32:04,129+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID=u'2b136ed7-8b9a-45c1-9edc-64ee96c87c43', options=None) from=internal, task_id=b7a713df-d6ec-41f3-8033-6e9674731770 (api:48) 2019-04-13 21:32:04,133+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '6477462016', 'apparentsize': '21474836480'} from=internal, task_id=a5aa5e89-7380-4dcf-ad56-db7882b62641 (api:54) 2019-04-13 21:32:04,134+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID=u'44b3ceca-f0db-43da-af29-aa681df146ac', options=None) from=internal, task_id=71f25f03-ea72-46fa-9355-04db83636fdb (api:48) 2019-04-13 21:32:04,134+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '55059087872', 'apparentsize': '171798691840'} from=internal, task_id=82c0a08a-052f-4314-a944-cdc12c1fc97c (api:54) 2019-04-13 21:32:04,135+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=e02c67c4-49fc-465f-ae4e-5e0ad28b2610 (api:48) 2019-04-13 21:32:04,137+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2524562432', 'apparentsize': '12884901888'} from=internal, task_id=23790d9a-408a-4dd0-9aba-0034fea5f413 (api:54) 2019-04-13 21:32:04,138+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID='123e2cc8-8f24-4ddd-b401-68393535f555', options=None) from=internal, task_id=2bf1eacb-c268-4fd4-b325-0e511cf27818 (api:48) 2019-04-13 21:32:04,143+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2024405504', 'apparentsize': '10737418240'} from=internal, task_id=b7a713df-d6ec-41f3-8033-6e9674731770 (api:54) 2019-04-13 21:32:04,144+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID='3f6c31c6-ec38-4979-a617-55537123ee61', options=None) from=internal, task_id=0ecbc10d-1340-4f70-bdc9-67b698a1b3d0 (api:48) 2019-04-13 21:32:04,147+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '11196651008', 'apparentsize': '21474836480'} from=internal, task_id=71f25f03-ea72-46fa-9355-04db83636fdb (api:54) 2019-04-13 21:32:04,148+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', options=None) from=internal, task_id=249321a3-a241-45bc-a678-1c0250ed6677 (api:48) 2019-04-13 21:32:04,149+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2116486656', 'apparentsize': '3948150784'} from=internal, task_id=e02c67c4-49fc-465f-ae4e-5e0ad28b2610 (api:54) 2019-04-13 21:32:04,150+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID=u'a1d583c1-e135-4e3f-b6fa-0f29a5261366', options=None) from=internal, task_id=b4591b77-8ddc-41a1-b969-cd28eb072656 (api:48) 2019-04-13 21:32:04,150+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2283003904', 'apparentsize': '16106127360'} from=internal, task_id=2bf1eacb-c268-4fd4-b325-0e511cf27818 (api:54) 2019-04-13 21:32:04,151+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', options=None) from=internal, task_id=c6db1565-b175-43ec-9ad6-4c14eda8c851 (api:48) 2019-04-13 21:32:04,158+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '4998711296', 'apparentsize': '16106127360'} from=internal, task_id=0ecbc10d-1340-4f70-bdc9-67b698a1b3d0 (api:54) 2019-04-13 21:32:04,159+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID='a884b4ce-381f-4653-be14-a31d15181f48', options=None) from=internal, task_id=13627229-f072-4781-88f0-0866f8b8e514 (api:48) 2019-04-13 21:32:04,161+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2109410304', 'apparentsize': '8589934592'} from=internal, task_id=249321a3-a241-45bc-a678-1c0250ed6677 (api:54) 2019-04-13 21:32:04,162+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', options=None) from=internal, task_id=e4c2665c-bcc3-430d-a912-692518bd638e (api:48) 2019-04-13 21:32:04,163+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '1294889472', 'apparentsize': '12884901888'} from=internal, task_id=b4591b77-8ddc-41a1-b969-cd28eb072656 (api:54) 2019-04-13 21:32:04,165+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '7263185920', 'apparentsize': '21474836480'} from=internal, task_id=c6db1565-b175-43ec-9ad6-4c14eda8c851 (api:54) 2019-04-13 21:32:04,179+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '3447286784', 'apparentsize': '16106127360'} from=internal, task_id=13627229-f072-4781-88f0-0866f8b8e514 (api:54) 2019-04-13 21:32:04,182+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2879759360', 'apparentsize': '8589934592'} from=internal, task_id=e4c2665c-bcc3-430d-a912-692518bd638e (api:54) 2019-04-13 21:32:04,836+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:32:04,876+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:32:04,877+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:32:04,878+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:32:07,800+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:32:07,848+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:32:07,848+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:32:07,850+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.05 seconds (__init__:312) 2019-04-13 21:32:12,915+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=8e30be27-6386-43e9-a00e-1a64194416e4 (api:48) 2019-04-13 21:32:12,920+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=8e30be27-6386-43e9-a00e-1a64194416e4 (api:54) 2019-04-13 21:32:12,921+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:32:12,932+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=bb1a8378-0379-43a5-b47f-f5c646a6675d (api:48) 2019-04-13 21:32:12,940+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706981449728', 'isoprefix': '', 'alerts': [], 'disktotal': '3898079182848', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706981449728', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719730036736', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706981449728', 'isoprefix': '', 'alerts': [], 'disktotal': '4364370444288', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=bb1a8378-0379-43a5-b47f-f5c646a6675d (api:54) 2019-04-13 21:32:12,941+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:32:16,385+0200 INFO (jsonrpc/6) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:32:16,453+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=e7b610bf-3446-4c6e-b137-2891fd069939 (api:48) 2019-04-13 21:32:16,454+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000532271', 'lastCheck': '2.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000683931', 'lastCheck': '2.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000542913', 'lastCheck': '2.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000388158', 'lastCheck': '2.3', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=e7b610bf-3446-4c6e-b137-2891fd069939 (api:54) 2019-04-13 21:32:16,455+0200 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=44124b0c-9fd4-4a68-91de-578bfe803524 (api:48) 2019-04-13 21:32:16,455+0200 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=44124b0c-9fd4-4a68-91de-578bfe803524 (api:54) 2019-04-13 21:32:16,469+0200 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '20.94', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '76.61'}, '21': {'cpuUser': '24.85', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '72.83'}, '22': {'cpuUser': '29.95', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '67.93'}, '23': {'cpuUser': '21.54', 'nodeIndex': 1, 'cpuSys': '2.92', 'cpuIdle': '75.54'}, '1': {'cpuUser': '50.83', 'nodeIndex': 0, 'cpuSys': '4.17', 'cpuIdle': '45.00'}, '0': {'cpuUser': '60.97', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '37.44'}, '3': {'cpuUser': '57.79', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '40.29'}, '2': {'cpuUser': '52.95', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '45.33'}, '5': {'cpuUser': '56.86', 'nodeIndex': 0, 'cpuSys': '1.26', 'cpuIdle': '41.88'}, '4': {'cpuUser': '59.91', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '38.37'}, '7': {'cpuUser': '40.95', 'nodeIndex': 1, 'cpuSys': '3.25', 'cpuIdle': '55.80'}, '6': {'cpuUser': '35.12', 'nodeIndex': 1, 'cpuSys': '3.45', 'cpuIdle': '61.43'}, '9': {'cpuUser': '33.53', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '63.16'}, '8': {'cpuUser': '41.68', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '55.47'}, '11': {'cpuUser': '33.53', 'nodeIndex': 1, 'cpuSys': '3.71', 'cpuIdle': '62.76'}, '10': {'cpuUser': '32.87', 'nodeIndex': 1, 'cpuSys': '3.58', 'cpuIdle': '63.55'}, '13': {'cpuUser': '24.19', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '73.36'}, '12': {'cpuUser': '24.25', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '74.16'}, '15': {'cpuUser': '17.30', 'nodeIndex': 0, 'cpuSys': '2.92', 'cpuIdle': '79.78'}, '14': {'cpuUser': '33.60', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '64.48'}, '17': {'cpuUser': '33.27', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '64.87'}, '16': {'cpuUser': '19.35', 'nodeIndex': 0, 'cpuSys': '2.65', 'cpuIdle': '78.00'}, '19': {'cpuUser': '20.08', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '77.27'}, '18': {'cpuUser': '25.84', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '71.71'}}, 'numaNodeMemFree': {'1': {'memPercent': 29, 'memFree': '70005'}, '0': {'memPercent': 21, 'memFree': '77721'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000532271', 'lastCheck': '2.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000542913', 'lastCheck': '2.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000388158', 'lastCheck': '2.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000683931', 'lastCheck': '2.1', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168863002', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '5091492678', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590469621146', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '2705646238502', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433959995244', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '259124806395', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239877482', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '366096848', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122802366', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121323863', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '857093', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423806637754', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '253705764859', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215281452', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680508139', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '29547813541', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653295988', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '29289496387', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255894994645', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '7169675407', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121698206', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75384355493', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '61990680307', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122257136', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418945037', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '1012776767', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121509673', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '1041488', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183936.452584, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183936.452584, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183936.452584, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183936.452584, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121279824', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458763044707', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183936.452584, 'rx': '2577003608779', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322606749696', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183936.452584, 'rx': '1322606749696', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183936.452584, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940866737', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '2700695303156', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571528754409', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '4950935346', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329933765555', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183936.452584, 'rx': '980458507769', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183936.452584, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183936.452584, 'rx': '45368875', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183936.452584, 'rx': '39846776', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40566', 'ksmPages': 100, 'elapsedTime': '1728684.55', 'cpuLoad': '8.31', 'cpuSys': '2.54', 'diskStats': {'/var/log': {'free': '47798'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47798'}}, 'cpuUserVdsmd': '2.78', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150380, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:32:16 GMT', 'cpuUser': '35.51', 'memFree': 150124, 'cpuIdle': '61.95', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.39'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:32:16,475+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:32:17,571+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=612f64eb-751b-4507-8ec6-65a4b439c492 (api:48) 2019-04-13 21:32:17,571+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000532271', 'lastCheck': '3.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000683931', 'lastCheck': '3.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000542913', 'lastCheck': '3.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000388158', 'lastCheck': '3.4', 'valid': True}} from=internal, task_id=612f64eb-751b-4507-8ec6-65a4b439c492 (api:54) 2019-04-13 21:32:19,916+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:32:19,954+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:32:19,955+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:32:19,955+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:32:22,858+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:32:22,888+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:32:22,889+0200 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:32:22,890+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:32:22,989+0200 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=7c6930fe-83ee-4bd5-8491-88055c3cc749 (api:48) 2019-04-13 21:32:22,995+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=7c6930fe-83ee-4bd5-8491-88055c3cc749 (api:54) 2019-04-13 21:32:22,995+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:32:23,013+0200 INFO (jsonrpc/2) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=5321b33a-c8fd-4f2e-9139-f7f147167f26 (api:48) 2019-04-13 21:32:23,020+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706981449728', 'isoprefix': '', 'alerts': [], 'disktotal': '3898079182848', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706981449728', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719730036736', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706981449728', 'isoprefix': '', 'alerts': [], 'disktotal': '4364370444288', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=5321b33a-c8fd-4f2e-9139-f7f147167f26 (api:54) 2019-04-13 21:32:23,021+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:32:24,316+0200 INFO (monitor/93f02ed) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_isos) Closing client (__init__:584) 2019-04-13 21:32:24,316+0200 INFO (monitor/93f02ed) [IOProcessClient] (/glusterSD/onode-2.discovery.intern:vol-node-2) Closing client (__init__:584) 2019-04-13 21:32:24,317+0200 INFO (monitor/93f02ed) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_vms) Closing client (__init__:584) 2019-04-13 21:32:24,317+0200 INFO (monitor/93f02ed) [IOProcessClient] (/onode-1.discovery.intern:_mnt_store) Closing client (__init__:584) 2019-04-13 21:32:24,317+0200 INFO (monitor/93f02ed) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Closing client (__init__:584) 2019-04-13 21:32:31,436+0200 INFO (jsonrpc/1) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:32:31,508+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=8cd12621-1b1e-4ca0-a17a-af5728281544 (api:48) 2019-04-13 21:32:31,510+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000651267', 'lastCheck': '7.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000772895', 'lastCheck': '7.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000437479', 'lastCheck': '7.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00034035', 'lastCheck': '3.7', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=8cd12621-1b1e-4ca0-a17a-af5728281544 (api:54) 2019-04-13 21:32:31,511+0200 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=95499c1a-454b-49df-95ea-04db8bdaff5d (api:48) 2019-04-13 21:32:31,511+0200 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=95499c1a-454b-49df-95ea-04db8bdaff5d (api:54) 2019-04-13 21:32:31,530+0200 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '23.65', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '74.03'}, '21': {'cpuUser': '20.40', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '77.02'}, '22': {'cpuUser': '19.94', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '77.81'}, '23': {'cpuUser': '15.10', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '82.12'}, '1': {'cpuUser': '54.85', 'nodeIndex': 0, 'cpuSys': '3.91', 'cpuIdle': '41.24'}, '0': {'cpuUser': '51.60', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '46.61'}, '3': {'cpuUser': '47.36', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '50.45'}, '2': {'cpuUser': '47.36', 'nodeIndex': 0, 'cpuSys': '2.65', 'cpuIdle': '49.99'}, '5': {'cpuUser': '45.31', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '52.57'}, '4': {'cpuUser': '49.95', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '48.33'}, '7': {'cpuUser': '35.11', 'nodeIndex': 1, 'cpuSys': '3.58', 'cpuIdle': '61.31'}, '6': {'cpuUser': '34.91', 'nodeIndex': 1, 'cpuSys': '3.71', 'cpuIdle': '61.38'}, '9': {'cpuUser': '39.88', 'nodeIndex': 1, 'cpuSys': '3.25', 'cpuIdle': '56.87'}, '8': {'cpuUser': '35.77', 'nodeIndex': 1, 'cpuSys': '3.38', 'cpuIdle': '60.85'}, '11': {'cpuUser': '35.24', 'nodeIndex': 1, 'cpuSys': '3.38', 'cpuIdle': '61.38'}, '10': {'cpuUser': '37.36', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '59.59'}, '13': {'cpuUser': '27.56', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '70.59'}, '12': {'cpuUser': '30.74', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '67.21'}, '15': {'cpuUser': '40.48', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '57.53'}, '14': {'cpuUser': '24.58', 'nodeIndex': 0, 'cpuSys': '2.78', 'cpuIdle': '72.64'}, '17': {'cpuUser': '41.67', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '56.41'}, '16': {'cpuUser': '29.55', 'nodeIndex': 0, 'cpuSys': '2.52', 'cpuIdle': '67.93'}, '19': {'cpuUser': '20.07', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '77.35'}, '18': {'cpuUser': '17.16', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '80.32'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69703'}, '0': {'memPercent': 21, 'memFree': '78018'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000651267', 'lastCheck': '7.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000437479', 'lastCheck': '7.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00034035', 'lastCheck': '3.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000772895', 'lastCheck': '7.2', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168863166', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '5091492678', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590470220552', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '2705647375848', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433964319658', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '259127435761', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239877646', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '366096848', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122802530', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121324027', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '857093', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423810664280', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '253708196979', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215281616', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680508303', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '29547813541', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653295988', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '29289496551', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255897630069', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '7171556284', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121698370', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75386421127', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '61999370245', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122257300', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '24423330', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418945565', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '1012776941', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121509957', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '1041572', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183951.506379, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183951.506379, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183951.506379, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183951.506379, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121279988', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458763562487', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183951.506379, 'rx': '2577004695063', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322617116839', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183951.506379, 'rx': '1322617116839', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183951.506379, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940866823', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '2700696440284', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571529353729', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '4950935564', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329947274203', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183951.506379, 'rx': '980469169906', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183951.506379, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183951.506379, 'rx': '45369265', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183951.506379, 'rx': '39847098', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40570', 'ksmPages': 100, 'elapsedTime': '1728699.60', 'cpuLoad': '8.36', 'cpuSys': '2.61', 'diskStats': {'/var/log': {'free': '47798'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47798'}}, 'cpuUserVdsmd': '3.84', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150376, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:32:31 GMT', 'cpuUser': '34.40', 'memFree': 150120, 'cpuIdle': '62.99', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.32'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:32:31,537+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.10 seconds (__init__:312) 2019-04-13 21:32:32,687+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=5784b121-c283-4534-b5cb-d2d4951fd6fb (api:48) 2019-04-13 21:32:32,688+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000651267', 'lastCheck': '8.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000772895', 'lastCheck': '8.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000437479', 'lastCheck': '8.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00034035', 'lastCheck': '4.8', 'valid': True}} from=internal, task_id=5784b121-c283-4534-b5cb-d2d4951fd6fb (api:54) 2019-04-13 21:32:33,067+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=6356f44b-fd81-46b1-a53b-f7694f05bcb3 (api:48) 2019-04-13 21:32:33,071+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=6356f44b-fd81-46b1-a53b-f7694f05bcb3 (api:54) 2019-04-13 21:32:33,072+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:32:33,085+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=e00ec27e-7e89-4c42-980f-79251efb4b8f (api:48) 2019-04-13 21:32:33,094+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706981449728', 'isoprefix': '', 'alerts': [], 'disktotal': '3898079182848', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706981449728', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719730036736', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706981449728', 'isoprefix': '', 'alerts': [], 'disktotal': '4364370444288', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=e00ec27e-7e89-4c42-980f-79251efb4b8f (api:54) 2019-04-13 21:32:33,096+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:32:34,995+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:32:35,016+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:32:35,017+0200 ERROR (jsonrpc/6) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:32:35,017+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.02 seconds (__init__:312) 2019-04-13 21:32:37,896+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:32:37,937+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:32:37,938+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:32:37,939+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:32:43,150+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=dbd44276-f3b3-460e-bd48-be845b15a815 (api:48) 2019-04-13 21:32:43,155+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=dbd44276-f3b3-460e-bd48-be845b15a815 (api:54) 2019-04-13 21:32:43,155+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:32:43,270+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=8501c69d-0234-41b9-910c-cc3eedef9872 (api:48) 2019-04-13 21:32:43,278+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706973061120', 'isoprefix': '', 'alerts': [], 'disktotal': '3898070794240', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706973061120', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719721648128', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706973061120', 'isoprefix': '', 'alerts': [], 'disktotal': '4364362055680', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=8501c69d-0234-41b9-910c-cc3eedef9872 (api:54) 2019-04-13 21:32:43,280+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:32:46,494+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:32:46,582+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=d93b99f8-e0d4-4704-8579-e87e3907ab33 (api:48) 2019-04-13 21:32:46,582+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000411545', 'lastCheck': '2.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000513023', 'lastCheck': '2.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000515113', 'lastCheck': '2.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000383411', 'lastCheck': '2.4', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=d93b99f8-e0d4-4704-8579-e87e3907ab33 (api:54) 2019-04-13 21:32:46,583+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=49199b1a-a45d-4e7f-be42-805df4550333 (api:48) 2019-04-13 21:32:46,584+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=49199b1a-a45d-4e7f-be42-805df4550333 (api:54) 2019-04-13 21:32:46,599+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '27.24', 'nodeIndex': 1, 'cpuSys': '1.92', 'cpuIdle': '70.84'}, '21': {'cpuUser': '23.40', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '74.15'}, '22': {'cpuUser': '25.45', 'nodeIndex': 1, 'cpuSys': '1.79', 'cpuIdle': '72.76'}, '23': {'cpuUser': '27.31', 'nodeIndex': 1, 'cpuSys': '1.86', 'cpuIdle': '70.83'}, '1': {'cpuUser': '40.50', 'nodeIndex': 0, 'cpuSys': '4.18', 'cpuIdle': '55.32'}, '0': {'cpuUser': '43.55', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '54.46'}, '3': {'cpuUser': '49.78', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '47.90'}, '2': {'cpuUser': '50.45', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '47.50'}, '5': {'cpuUser': '59.26', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '39.08'}, '4': {'cpuUser': '42.76', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '54.79'}, '7': {'cpuUser': '36.46', 'nodeIndex': 1, 'cpuSys': '3.12', 'cpuIdle': '60.42'}, '6': {'cpuUser': '36.46', 'nodeIndex': 1, 'cpuSys': '2.92', 'cpuIdle': '60.62'}, '9': {'cpuUser': '34.20', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '62.95'}, '8': {'cpuUser': '34.93', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '61.89'}, '11': {'cpuUser': '35.13', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '62.02'}, '10': {'cpuUser': '34.40', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '62.55'}, '13': {'cpuUser': '26.71', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '71.10'}, '12': {'cpuUser': '33.41', 'nodeIndex': 0, 'cpuSys': '1.46', 'cpuIdle': '65.13'}, '15': {'cpuUser': '15.98', 'nodeIndex': 0, 'cpuSys': '2.65', 'cpuIdle': '81.37'}, '14': {'cpuUser': '23.40', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '74.68'}, '17': {'cpuUser': '15.31', 'nodeIndex': 0, 'cpuSys': '2.65', 'cpuIdle': '82.04'}, '16': {'cpuUser': '27.31', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '70.37'}, '19': {'cpuUser': '25.79', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '72.02'}, '18': {'cpuUser': '23.47', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '74.01'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69543'}, '0': {'memPercent': 21, 'memFree': '78188'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000411545', 'lastCheck': '2.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000515113', 'lastCheck': '2.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000383411', 'lastCheck': '2.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000513023', 'lastCheck': '2.2', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168863712', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '5091493170', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590470551734', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '2705647403742', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433968148634', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '259130214423', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239877766', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '366096848', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122802650', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121324147', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '857093', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423814947238', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '253710917287', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215281736', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680508423', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '29547813541', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653296108', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '29289496551', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255900424745', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '7173600237', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121698490', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75388807833', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '62003860639', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122257510', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '24423420', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418946165', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '1012777025', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121510077', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '1041572', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183966.580888, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183966.580888, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183966.580888, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183966.580888, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121280108', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458763881539', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183966.580888, 'rx': '2577004719389', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322630624908', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183966.580888, 'rx': '1322630624908', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183966.580888, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940867469', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '2700696467476', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571529684265', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '4950936266', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329956864047', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183966.580888, 'rx': '980480102183', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183966.580888, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183966.580888, 'rx': '45369655', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183966.580888, 'rx': '39847466', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40564', 'ksmPages': 100, 'elapsedTime': '1728714.66', 'cpuLoad': '8.46', 'cpuSys': '2.44', 'diskStats': {'/var/log': {'free': '47798'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47798'}}, 'cpuUserVdsmd': '2.59', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150391, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:32:46 GMT', 'cpuUser': '33.03', 'memFree': 150135, 'cpuIdle': '64.52', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.19'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:32:46,606+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.11 seconds (__init__:312) 2019-04-13 21:32:47,761+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=eba9d0d3-981a-4136-a48f-accdfdd28098 (api:48) 2019-04-13 21:32:47,762+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000411545', 'lastCheck': '3.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000513023', 'lastCheck': '3.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000515113', 'lastCheck': '3.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000383411', 'lastCheck': '3.6', 'valid': True}} from=internal, task_id=eba9d0d3-981a-4136-a48f-accdfdd28098 (api:54) 2019-04-13 21:32:50,044+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:32:50,074+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:32:50,075+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:32:50,076+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:32:52,946+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:32:52,977+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:32:52,978+0200 ERROR (jsonrpc/0) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:32:52,978+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:32:53,333+0200 INFO (jsonrpc/3) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=4f4f2134-18e6-4f05-b0dc-f646b6bc09d8 (api:48) 2019-04-13 21:32:53,338+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=4f4f2134-18e6-4f05-b0dc-f646b6bc09d8 (api:54) 2019-04-13 21:32:53,338+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:32:53,353+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=035de54f-0961-41da-9e47-46f307774092 (api:48) 2019-04-13 21:32:53,360+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706973061120', 'isoprefix': '', 'alerts': [], 'disktotal': '3898070794240', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706973061120', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719721648128', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706973061120', 'isoprefix': '', 'alerts': [], 'disktotal': '4364362055680', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=035de54f-0961-41da-9e47-46f307774092 (api:54) 2019-04-13 21:32:53,361+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:33:01,543+0200 INFO (jsonrpc/4) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:33:01,635+0200 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=64796d61-ac68-4e08-b701-abaca723efde (api:48) 2019-04-13 21:33:01,636+0200 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000725261', 'lastCheck': '7.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000745114', 'lastCheck': '7.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000407577', 'lastCheck': '7.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000404988', 'lastCheck': '3.8', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=64796d61-ac68-4e08-b701-abaca723efde (api:54) 2019-04-13 21:33:01,637+0200 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=18a1fd2a-87dd-42c9-8e38-bdc2938f6ed1 (api:48) 2019-04-13 21:33:01,637+0200 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=18a1fd2a-87dd-42c9-8e38-bdc2938f6ed1 (api:54) 2019-04-13 21:33:01,653+0200 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '32.11', 'nodeIndex': 1, 'cpuSys': '1.52', 'cpuIdle': '66.37'}, '21': {'cpuUser': '23.30', 'nodeIndex': 1, 'cpuSys': '2.18', 'cpuIdle': '74.52'}, '22': {'cpuUser': '23.64', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '73.91'}, '23': {'cpuUser': '25.69', 'nodeIndex': 1, 'cpuSys': '2.18', 'cpuIdle': '72.13'}, '1': {'cpuUser': '39.59', 'nodeIndex': 0, 'cpuSys': '4.77', 'cpuIdle': '55.64'}, '0': {'cpuUser': '40.78', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '56.97'}, '3': {'cpuUser': '45.81', 'nodeIndex': 0, 'cpuSys': '2.52', 'cpuIdle': '51.67'}, '2': {'cpuUser': '40.85', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '57.03'}, '5': {'cpuUser': '37.08', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '60.60'}, '4': {'cpuUser': '41.58', 'nodeIndex': 0, 'cpuSys': '2.18', 'cpuIdle': '56.24'}, '7': {'cpuUser': '42.24', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '55.05'}, '6': {'cpuUser': '42.64', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '54.65'}, '9': {'cpuUser': '38.93', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '58.36'}, '8': {'cpuUser': '40.52', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '56.96'}, '11': {'cpuUser': '36.48', 'nodeIndex': 1, 'cpuSys': '2.91', 'cpuIdle': '60.61'}, '10': {'cpuUser': '38.07', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '59.15'}, '13': {'cpuUser': '16.29', 'nodeIndex': 0, 'cpuSys': '2.18', 'cpuIdle': '81.53'}, '12': {'cpuUser': '23.30', 'nodeIndex': 0, 'cpuSys': '1.59', 'cpuIdle': '75.11'}, '15': {'cpuUser': '14.63', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '83.05'}, '14': {'cpuUser': '22.91', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '75.10'}, '17': {'cpuUser': '26.22', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '71.86'}, '16': {'cpuUser': '24.89', 'nodeIndex': 0, 'cpuSys': '1.85', 'cpuIdle': '73.26'}, '19': {'cpuUser': '16.35', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '80.94'}, '18': {'cpuUser': '15.49', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '81.33'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69480'}, '0': {'memPercent': 21, 'memFree': '78253'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000725261', 'lastCheck': '7.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000407577', 'lastCheck': '7.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000404988', 'lastCheck': '3.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000745114', 'lastCheck': '7.3', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168863914', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '5091493254', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590470972914', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '2705648942814', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433971730616', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '259132501192', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239877848', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '366096848', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122802732', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121324229', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '857093', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423818548186', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '253713561440', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215281818', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680508505', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '29547813541', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653296108', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '29289496633', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255902746345', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '7177018012', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121698572', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75391039815', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '62003881783', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122257712', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '24423504', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418946585', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '1012777025', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121510159', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '1041572', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183981.632872, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183981.632872, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183981.632872, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183981.632872, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121280190', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458764287443', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183981.632872, 'rx': '2577006191765', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322639943011', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183981.632872, 'rx': '1322639943011', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183981.632872, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940867597', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '2700698006202', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571530105317', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '4950936612', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329962684032', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183981.632872, 'rx': '980489918316', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183981.632872, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183981.632872, 'rx': '45370045', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183981.632872, 'rx': '39847788', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1728729.71', 'cpuLoad': '8.35', 'cpuSys': '2.45', 'diskStats': {'/var/log': {'free': '47798'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47798'}}, 'cpuUserVdsmd': '3.05', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150391, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:33:01 GMT', 'cpuUser': '31.23', 'memFree': 150135, 'cpuIdle': '66.32', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.39'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:33:01,660+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.12 seconds (__init__:312) 2019-04-13 21:33:02,842+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=01f4cbed-060f-4d13-ab63-60249aca8ce9 (api:48) 2019-04-13 21:33:02,843+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000725261', 'lastCheck': '8.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000745114', 'lastCheck': '8.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000407577', 'lastCheck': '8.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000404988', 'lastCheck': '5.0', 'valid': True}} from=internal, task_id=01f4cbed-060f-4d13-ab63-60249aca8ce9 (api:54) 2019-04-13 21:33:03,412+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=155087b5-3ef0-456d-a07e-af9d0762644c (api:48) 2019-04-13 21:33:03,417+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=155087b5-3ef0-456d-a07e-af9d0762644c (api:54) 2019-04-13 21:33:03,417+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:33:03,434+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=cccaf215-2a60-483c-bf7f-c96bf6ca7154 (api:48) 2019-04-13 21:33:03,441+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706968866816', 'isoprefix': '', 'alerts': [], 'disktotal': '3898065551360', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706968866816', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719717453824', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706968866816', 'isoprefix': '', 'alerts': [], 'disktotal': '4364356812800', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=cccaf215-2a60-483c-bf7f-c96bf6ca7154 (api:54) 2019-04-13 21:33:03,442+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:33:04,110+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID=u'ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', options=None) from=internal, task_id=3a5d4fb6-2bc9-494f-8115-94dee8842438 (api:48) 2019-04-13 21:33:04,111+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', options=None) from=internal, task_id=36a4d21f-1c0e-4e23-a08b-4f55d7e43e99 (api:48) 2019-04-13 21:33:04,113+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', options=None) from=internal, task_id=55a2642a-dcce-4c01-b27d-ebd85f71dbf4 (api:48) 2019-04-13 21:33:04,118+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '4579679744', 'apparentsize': '16106127360'} from=internal, task_id=36a4d21f-1c0e-4e23-a08b-4f55d7e43e99 (api:54) 2019-04-13 21:33:04,119+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID='ef6741d1-380a-4736-a924-f88ed68e1270', options=None) from=internal, task_id=1562d117-1102-4e8a-979d-f07e42299833 (api:48) 2019-04-13 21:33:04,120+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '53760849408', 'apparentsize': '139586437120'} from=internal, task_id=3a5d4fb6-2bc9-494f-8115-94dee8842438 (api:54) 2019-04-13 21:33:04,121+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID=u'67a4dd71-69ff-49a8-b8a4-c7101c75e105', options=None) from=internal, task_id=3129cb20-0751-49eb-8b81-29a4c1ac8174 (api:48) 2019-04-13 21:33:04,124+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '5697701376', 'apparentsize': '16106127360'} from=internal, task_id=55a2642a-dcce-4c01-b27d-ebd85f71dbf4 (api:54) 2019-04-13 21:33:04,125+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', options=None) from=internal, task_id=190926d9-11d9-491b-83ac-a0f8a6132792 (api:48) 2019-04-13 21:33:04,130+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID=u'2b136ed7-8b9a-45c1-9edc-64ee96c87c43', options=None) from=internal, task_id=35b62047-2a72-4685-8ab6-98ce32cd6764 (api:48) 2019-04-13 21:33:04,132+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '6477456384', 'apparentsize': '21474836480'} from=internal, task_id=1562d117-1102-4e8a-979d-f07e42299833 (api:54) 2019-04-13 21:33:04,133+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID=u'44b3ceca-f0db-43da-af29-aa681df146ac', options=None) from=internal, task_id=41076778-7dfd-466a-92a3-7365683b18f8 (api:48) 2019-04-13 21:33:04,134+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '55059087872', 'apparentsize': '171798691840'} from=internal, task_id=3129cb20-0751-49eb-8b81-29a4c1ac8174 (api:54) 2019-04-13 21:33:04,135+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=d248fa28-ef43-401b-b9bd-4727bfabc344 (api:48) 2019-04-13 21:33:04,138+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2524562432', 'apparentsize': '12884901888'} from=internal, task_id=190926d9-11d9-491b-83ac-a0f8a6132792 (api:54) 2019-04-13 21:33:04,139+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID='123e2cc8-8f24-4ddd-b401-68393535f555', options=None) from=internal, task_id=62edd2f1-0f1b-44ea-a561-94175d03c391 (api:48) 2019-04-13 21:33:04,142+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2024405504', 'apparentsize': '10737418240'} from=internal, task_id=35b62047-2a72-4685-8ab6-98ce32cd6764 (api:54) 2019-04-13 21:33:04,143+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID='3f6c31c6-ec38-4979-a617-55537123ee61', options=None) from=internal, task_id=3b24bb3a-28c6-4d1f-b934-a50fee5d76a3 (api:48) 2019-04-13 21:33:04,146+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '11196651008', 'apparentsize': '21474836480'} from=internal, task_id=41076778-7dfd-466a-92a3-7365683b18f8 (api:54) 2019-04-13 21:33:04,147+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', options=None) from=internal, task_id=ec4e5356-3c7b-4970-8585-91cff920e868 (api:48) 2019-04-13 21:33:04,148+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2116486656', 'apparentsize': '3948150784'} from=internal, task_id=d248fa28-ef43-401b-b9bd-4727bfabc344 (api:54) 2019-04-13 21:33:04,149+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID=u'a1d583c1-e135-4e3f-b6fa-0f29a5261366', options=None) from=internal, task_id=36d0154a-0a93-476e-89df-d8994172ad06 (api:48) 2019-04-13 21:33:04,152+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2282998272', 'apparentsize': '16106127360'} from=internal, task_id=62edd2f1-0f1b-44ea-a561-94175d03c391 (api:54) 2019-04-13 21:33:04,153+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', options=None) from=internal, task_id=74fcbd23-1136-4e8f-8cc2-f3f3bbb2bcee (api:48) 2019-04-13 21:33:04,157+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '2109410304', 'apparentsize': '8589934592'} from=internal, task_id=ec4e5356-3c7b-4970-8585-91cff920e868 (api:54) 2019-04-13 21:33:04,158+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID='a884b4ce-381f-4653-be14-a31d15181f48', options=None) from=internal, task_id=f75e4caa-9201-4914-a56e-aeaf978a0cb3 (api:48) 2019-04-13 21:33:04,159+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '4998711296', 'apparentsize': '16106127360'} from=internal, task_id=3b24bb3a-28c6-4d1f-b934-a50fee5d76a3 (api:54) 2019-04-13 21:33:04,160+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', options=None) from=internal, task_id=8ba118fa-80ba-4518-85f4-70b95b0b8604 (api:48) 2019-04-13 21:33:04,161+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '1294889472', 'apparentsize': '12884901888'} from=internal, task_id=36d0154a-0a93-476e-89df-d8994172ad06 (api:54) 2019-04-13 21:33:04,164+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '7263185920', 'apparentsize': '21474836480'} from=internal, task_id=74fcbd23-1136-4e8f-8cc2-f3f3bbb2bcee (api:54) 2019-04-13 21:33:04,178+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2879770112', 'apparentsize': '8589934592'} from=internal, task_id=8ba118fa-80ba-4518-85f4-70b95b0b8604 (api:54) 2019-04-13 21:33:04,180+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '3447286784', 'apparentsize': '16106127360'} from=internal, task_id=f75e4caa-9201-4914-a56e-aeaf978a0cb3 (api:54) 2019-04-13 21:33:05,119+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:33:05,161+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:33:05,161+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:33:05,163+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:33:07,986+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:33:08,024+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:33:08,025+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:33:08,025+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:33:13,488+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=2aff69ca-40d4-4302-ad41-daddeaa080b0 (api:48) 2019-04-13 21:33:13,493+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=2aff69ca-40d4-4302-ad41-daddeaa080b0 (api:54) 2019-04-13 21:33:13,493+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:33:13,506+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=b9389c09-694e-45a3-9b00-689633247a9b (api:48) 2019-04-13 21:33:13,515+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706968866816', 'isoprefix': '', 'alerts': [], 'disktotal': '3898065551360', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706968866816', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719717453824', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706968866816', 'isoprefix': '', 'alerts': [], 'disktotal': '4364356812800', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=b9389c09-694e-45a3-9b00-689633247a9b (api:54) 2019-04-13 21:33:13,516+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:33:16,591+0200 INFO (jsonrpc/6) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:33:16,652+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=be4f4531-1a0d-4a13-97f3-479d340f0532 (api:48) 2019-04-13 21:33:16,653+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000525091', 'lastCheck': '2.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000768923', 'lastCheck': '2.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000438175', 'lastCheck': '2.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000319779', 'lastCheck': '2.5', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=be4f4531-1a0d-4a13-97f3-479d340f0532 (api:54) 2019-04-13 21:33:16,653+0200 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=42784c0b-64a1-4adf-b09a-e9ef7b2cf379 (api:48) 2019-04-13 21:33:16,654+0200 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=42784c0b-64a1-4adf-b09a-e9ef7b2cf379 (api:54) 2019-04-13 21:33:16,667+0200 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '19.43', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '78.38'}, '21': {'cpuUser': '20.82', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '76.53'}, '22': {'cpuUser': '24.60', 'nodeIndex': 1, 'cpuSys': '1.99', 'cpuIdle': '73.41'}, '23': {'cpuUser': '23.54', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '74.34'}, '1': {'cpuUser': '38.19', 'nodeIndex': 0, 'cpuSys': '3.91', 'cpuIdle': '57.90'}, '0': {'cpuUser': '44.02', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '53.73'}, '3': {'cpuUser': '51.12', 'nodeIndex': 0, 'cpuSys': '2.06', 'cpuIdle': '46.82'}, '2': {'cpuUser': '53.37', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '44.77'}, '5': {'cpuUser': '46.74', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '51.14'}, '4': {'cpuUser': '49.13', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '49.08'}, '7': {'cpuUser': '36.99', 'nodeIndex': 1, 'cpuSys': '3.38', 'cpuIdle': '59.63'}, '6': {'cpuUser': '35.67', 'nodeIndex': 1, 'cpuSys': '2.92', 'cpuIdle': '61.41'}, '9': {'cpuUser': '35.93', 'nodeIndex': 1, 'cpuSys': '3.25', 'cpuIdle': '60.82'}, '8': {'cpuUser': '34.21', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '62.48'}, '11': {'cpuUser': '34.41', 'nodeIndex': 1, 'cpuSys': '3.38', 'cpuIdle': '62.21'}, '10': {'cpuUser': '34.81', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '62.21'}, '13': {'cpuUser': '38.78', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '59.30'}, '12': {'cpuUser': '37.26', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '60.88'}, '15': {'cpuUser': '16.91', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '80.97'}, '14': {'cpuUser': '26.72', 'nodeIndex': 0, 'cpuSys': '2.06', 'cpuIdle': '71.22'}, '17': {'cpuUser': '26.59', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '71.62'}, '16': {'cpuUser': '25.26', 'nodeIndex': 0, 'cpuSys': '1.52', 'cpuIdle': '73.22'}, '19': {'cpuUser': '15.65', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '81.50'}, '18': {'cpuUser': '21.41', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '76.47'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '69062'}, '0': {'memPercent': 20, 'memFree': '78671'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000525091', 'lastCheck': '2.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000438175', 'lastCheck': '2.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000319779', 'lastCheck': '2.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000768923', 'lastCheck': '2.3', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168864106', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '5091493254', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590471291352', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '2705650071196', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433975497104', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '259134611072', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239878376', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '366097246', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122802924', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121324421', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '857093', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423822018266', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '253715856902', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215282010', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680508697', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '29547813541', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653296218', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '29289496715', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255904909043', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '7179668046', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121698764', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75393887943', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '62011527904', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122257904', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '24423504', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418947065', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '1012777025', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121510351', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '1041572', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183996.651071, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183996.651071, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183996.651071, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183996.651071, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121280382', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458764610999', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183996.651071, 'rx': '2577007269867', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322653446362', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183996.651071, 'rx': '1322653446362', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183996.651071, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940867811', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '2700699133990', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571530423541', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '4950937206', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329976190182', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555183996.651071, 'rx': '980500462173', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183996.651071, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183996.651071, 'rx': '45370435', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555183996.651071, 'rx': '39848156', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1728744.75', 'cpuLoad': '8.27', 'cpuSys': '2.43', 'diskStats': {'/var/log': {'free': '47798'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47798'}}, 'cpuUserVdsmd': '3.38', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150389, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:33:16 GMT', 'cpuUser': '32.97', 'memFree': 150133, 'cpuIdle': '64.60', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.19'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:33:16,673+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.09 seconds (__init__:312) 2019-04-13 21:33:17,940+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=9e6f9633-c549-42fd-aa68-38ac93d0deb4 (api:48) 2019-04-13 21:33:17,941+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000525091', 'lastCheck': '3.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000768923', 'lastCheck': '3.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000438175', 'lastCheck': '3.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000319779', 'lastCheck': '0.0', 'valid': True}} from=internal, task_id=9e6f9633-c549-42fd-aa68-38ac93d0deb4 (api:54) 2019-04-13 21:33:20,199+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:33:20,227+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:33:20,227+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:33:20,228+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:33:23,032+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:33:23,067+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:33:23,068+0200 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:33:23,069+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:33:23,563+0200 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=2e8a559c-0adc-4d54-929e-0e592f8d0d3f (api:48) 2019-04-13 21:33:23,568+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=2e8a559c-0adc-4d54-929e-0e592f8d0d3f (api:54) 2019-04-13 21:33:23,568+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:33:23,581+0200 INFO (jsonrpc/2) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=98d4baeb-fc1f-4a2c-b394-89b23ba5358f (api:48) 2019-04-13 21:33:23,590+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706961526784', 'isoprefix': '', 'alerts': [], 'disktotal': '3898058211328', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706961526784', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719709065216', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706961526784', 'isoprefix': '', 'alerts': [], 'disktotal': '4364349472768', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=98d4baeb-fc1f-4a2c-b394-89b23ba5358f (api:54) 2019-04-13 21:33:23,592+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:33:31,636+0200 INFO (jsonrpc/1) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:33:31,697+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=78774121-7e73-43a1-8cf9-d00d6f346962 (api:48) 2019-04-13 21:33:31,698+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00061937', 'lastCheck': '7.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000806325', 'lastCheck': '7.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000469262', 'lastCheck': '7.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000372068', 'lastCheck': '3.8', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=78774121-7e73-43a1-8cf9-d00d6f346962 (api:54) 2019-04-13 21:33:31,698+0200 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=607018e1-655d-4077-b0e5-e5d204823805 (api:48) 2019-04-13 21:33:31,699+0200 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=607018e1-655d-4077-b0e5-e5d204823805 (api:54) 2019-04-13 21:33:31,714+0200 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '17.37', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '79.91'}, '21': {'cpuUser': '20.75', 'nodeIndex': 1, 'cpuSys': '2.19', 'cpuIdle': '77.06'}, '22': {'cpuUser': '25.99', 'nodeIndex': 1, 'cpuSys': '1.46', 'cpuIdle': '72.55'}, '23': {'cpuUser': '15.38', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '82.17'}, '1': {'cpuUser': '42.42', 'nodeIndex': 0, 'cpuSys': '4.11', 'cpuIdle': '53.47'}, '0': {'cpuUser': '42.96', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '55.25'}, '3': {'cpuUser': '62.64', 'nodeIndex': 0, 'cpuSys': '1.46', 'cpuIdle': '35.90'}, '2': {'cpuUser': '41.63', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '56.38'}, '5': {'cpuUser': '55.62', 'nodeIndex': 0, 'cpuSys': '1.33', 'cpuIdle': '43.05'}, '4': {'cpuUser': '60.32', 'nodeIndex': 0, 'cpuSys': '1.46', 'cpuIdle': '38.22'}, '7': {'cpuUser': '33.67', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '63.35'}, '6': {'cpuUser': '30.89', 'nodeIndex': 1, 'cpuSys': '3.25', 'cpuIdle': '65.86'}, '9': {'cpuUser': '33.08', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '64.07'}, '8': {'cpuUser': '31.82', 'nodeIndex': 1, 'cpuSys': '3.12', 'cpuIdle': '65.06'}, '11': {'cpuUser': '34.21', 'nodeIndex': 1, 'cpuSys': '3.12', 'cpuIdle': '62.67'}, '10': {'cpuUser': '30.96', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '65.73'}, '13': {'cpuUser': '26.91', 'nodeIndex': 0, 'cpuSys': '2.59', 'cpuIdle': '70.50'}, '12': {'cpuUser': '33.61', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '64.34'}, '15': {'cpuUser': '13.39', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '84.29'}, '14': {'cpuUser': '39.44', 'nodeIndex': 0, 'cpuSys': '1.39', 'cpuIdle': '59.17'}, '17': {'cpuUser': '26.52', 'nodeIndex': 0, 'cpuSys': '2.25', 'cpuIdle': '71.23'}, '16': {'cpuUser': '17.43', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '80.45'}, '19': {'cpuUser': '19.49', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '78.26'}, '18': {'cpuUser': '17.70', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '79.78'}}, 'numaNodeMemFree': {'1': {'memPercent': 30, 'memFree': '68835'}, '0': {'memPercent': 20, 'memFree': '78896'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00061937', 'lastCheck': '7.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000469262', 'lastCheck': '7.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000372068', 'lastCheck': '3.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000806325', 'lastCheck': '7.3', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168864770', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '5091493878', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590471555554', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '2705651197986', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433979450476', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '259137306353', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239878500', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '366097246', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122803048', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121324545', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '857093', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423826106850', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '253718042092', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215282134', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680508821', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '29547813541', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653296342', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '29289496715', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255907623192', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '7181428371', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121698888', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75396622749', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '62012604046', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122258028', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '24423504', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418947593', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '1012777199', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121510475', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '1041572', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184011.696042, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184011.696042, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184011.696042, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184011.696042, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121280506', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458764863893', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184011.696042, 'rx': '2577008388825', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322663819745', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184011.696042, 'rx': '1322663819745', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184011.696042, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940868451', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '2700700260104', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571530687103', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '4950937882', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329981261521', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184011.696042, 'rx': '980511690728', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184011.696042, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184011.696042, 'rx': '45370825', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184011.696042, 'rx': '39848478', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1728759.80', 'cpuLoad': '8.26', 'cpuSys': '2.38', 'diskStats': {'/var/log': {'free': '47798'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47798'}}, 'cpuUserVdsmd': '3.25', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150391, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:33:31 GMT', 'cpuUser': '32.26', 'memFree': 150135, 'cpuIdle': '65.36', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.19'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:33:31,720+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.08 seconds (__init__:312) 2019-04-13 21:33:33,040+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=c2a96aaf-8f1e-4a9c-995f-bed55d211cee (api:48) 2019-04-13 21:33:33,041+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00061937', 'lastCheck': '8.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000806325', 'lastCheck': '8.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000469262', 'lastCheck': '8.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000372068', 'lastCheck': '5.1', 'valid': True}} from=internal, task_id=c2a96aaf-8f1e-4a9c-995f-bed55d211cee (api:54) 2019-04-13 21:33:33,640+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=ad2fdcf3-2907-427e-a461-e2cca951f94d (api:48) 2019-04-13 21:33:33,645+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=ad2fdcf3-2907-427e-a461-e2cca951f94d (api:54) 2019-04-13 21:33:33,645+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:33:33,661+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=a3f7a392-edb3-45de-ab96-79a71c748eb9 (api:48) 2019-04-13 21:33:33,670+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706960478208', 'isoprefix': '', 'alerts': [], 'disktotal': '3898057162752', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706960478208', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719708016640', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706960478208', 'isoprefix': '', 'alerts': [], 'disktotal': '4364348424192', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=a3f7a392-edb3-45de-ab96-79a71c748eb9 (api:54) 2019-04-13 21:33:33,672+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:33:35,265+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:33:35,296+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:33:35,296+0200 ERROR (jsonrpc/6) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:33:35,297+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:33:38,076+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:33:38,106+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:33:38,107+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:33:38,107+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:33:43,718+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=459f342d-e49d-461d-bb43-20df925b765f (api:48) 2019-04-13 21:33:43,724+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=459f342d-e49d-461d-bb43-20df925b765f (api:54) 2019-04-13 21:33:43,724+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:33:43,737+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=e6b6574d-f7e2-4eaf-9e2a-bb62f68fdd1d (api:48) 2019-04-13 21:33:43,744+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706960478208', 'isoprefix': '', 'alerts': [], 'disktotal': '3898057162752', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706960478208', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719708016640', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706960478208', 'isoprefix': '', 'alerts': [], 'disktotal': '4364348424192', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=e6b6574d-f7e2-4eaf-9e2a-bb62f68fdd1d (api:54) 2019-04-13 21:33:43,745+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:33:46,713+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:33:46,786+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=b1e1f7de-8ceb-4f8b-9604-53129d830da9 (api:48) 2019-04-13 21:33:46,787+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000632285', 'lastCheck': '2.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000685997', 'lastCheck': '2.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000642624', 'lastCheck': '2.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000378187', 'lastCheck': '2.6', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=b1e1f7de-8ceb-4f8b-9604-53129d830da9 (api:54) 2019-04-13 21:33:46,788+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=9406f2df-d83b-40de-8d52-bb93db64d979 (api:48) 2019-04-13 21:33:46,788+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=9406f2df-d83b-40de-8d52-bb93db64d979 (api:54) 2019-04-13 21:33:46,803+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '23.72', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '74.16'}, '21': {'cpuUser': '21.54', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '76.01'}, '22': {'cpuUser': '21.34', 'nodeIndex': 1, 'cpuSys': '1.86', 'cpuIdle': '76.80'}, '23': {'cpuUser': '17.30', 'nodeIndex': 1, 'cpuSys': '2.39', 'cpuIdle': '80.31'}, '1': {'cpuUser': '45.92', 'nodeIndex': 0, 'cpuSys': '3.64', 'cpuIdle': '50.44'}, '0': {'cpuUser': '59.84', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '38.44'}, '3': {'cpuUser': '53.01', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '45.00'}, '2': {'cpuUser': '49.04', 'nodeIndex': 0, 'cpuSys': '2.45', 'cpuIdle': '48.51'}, '5': {'cpuUser': '48.90', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '49.24'}, '4': {'cpuUser': '49.23', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '48.72'}, '7': {'cpuUser': '30.02', 'nodeIndex': 1, 'cpuSys': '3.38', 'cpuIdle': '66.60'}, '6': {'cpuUser': '36.51', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '60.84'}, '9': {'cpuUser': '34.26', 'nodeIndex': 1, 'cpuSys': '2.92', 'cpuIdle': '62.82'}, '8': {'cpuUser': '31.28', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '65.74'}, '11': {'cpuUser': '34.66', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '62.16'}, '10': {'cpuUser': '34.79', 'nodeIndex': 1, 'cpuSys': '2.72', 'cpuIdle': '62.49'}, '13': {'cpuUser': '35.25', 'nodeIndex': 0, 'cpuSys': '2.39', 'cpuIdle': '62.36'}, '12': {'cpuUser': '21.54', 'nodeIndex': 0, 'cpuSys': '2.32', 'cpuIdle': '76.14'}, '15': {'cpuUser': '19.02', 'nodeIndex': 0, 'cpuSys': '2.58', 'cpuIdle': '78.40'}, '14': {'cpuUser': '26.24', 'nodeIndex': 0, 'cpuSys': '2.19', 'cpuIdle': '71.57'}, '17': {'cpuUser': '27.10', 'nodeIndex': 0, 'cpuSys': '2.39', 'cpuIdle': '70.51'}, '16': {'cpuUser': '35.19', 'nodeIndex': 0, 'cpuSys': '1.66', 'cpuIdle': '63.15'}, '19': {'cpuUser': '24.05', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '73.37'}, '18': {'cpuUser': '24.85', 'nodeIndex': 1, 'cpuSys': '1.66', 'cpuIdle': '73.49'}}, 'numaNodeMemFree': {'1': {'memPercent': 31, 'memFree': '68769'}, '0': {'memPercent': 20, 'memFree': '78964'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000632285', 'lastCheck': '2.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000642624', 'lastCheck': '2.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000378187', 'lastCheck': '2.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000685997', 'lastCheck': '2.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168865411', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '5091493962', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590473718108', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '2705653464695', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433983505702', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '259139882362', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239879021', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '366097246', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122803569', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121325066', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '857093', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423830145446', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '253720886744', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215282655', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680511585', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '29547818392', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653301550', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '29289499122', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255910249822', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '7185071789', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121699409', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75399475281', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '62019723043', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122258549', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '24423504', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418948193', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '1012777283', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121510996', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '1041572', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184026.785028, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184026.785028, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184026.785028, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184026.785028, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121281027', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458766928205', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184026.785028, 'rx': '2577010510588', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322675249806', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184026.785028, 'rx': '1322675249806', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184026.785028, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940868665', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '2700702526211', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571532849443', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '4950938484', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '329994668310', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184026.785028, 'rx': '980522816945', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184026.785028, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184026.785028, 'rx': '45371215', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184026.785028, 'rx': '39848846', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1728774.87', 'cpuLoad': '8.24', 'cpuSys': '2.42', 'diskStats': {'/var/log': {'free': '47798'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47798'}}, 'cpuUserVdsmd': '3.11', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150389, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:33:46 GMT', 'cpuUser': '33.52', 'memFree': 150133, 'cpuIdle': '64.06', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.06'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:33:46,809+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.10 seconds (__init__:312) 2019-04-13 21:33:48,150+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=079fe424-dd4b-42e8-830e-42c043c6571e (api:48) 2019-04-13 21:33:48,151+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000632285', 'lastCheck': '3.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000685997', 'lastCheck': '3.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000642624', 'lastCheck': '3.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000378187', 'lastCheck': '0.2', 'valid': True}} from=internal, task_id=079fe424-dd4b-42e8-830e-42c043c6571e (api:54) 2019-04-13 21:33:50,345+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:33:50,383+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:33:50,383+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:33:50,384+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:33:53,115+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:33:53,158+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:33:53,159+0200 ERROR (jsonrpc/0) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:33:53,160+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.04 seconds (__init__:312) 2019-04-13 21:33:53,789+0200 INFO (jsonrpc/3) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=191706e8-c302-423d-bb11-d5e0006b5276 (api:48) 2019-04-13 21:33:53,794+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=191706e8-c302-423d-bb11-d5e0006b5276 (api:54) 2019-04-13 21:33:53,795+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:33:53,809+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=fb2580f5-abf7-4a6d-ba82-345629ec9c9e (api:48) 2019-04-13 21:33:53,817+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706960478208', 'isoprefix': '', 'alerts': [], 'disktotal': '3898057162752', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706960478208', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719708016640', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706960478208', 'isoprefix': '', 'alerts': [], 'disktotal': '4364348424192', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=fb2580f5-abf7-4a6d-ba82-345629ec9c9e (api:54) 2019-04-13 21:33:53,818+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:33:57,913+0200 INFO (itmap/0) [IOProcessClient] (/onode-1.discovery.intern:_mnt_store) Starting client (__init__:308) 2019-04-13 21:33:57,934+0200 INFO (itmap/1) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_vms) Starting client (__init__:308) 2019-04-13 21:33:57,942+0200 INFO (ioprocess/29988) [IOProcess] (/onode-1.discovery.intern:_mnt_store) Starting ioprocess (__init__:434) 2019-04-13 21:33:57,960+0200 INFO (itmap/2) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_isos) Starting client (__init__:308) 2019-04-13 21:33:57,966+0200 INFO (ioprocess/29996) [IOProcess] (/nas.discovery.intern:_mnt_d2600_vms) Starting ioprocess (__init__:434) 2019-04-13 21:33:57,981+0200 INFO (itmap/3) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting client (__init__:308) 2019-04-13 21:33:58,000+0200 INFO (ioprocess/30004) [IOProcess] (/nas.discovery.intern:_mnt_d2600_isos) Starting ioprocess (__init__:434) 2019-04-13 21:33:58,003+0200 INFO (itmap/4) [IOProcessClient] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting client (__init__:308) 2019-04-13 21:33:58,009+0200 INFO (ioprocess/30007) [IOProcess] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting ioprocess (__init__:434) 2019-04-13 21:33:58,029+0200 INFO (ioprocess/30018) [IOProcess] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting ioprocess (__init__:434) 2019-04-13 21:33:58,037+0200 INFO (monitor/6a4cde7) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:34:01,780+0200 INFO (jsonrpc/4) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:34:01,839+0200 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=0ca8870c-d6f1-45de-b93e-67c69b102863 (api:48) 2019-04-13 21:34:01,840+0200 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000768739', 'lastCheck': '7.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000505049', 'lastCheck': '7.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000689267', 'lastCheck': '7.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000362478', 'lastCheck': '3.8', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=0ca8870c-d6f1-45de-b93e-67c69b102863 (api:54) 2019-04-13 21:34:01,840+0200 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=5ab65d08-3d41-43d0-a703-919a23963246 (api:48) 2019-04-13 21:34:01,841+0200 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=5ab65d08-3d41-43d0-a703-919a23963246 (api:54) 2019-04-13 21:34:01,854+0200 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '18.40', 'nodeIndex': 1, 'cpuSys': '2.85', 'cpuIdle': '78.75'}, '21': {'cpuUser': '23.10', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '74.58'}, '22': {'cpuUser': '19.06', 'nodeIndex': 1, 'cpuSys': '2.52', 'cpuIdle': '78.42'}, '23': {'cpuUser': '21.97', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '75.91'}, '1': {'cpuUser': '45.93', 'nodeIndex': 0, 'cpuSys': '4.57', 'cpuIdle': '49.50'}, '0': {'cpuUser': '47.39', 'nodeIndex': 0, 'cpuSys': '2.38', 'cpuIdle': '50.23'}, '3': {'cpuUser': '59.57', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '38.44'}, '2': {'cpuUser': '53.55', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '44.66'}, '5': {'cpuUser': '55.66', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '42.55'}, '4': {'cpuUser': '57.25', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '40.96'}, '7': {'cpuUser': '33.36', 'nodeIndex': 1, 'cpuSys': '3.57', 'cpuIdle': '63.07'}, '6': {'cpuUser': '34.95', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '61.87'}, '9': {'cpuUser': '41.17', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '56.18'}, '8': {'cpuUser': '38.52', 'nodeIndex': 1, 'cpuSys': '3.57', 'cpuIdle': '57.91'}, '11': {'cpuUser': '34.55', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '62.67'}, '10': {'cpuUser': '34.09', 'nodeIndex': 1, 'cpuSys': '3.44', 'cpuIdle': '62.47'}, '13': {'cpuUser': '18.20', 'nodeIndex': 0, 'cpuSys': '2.52', 'cpuIdle': '79.28'}, '12': {'cpuUser': '39.98', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '58.23'}, '15': {'cpuUser': '25.08', 'nodeIndex': 0, 'cpuSys': '2.05', 'cpuIdle': '72.87'}, '14': {'cpuUser': '34.35', 'nodeIndex': 0, 'cpuSys': '1.52', 'cpuIdle': '64.13'}, '17': {'cpuUser': '33.62', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '64.46'}, '16': {'cpuUser': '21.71', 'nodeIndex': 0, 'cpuSys': '2.12', 'cpuIdle': '76.17'}, '19': {'cpuUser': '21.58', 'nodeIndex': 1, 'cpuSys': '2.32', 'cpuIdle': '76.10'}, '18': {'cpuUser': '21.18', 'nodeIndex': 1, 'cpuSys': '2.12', 'cpuIdle': '76.70'}}, 'numaNodeMemFree': {'1': {'memPercent': 31, 'memFree': '68670'}, '0': {'memPercent': 20, 'memFree': '79062'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000768739', 'lastCheck': '7.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000689267', 'lastCheck': '7.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000362478', 'lastCheck': '3.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000505049', 'lastCheck': '7.5', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168869865', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '5091493962', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590474753956', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '2705655104019', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433987603382', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '259142406375', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239883475', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '366097246', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122808023', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121329520', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '857093', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423834112568', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '253723324164', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215287109', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680516039', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '29547818392', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653306004', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '29289499122', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255912844262', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '7189343484', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121703863', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '75430063613', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '62024520230', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122263003', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '24423504', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418948613', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '1012777283', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121515450', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '1041572', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184041.838354, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184041.838354, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184041.838354, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184041.838354, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121285481', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458767925727', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184041.838354, 'rx': '2577012078142', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322686652173', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184041.838354, 'rx': '1322686652173', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184041.838354, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940868665', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '2700704165535', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571533885291', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '4950938484', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '330006965507', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184041.838354, 'rx': '980533335803', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184041.838354, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184041.838354, 'rx': '45371605', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184041.838354, 'rx': '39849168', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40562', 'ksmPages': 100, 'elapsedTime': '1728789.94', 'cpuLoad': '8.12', 'cpuSys': '2.48', 'diskStats': {'/var/log': {'free': '47798'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47798'}}, 'cpuUserVdsmd': '3.04', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150381, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:34:01 GMT', 'cpuUser': '34.76', 'memFree': 150125, 'cpuIdle': '62.76', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.39'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:34:01,860+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.08 seconds (__init__:312) 2019-04-13 21:34:03,237+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=878f0572-eb85-4ad3-8d7c-dca575d2124a (api:48) 2019-04-13 21:34:03,238+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000768739', 'lastCheck': '8.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000505049', 'lastCheck': '8.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000689267', 'lastCheck': '8.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000362478', 'lastCheck': '5.2', 'valid': True}} from=internal, task_id=878f0572-eb85-4ad3-8d7c-dca575d2124a (api:54) 2019-04-13 21:34:03,866+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=4bd4252e-c27f-4c4c-bbaa-16962df2262d (api:48) 2019-04-13 21:34:03,870+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=4bd4252e-c27f-4c4c-bbaa-16962df2262d (api:54) 2019-04-13 21:34:03,871+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:34:03,881+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=d99e9949-db92-4d2b-ba4e-5aec6d057e86 (api:48) 2019-04-13 21:34:03,889+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706953138176', 'isoprefix': '', 'alerts': [], 'disktotal': '3898049822720', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706953138176', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719701725184', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706953138176', 'isoprefix': '', 'alerts': [], 'disktotal': '4364341084160', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=d99e9949-db92-4d2b-ba4e-5aec6d057e86 (api:54) 2019-04-13 21:34:03,890+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:34:04,110+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID=u'ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', options=None) from=internal, task_id=4fe1abd7-1f69-455b-908f-66dd0941e3d9 (api:48) 2019-04-13 21:34:04,113+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', options=None) from=internal, task_id=6f317f3e-3b28-4aef-b69f-de5f94f93b0b (api:48) 2019-04-13 21:34:04,115+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', options=None) from=internal, task_id=6e02f019-4dfd-49ec-a6fc-497af590b8b2 (api:48) 2019-04-13 21:34:04,121+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '5697701376', 'apparentsize': '16106127360'} from=internal, task_id=6e02f019-4dfd-49ec-a6fc-497af590b8b2 (api:54) 2019-04-13 21:34:04,122+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID='ef6741d1-380a-4736-a924-f88ed68e1270', options=None) from=internal, task_id=c742ede6-f56a-4063-baf2-cb8c97ec7458 (api:48) 2019-04-13 21:34:04,123+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '4579679744', 'apparentsize': '16106127360'} from=internal, task_id=6f317f3e-3b28-4aef-b69f-de5f94f93b0b (api:54) 2019-04-13 21:34:04,124+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID=u'67a4dd71-69ff-49a8-b8a4-c7101c75e105', options=None) from=internal, task_id=563c27eb-0a62-4e6a-8795-fb4fe72898fc (api:48) 2019-04-13 21:34:04,126+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '53760838144', 'apparentsize': '139586437120'} from=internal, task_id=4fe1abd7-1f69-455b-908f-66dd0941e3d9 (api:54) 2019-04-13 21:34:04,127+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', options=None) from=internal, task_id=5c0c1443-6d98-492f-88de-d203a44bf826 (api:48) 2019-04-13 21:34:04,133+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '6477462016', 'apparentsize': '21474836480'} from=internal, task_id=c742ede6-f56a-4063-baf2-cb8c97ec7458 (api:54) 2019-04-13 21:34:04,134+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID=u'2b136ed7-8b9a-45c1-9edc-64ee96c87c43', options=None) from=internal, task_id=3c65c2b4-6636-4816-9a70-6438978bf114 (api:48) 2019-04-13 21:34:04,135+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID=u'44b3ceca-f0db-43da-af29-aa681df146ac', options=None) from=internal, task_id=daf05424-c3f2-48a6-8e7b-4ca742a6fb87 (api:48) 2019-04-13 21:34:04,139+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '55059099136', 'apparentsize': '171798691840'} from=internal, task_id=563c27eb-0a62-4e6a-8795-fb4fe72898fc (api:54) 2019-04-13 21:34:04,140+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=ed3e703c-428c-4661-b161-125c471c2ecf (api:48) 2019-04-13 21:34:04,140+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2524562432', 'apparentsize': '12884901888'} from=internal, task_id=5c0c1443-6d98-492f-88de-d203a44bf826 (api:54) 2019-04-13 21:34:04,141+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID='123e2cc8-8f24-4ddd-b401-68393535f555', options=None) from=internal, task_id=141d0ac8-2142-4fbd-be7d-4725efaa5c6b (api:48) 2019-04-13 21:34:04,147+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '11196656640', 'apparentsize': '21474836480'} from=internal, task_id=daf05424-c3f2-48a6-8e7b-4ca742a6fb87 (api:54) 2019-04-13 21:34:04,148+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID='3f6c31c6-ec38-4979-a617-55537123ee61', options=None) from=internal, task_id=f39d3ebf-deac-4ab2-a99d-1f769c2467d8 (api:48) 2019-04-13 21:34:04,150+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2024405504', 'apparentsize': '10737418240'} from=internal, task_id=3c65c2b4-6636-4816-9a70-6438978bf114 (api:54) 2019-04-13 21:34:04,151+0200 INFO (periodic/0) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', options=None) from=internal, task_id=2dd163a1-e59a-4375-b2ad-4e4ecf24d58c (api:48) 2019-04-13 21:34:04,153+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2116486656', 'apparentsize': '3948150784'} from=internal, task_id=ed3e703c-428c-4661-b161-125c471c2ecf (api:54) 2019-04-13 21:34:04,153+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID=u'9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID=u'fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID=u'a1d583c1-e135-4e3f-b6fa-0f29a5261366', options=None) from=internal, task_id=819ffac6-4cd7-4d97-b78b-5e8e74ef8c8b (api:48) 2019-04-13 21:34:04,154+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '2283003904', 'apparentsize': '16106127360'} from=internal, task_id=141d0ac8-2142-4fbd-be7d-4725efaa5c6b (api:54) 2019-04-13 21:34:04,154+0200 INFO (periodic/3) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', options=None) from=internal, task_id=87c018bc-1966-45e4-a51b-01ffd928d568 (api:48) 2019-04-13 21:34:04,157+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '4998711296', 'apparentsize': '16106127360'} from=internal, task_id=f39d3ebf-deac-4ab2-a99d-1f769c2467d8 (api:54) 2019-04-13 21:34:04,157+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID='a884b4ce-381f-4653-be14-a31d15181f48', options=None) from=internal, task_id=b99d42ec-8a06-434b-9f8f-bd51d8ade5a5 (api:48) 2019-04-13 21:34:04,159+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '1294889472', 'apparentsize': '12884901888'} from=internal, task_id=819ffac6-4cd7-4d97-b78b-5e8e74ef8c8b (api:54) 2019-04-13 21:34:04,160+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', options=None) from=internal, task_id=1e9f317b-ba45-4167-8e74-4ee07376e3d1 (api:48) 2019-04-13 21:34:04,160+0200 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'truesize': '7263185920', 'apparentsize': '21474836480'} from=internal, task_id=87c018bc-1966-45e4-a51b-01ffd928d568 (api:54) 2019-04-13 21:34:04,161+0200 INFO (periodic/0) [vdsm.api] FINISH getVolumeSize return={'truesize': '2109410304', 'apparentsize': '8589934592'} from=internal, task_id=2dd163a1-e59a-4375-b2ad-4e4ecf24d58c (api:54) 2019-04-13 21:34:04,162+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'truesize': '3447286784', 'apparentsize': '16106127360'} from=internal, task_id=b99d42ec-8a06-434b-9f8f-bd51d8ade5a5 (api:54) 2019-04-13 21:34:04,169+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'truesize': '2879770112', 'apparentsize': '8589934592'} from=internal, task_id=1e9f317b-ba45-4167-8e74-4ee07376e3d1 (api:54) 2019-04-13 21:34:05,422+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:34:05,451+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:34:05,451+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:34:05,452+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:34:08,167+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:34:08,196+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:34:08,196+0200 ERROR (jsonrpc/1) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:34:08,197+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:34:13,925+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=2d47d89b-2dfb-47e7-80ae-2bc2b8f53b88 (api:48) 2019-04-13 21:34:13,930+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=2d47d89b-2dfb-47e7-80ae-2bc2b8f53b88 (api:54) 2019-04-13 21:34:13,930+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:34:14,033+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=9f152895-fb1d-4d04-bdbc-8106faf2948a (api:48) 2019-04-13 21:34:14,041+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706948943872', 'isoprefix': '', 'alerts': [], 'disktotal': '3898045628416', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706948943872', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719696482304', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706948943872', 'isoprefix': '', 'alerts': [], 'disktotal': '4364336889856', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=9f152895-fb1d-4d04-bdbc-8106faf2948a (api:54) 2019-04-13 21:34:14,042+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:34:16,845+0200 INFO (jsonrpc/6) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:34:16,910+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=50632cf6-6883-4813-8b47-a4949495d7aa (api:48) 2019-04-13 21:34:16,911+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000900675', 'lastCheck': '2.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000583341', 'lastCheck': '2.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000481761', 'lastCheck': '2.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000390818', 'lastCheck': '2.7', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=50632cf6-6883-4813-8b47-a4949495d7aa (api:54) 2019-04-13 21:34:16,912+0200 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=092fadfc-afb7-4388-853f-003429a88308 (api:48) 2019-04-13 21:34:16,912+0200 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=092fadfc-afb7-4388-853f-003429a88308 (api:54) 2019-04-13 21:34:16,925+0200 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '20.06', 'nodeIndex': 1, 'cpuSys': '2.65', 'cpuIdle': '77.29'}, '21': {'cpuUser': '17.21', 'nodeIndex': 1, 'cpuSys': '2.71', 'cpuIdle': '80.08'}, '22': {'cpuUser': '14.70', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '82.32'}, '23': {'cpuUser': '25.69', 'nodeIndex': 1, 'cpuSys': '2.25', 'cpuIdle': '72.06'}, '1': {'cpuUser': '43.37', 'nodeIndex': 0, 'cpuSys': '4.63', 'cpuIdle': '52.00'}, '0': {'cpuUser': '37.87', 'nodeIndex': 0, 'cpuSys': '2.65', 'cpuIdle': '59.48'}, '3': {'cpuUser': '34.69', 'nodeIndex': 0, 'cpuSys': '2.65', 'cpuIdle': '62.66'}, '2': {'cpuUser': '36.81', 'nodeIndex': 0, 'cpuSys': '2.58', 'cpuIdle': '60.61'}, '5': {'cpuUser': '33.10', 'nodeIndex': 0, 'cpuSys': '2.85', 'cpuIdle': '64.05'}, '4': {'cpuUser': '34.30', 'nodeIndex': 0, 'cpuSys': '2.98', 'cpuIdle': '62.72'}, '7': {'cpuUser': '34.16', 'nodeIndex': 1, 'cpuSys': '3.05', 'cpuIdle': '62.79'}, '6': {'cpuUser': '36.22', 'nodeIndex': 1, 'cpuSys': '2.98', 'cpuIdle': '60.80'}, '9': {'cpuUser': '35.29', 'nodeIndex': 1, 'cpuSys': '3.18', 'cpuIdle': '61.53'}, '8': {'cpuUser': '36.61', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '60.08'}, '11': {'cpuUser': '35.22', 'nodeIndex': 1, 'cpuSys': '2.78', 'cpuIdle': '62.00'}, '10': {'cpuUser': '35.88', 'nodeIndex': 1, 'cpuSys': '3.31', 'cpuIdle': '60.81'}, '13': {'cpuUser': '14.50', 'nodeIndex': 0, 'cpuSys': '1.92', 'cpuIdle': '83.58'}, '12': {'cpuUser': '15.82', 'nodeIndex': 0, 'cpuSys': '1.79', 'cpuIdle': '82.39'}, '15': {'cpuUser': '16.88', 'nodeIndex': 0, 'cpuSys': '1.99', 'cpuIdle': '81.13'}, '14': {'cpuUser': '17.35', 'nodeIndex': 0, 'cpuSys': '2.18', 'cpuIdle': '80.47'}, '17': {'cpuUser': '19.27', 'nodeIndex': 0, 'cpuSys': '1.72', 'cpuIdle': '79.01'}, '16': {'cpuUser': '19.00', 'nodeIndex': 0, 'cpuSys': '2.58', 'cpuIdle': '78.42'}, '19': {'cpuUser': '21.98', 'nodeIndex': 1, 'cpuSys': '2.45', 'cpuIdle': '75.57'}, '18': {'cpuUser': '22.38', 'nodeIndex': 1, 'cpuSys': '2.58', 'cpuIdle': '75.04'}}, 'numaNodeMemFree': {'1': {'memPercent': 31, 'memFree': '68719'}, '0': {'memPercent': 20, 'memFree': '79006'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000900675', 'lastCheck': '2.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000481761', 'lastCheck': '2.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000390818', 'lastCheck': '2.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000583341', 'lastCheck': '2.5', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168869865', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '5091493962', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590475892838', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '2705656335873', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433991942008', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '259145264480', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239884225', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '366098036', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122808023', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121329520', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '857093', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423838428788', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '253726004458', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215287109', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680518533', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '29547819422', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653307034', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '29289501616', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255915702995', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '7191448409', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121703863', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '76003900511', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '62366903188', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122263003', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '24423504', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418949093', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '1012777283', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121515450', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '1041572', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184056.909478, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184056.909478, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184056.909478, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184056.909478, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121285481', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458769021685', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184056.909478, 'rx': '2577013252588', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322699134008', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184056.909478, 'rx': '1322699134008', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184056.909478, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940868887', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '2700705396273', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571535023951', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '4950939600', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '330278942485', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184056.909478, 'rx': '981166620469', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184056.909478, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184056.909478, 'rx': '45371995', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184056.909478, 'rx': '39849536', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40566', 'ksmPages': 100, 'elapsedTime': '1728805.01', 'cpuLoad': '8.18', 'cpuSys': '2.70', 'diskStats': {'/var/log': {'free': '47798'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47798'}}, 'cpuUserVdsmd': '3.11', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150384, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:34:16 GMT', 'cpuUser': '27.44', 'memFree': 150128, 'cpuIdle': '69.86', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.19'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:34:16,931+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.08 seconds (__init__:312) 2019-04-13 21:34:18,309+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=5cd4ecd7-b2a0-4135-a8f5-47cd38fd2814 (api:48) 2019-04-13 21:34:18,309+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000900675', 'lastCheck': '3.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000583341', 'lastCheck': '3.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000481761', 'lastCheck': '3.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000390818', 'lastCheck': '0.3', 'valid': True}} from=internal, task_id=5cd4ecd7-b2a0-4135-a8f5-47cd38fd2814 (api:54) 2019-04-13 21:34:20,502+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:34:20,531+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:34:20,531+0200 ERROR (jsonrpc/4) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:34:20,532+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:34:23,203+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:34:23,231+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats error='appsList' from=::ffff:192.168.0.19,57576 (api:52) 2019-04-13 21:34:23,231+0200 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:34:23,232+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:34:24,104+0200 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=099a9e2e-1716-4d3c-b4eb-7a0cfe42373c (api:48) 2019-04-13 21:34:24,109+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=099a9e2e-1716-4d3c-b4eb-7a0cfe42373c (api:54) 2019-04-13 21:34:24,109+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:34:24,123+0200 INFO (jsonrpc/2) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=bd5284e4-d171-42a7-aa6e-10efb5762b50 (api:48) 2019-04-13 21:34:24,130+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706894417920', 'isoprefix': '', 'alerts': [], 'disktotal': '3897991102464', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706894417920', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719641956352', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706894417920', 'isoprefix': '', 'alerts': [], 'disktotal': '4364282363904', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=bd5284e4-d171-42a7-aa6e-10efb5762b50 (api:54) 2019-04-13 21:34:24,131+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:34:31,925+0200 INFO (jsonrpc/1) [api.host] START getStats() from=::ffff:192.168.0.19,57576 (api:48) 2019-04-13 21:34:31,999+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,57576, task_id=c56a8fc1-ab7d-4751-9823-aaf95496ebdd (api:48) 2019-04-13 21:34:32,000+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470044', 'lastCheck': '7.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000635744', 'lastCheck': '7.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000703247', 'lastCheck': '7.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000404144', 'lastCheck': '3.9', 'valid': True}} from=::ffff:192.168.0.19,57576, task_id=c56a8fc1-ab7d-4751-9823-aaf95496ebdd (api:54) 2019-04-13 21:34:32,001+0200 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,57576, task_id=5bf2e243-adfa-4ad8-ae8c-7d4c13159593 (api:48) 2019-04-13 21:34:32,001+0200 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,57576, task_id=5bf2e243-adfa-4ad8-ae8c-7d4c13159593 (api:54) 2019-04-13 21:34:32,020+0200 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '27.19', 'nodeIndex': 1, 'cpuSys': '3.98', 'cpuIdle': '68.83'}, '21': {'cpuUser': '21.55', 'nodeIndex': 1, 'cpuSys': '3.58', 'cpuIdle': '74.87'}, '22': {'cpuUser': '25.80', 'nodeIndex': 1, 'cpuSys': '3.12', 'cpuIdle': '71.08'}, '23': {'cpuUser': '24.60', 'nodeIndex': 1, 'cpuSys': '4.24', 'cpuIdle': '71.16'}, '1': {'cpuUser': '8.69', 'nodeIndex': 0, 'cpuSys': '6.10', 'cpuIdle': '85.21'}, '0': {'cpuUser': '9.62', 'nodeIndex': 0, 'cpuSys': '4.58', 'cpuIdle': '85.80'}, '3': {'cpuUser': '9.75', 'nodeIndex': 0, 'cpuSys': '4.31', 'cpuIdle': '85.94'}, '2': {'cpuUser': '8.02', 'nodeIndex': 0, 'cpuSys': '3.85', 'cpuIdle': '88.13'}, '5': {'cpuUser': '6.50', 'nodeIndex': 0, 'cpuSys': '4.51', 'cpuIdle': '88.99'}, '4': {'cpuUser': '8.42', 'nodeIndex': 0, 'cpuSys': '4.91', 'cpuIdle': '86.67'}, '7': {'cpuUser': '32.30', 'nodeIndex': 1, 'cpuSys': '9.35', 'cpuIdle': '58.35'}, '6': {'cpuUser': '39.92', 'nodeIndex': 1, 'cpuSys': '4.38', 'cpuIdle': '55.70'}, '9': {'cpuUser': '35.35', 'nodeIndex': 1, 'cpuSys': '4.64', 'cpuIdle': '60.01'}, '8': {'cpuUser': '37.27', 'nodeIndex': 1, 'cpuSys': '4.58', 'cpuIdle': '58.15'}, '11': {'cpuUser': '35.94', 'nodeIndex': 1, 'cpuSys': '4.58', 'cpuIdle': '59.48'}, '10': {'cpuUser': '39.99', 'nodeIndex': 1, 'cpuSys': '4.05', 'cpuIdle': '55.96'}, '13': {'cpuUser': '7.29', 'nodeIndex': 0, 'cpuSys': '5.44', 'cpuIdle': '87.27'}, '12': {'cpuUser': '3.58', 'nodeIndex': 0, 'cpuSys': '3.71', 'cpuIdle': '92.71'}, '15': {'cpuUser': '5.90', 'nodeIndex': 0, 'cpuSys': '1.19', 'cpuIdle': '92.91'}, '14': {'cpuUser': '5.70', 'nodeIndex': 0, 'cpuSys': '4.97', 'cpuIdle': '89.33'}, '17': {'cpuUser': '4.18', 'nodeIndex': 0, 'cpuSys': '2.52', 'cpuIdle': '93.30'}, '16': {'cpuUser': '10.74', 'nodeIndex': 0, 'cpuSys': '1.39', 'cpuIdle': '87.87'}, '19': {'cpuUser': '25.20', 'nodeIndex': 1, 'cpuSys': '4.05', 'cpuIdle': '70.75'}, '18': {'cpuUser': '25.33', 'nodeIndex': 1, 'cpuSys': '3.71', 'cpuIdle': '70.96'}}, 'numaNodeMemFree': {'1': {'memPercent': 31, 'memFree': '68622'}, '0': {'memPercent': 20, 'memFree': '79105'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '23', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470044', 'lastCheck': '7.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000703247', 'lastCheck': '7.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000404144', 'lastCheck': '3.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000635744', 'lastCheck': '7.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'vnet8': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet8', 'tx': '2168870245', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '5091494280', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '1590476558270', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '2705656823623', 'state': 'up'}, 'vnet12': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet12', 'tx': '433996213602', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '259148157727', 'state': 'up'}, 'vnet13': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet13', 'tx': '1239884509', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '366098120', 'state': 'up'}, 'vnet10': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet10', 'tx': '122808187', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '15226276', 'state': 'up'}, 'vnet11': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet11', 'tx': '121329684', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '857093', 'state': 'up'}, 'vnet14': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet14', 'tx': '423842945704', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '253728855541', 'state': 'up'}, 'vnet15': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet15', 'tx': '215287273', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '111421507', 'state': 'up'}, 'vnet0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', 'tx': '2680518697', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '29547819422', 'state': 'up'}, 'vnet1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet1', 'tx': '6653307034', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '29289501780', 'state': 'up'}, 'vnet2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet2', 'tx': '255918611486', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '7193936763', 'state': 'up'}, 'vnet3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet3', 'tx': '121704027', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '567406', 'state': 'up'}, 'vnet4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet4', 'tx': '76036524511', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '62629881842', 'state': 'up'}, 'vnet5': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet5', 'tx': '122263167', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '24423504', 'state': 'up'}, 'vnet6': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet6', 'tx': '13418949621', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '1012777457', 'state': 'up'}, 'vnet7': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet7', 'tx': '121515614', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '1041572', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184071.996874, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184071.996874, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184071.996874, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184071.996874, 'rx': '0', 'state': 'down'}, 'vnet9': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet9', 'tx': '121285645', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '221882', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '1458769684169', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184071.996874, 'rx': '2577013715704', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1322709505915', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184071.996874, 'rx': '1322709505915', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184071.996874, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '18940869431', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '2700705883671', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1571535688839', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '4950939952', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '330648336604', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184071.996874, 'rx': '981211075915', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184071.996874, 'rx': '0', 'state': 'down'}, 'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184071.996874, 'rx': '45372385', 'state': 'up'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184071.996874, 'rx': '39849858', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '40568', 'ksmPages': 100, 'elapsedTime': '1728820.09', 'cpuLoad': '7.99', 'cpuSys': '4.23', 'diskStats': {'/var/log': {'free': '47798'}, '/var/run/vdsm/': {'free': '96430'}, '/tmp': {'free': '47798'}}, 'cpuUserVdsmd': '3.18', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 150372, 'bootTime': '1553455195', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:34:32 GMT', 'cpuUser': '19.12', 'memFree': 150116, 'cpuIdle': '76.65', 'vmActive': 16, 'v2vJobs': {}, 'cpuSysVdsmd': '1.39'}} from=::ffff:192.168.0.19,57576 (api:54) 2019-04-13 21:34:32,027+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.10 seconds (__init__:312) 2019-04-13 21:34:33,388+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=8e20185d-9053-4b54-ad22-ce81c26aaa28 (api:48) 2019-04-13 21:34:33,389+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470044', 'lastCheck': '9.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000635744', 'lastCheck': '9.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000703247', 'lastCheck': '9.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000404144', 'lastCheck': '5.3', 'valid': True}} from=internal, task_id=8e20185d-9053-4b54-ad22-ce81c26aaa28 (api:54) 2019-04-13 21:34:34,181+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57576, task_id=f23fad68-2e2c-40c9-aa86-29a16754ddcb (api:48) 2019-04-13 21:34:34,193+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 26L}} from=::ffff:192.168.0.19,57576, task_id=f23fad68-2e2c-40c9-aa86-29a16754ddcb (api:54) 2019-04-13 21:34:34,194+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.02 seconds (__init__:312) 2019-04-13 21:34:34,206+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,57604, task_id=e049699c-e9b7-4959-8b95-a80b9d486cbf (api:48) 2019-04-13 21:34:34,215+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 26L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,57604, task_id=e049699c-e9b7-4959-8b95-a80b9d486cbf (api:54) 2019-04-13 21:34:34,216+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:34:35,581+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,44160 (api:48) 2019-04-13 21:34:35,611+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats error='appsList' from=::1,44160 (api:52) 2019-04-13 21:34:35,612+0200 ERROR (jsonrpc/6) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' 2019-04-13 21:34:35,613+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats failed (error -32603) in 0.03 seconds (__init__:312) 2019-04-13 21:40:11,351+0200 INFO (MainThread) [vds] (PID: 14318) I am the actual vdsm 4.30.11.1 onode-1.discovery.intern (3.10.0-957.10.1.el7.x86_64) (vdsmd:150) 2019-04-13 21:40:11,352+0200 INFO (MainThread) [vds] VDSM will run with cpu affinity: frozenset([1]) (vdsmd:262) 2019-04-13 21:40:11,356+0200 INFO (MainThread) [storage.HSM] START HSM init (hsm:367) 2019-04-13 21:40:11,360+0200 INFO (MainThread) [storage.HSM] Creating data-center mount directory '/rhev/data-center/mnt' (hsm:372) 2019-04-13 21:40:11,360+0200 INFO (MainThread) [storage.fileUtils] Creating directory: /rhev/data-center/mnt mode: None (fileUtils:199) 2019-04-13 21:40:11,557+0200 INFO (MainThread) [storage.HSM] Unlinking file '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782' (hsm:522) 2019-04-13 21:40:11,557+0200 INFO (MainThread) [storage.HSM] Unlinking file '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a' (hsm:522) 2019-04-13 21:40:11,558+0200 INFO (MainThread) [storage.HSM] Unlinking file '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d' (hsm:522) 2019-04-13 21:40:11,558+0200 INFO (MainThread) [storage.HSM] Unlinking file '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e' (hsm:522) 2019-04-13 21:40:11,558+0200 INFO (MainThread) [storage.HSM] Unlinking file '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd' (hsm:522) 2019-04-13 21:40:11,559+0200 INFO (MainThread) [storage.udev] Registering multipath event monitor (udev:182) 2019-04-13 21:40:11,559+0200 INFO (MainThread) [storage.udev] Starting multipath event listener (udev:116) 2019-04-13 21:40:11,585+0200 INFO (MainThread) [storage.check] Starting check service (check:91) 2019-04-13 21:40:11,594+0200 INFO (MainThread) [storage.Dispatcher] Starting StorageDispatcher... (dispatcher:47) 2019-04-13 21:40:11,595+0200 INFO (check/loop) [storage.asyncevent] Starting (asyncevent:125) 2019-04-13 21:40:11,628+0200 INFO (MainThread) [vdsm.api] START registerDomainStateChangeCallback(callbackFunc=) from=internal, task_id=32b66104-1ad6-4275-8dc0-8146870ee5c8 (api:48) 2019-04-13 21:40:11,628+0200 INFO (MainThread) [vdsm.api] FINISH registerDomainStateChangeCallback return=None from=internal, task_id=32b66104-1ad6-4275-8dc0-8146870ee5c8 (api:54) 2019-04-13 21:40:11,629+0200 INFO (MainThread) [MOM] Preparing MOM interface (momIF:45) 2019-04-13 21:40:11,629+0200 INFO (MainThread) [MOM] MOM: Using named unix socket: /var/run/vdsm/mom-vdsm.sock (momIF:55) 2019-04-13 21:40:11,630+0200 INFO (MainThread) [root] Unregistering all secrets (secret:93) 2019-04-13 21:40:11,652+0200 INFO (MainThread) [vds] Setting channels' timeout to 30 seconds. (vmchannels:224) 2019-04-13 21:40:11,653+0200 INFO (MainThread) [vds] Starting QEMU-GA poller (qemuguestagent:151) 2019-04-13 21:40:11,657+0200 INFO (MainThread) [vds.MultiProtocolAcceptor] Listening at :::54321 (protocoldetector:183) 2019-04-13 21:40:11,783+0200 INFO (http) [vds] Server running (http:59) 2019-04-13 21:40:11,992+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=cca1806e-9775-446d-9eb8-b7a289058ada (api:48) 2019-04-13 21:40:11,993+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=cca1806e-9775-446d-9eb8-b7a289058ada (api:54) 2019-04-13 21:40:11,994+0200 WARN (periodic/1) [throttled] MOM not available. (throttledlog:104) 2019-04-13 21:40:11,994+0200 WARN (periodic/1) [throttled] MOM not available, KSM stats will be missing. (throttledlog:104) 2019-04-13 21:40:12,140+0200 INFO (vmrecovery) [vds] VM recovered: b5713713-ac04-4a91-95b8-3c3f17eba32f (recovery:97) 2019-04-13 21:40:12,141+0200 INFO (vmrecovery) [vds] recovery [1:1/16]: recovered domain b5713713-ac04-4a91-95b8-3c3f17eba32f (recovery:123) 2019-04-13 21:40:12,159+0200 INFO (vm/b5713713) [virt.vm] (vmId='b5713713-ac04-4a91-95b8-3c3f17eba32f') VM wrapper has started (vm:2746) 2019-04-13 21:40:12,160+0200 INFO (vm/b5713713) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID='a1d583c1-e135-4e3f-b6fa-0f29a5261366', options=None) from=internal, task_id=6d835c86-ef85-4c83-9056-cfbd1b046191 (api:48) 2019-04-13 21:40:12,176+0200 INFO (vmrecovery) [vds] VM recovered: 0f8aff0a-7915-4214-a37c-c9fee6518869 (recovery:97) 2019-04-13 21:40:12,176+0200 INFO (vmrecovery) [vds] recovery [1:2/16]: recovered domain 0f8aff0a-7915-4214-a37c-c9fee6518869 (recovery:123) 2019-04-13 21:40:12,194+0200 INFO (vm/0f8aff0a) [virt.vm] (vmId='0f8aff0a-7915-4214-a37c-c9fee6518869') VM wrapper has started (vm:2746) 2019-04-13 21:40:12,196+0200 INFO (vm/0f8aff0a) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID='ef6741d1-380a-4736-a924-f88ed68e1270', options=None) from=internal, task_id=9eab2e4f-d237-40e3-ac88-d3ca77c71337 (api:48) 2019-04-13 21:40:12,200+0200 INFO (vm/c8df4609) [virt.vm] (vmId='c8df4609-fe26-4b73-9137-57724aff3ca4') VM wrapper has started (vm:2746) 2019-04-13 21:40:12,201+0200 INFO (vm/c8df4609) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', options=None) from=internal, task_id=2efcded7-b885-4814-ad9b-f26e692d1bcf (api:48) 2019-04-13 21:40:12,205+0200 INFO (vmrecovery) [vds] VM recovered: c8df4609-fe26-4b73-9137-57724aff3ca4 (recovery:97) 2019-04-13 21:40:12,205+0200 INFO (vmrecovery) [vds] recovery [1:3/16]: recovered domain c8df4609-fe26-4b73-9137-57724aff3ca4 (recovery:123) 2019-04-13 21:40:12,226+0200 INFO (vm/a40d738c) [virt.vm] (vmId='a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3') VM wrapper has started (vm:2746) 2019-04-13 21:40:12,227+0200 INFO (vm/a40d738c) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID='44b3ceca-f0db-43da-af29-aa681df146ac', options=None) from=internal, task_id=4c73bac8-09e2-4f4c-9e4f-6235251c17e3 (api:48) 2019-04-13 21:40:12,232+0200 INFO (vmrecovery) [vds] VM recovered: a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3 (recovery:97) 2019-04-13 21:40:12,232+0200 INFO (vmrecovery) [vds] recovery [1:4/16]: recovered domain a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3 (recovery:123) 2019-04-13 21:40:12,268+0200 INFO (vmrecovery) [vds] VM recovered: 28d7d5ff-2ca1-42c6-ae8d-065c721d3066 (recovery:97) 2019-04-13 21:40:12,269+0200 INFO (vmrecovery) [vds] recovery [1:5/16]: recovered domain 28d7d5ff-2ca1-42c6-ae8d-065c721d3066 (recovery:123) 2019-04-13 21:40:12,295+0200 INFO (vmrecovery) [vds] VM recovered: dd24e11f-40aa-47d2-9f0b-11cd7e83649d (recovery:97) 2019-04-13 21:40:12,295+0200 INFO (vmrecovery) [vds] recovery [1:6/16]: recovered domain dd24e11f-40aa-47d2-9f0b-11cd7e83649d (recovery:123) 2019-04-13 21:40:12,304+0200 WARN (vm/b5713713) [storage.LVM] Reloading VGs failed (vgs=['9c3f06cf-7475-448e-819b-f4f52fa7d782'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782']) (lvm:442) 2019-04-13 21:40:12,313+0200 INFO (itmap/0) [IOProcessClient] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting client (__init__:308) 2019-04-13 21:40:12,329+0200 INFO (vmrecovery) [vds] VM recovered: cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6 (recovery:97) 2019-04-13 21:40:12,330+0200 INFO (vmrecovery) [vds] recovery [1:7/16]: recovered domain cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6 (recovery:123) 2019-04-13 21:40:12,350+0200 INFO (ioprocess/14443) [IOProcess] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting ioprocess (__init__:434) 2019-04-13 21:40:12,354+0200 INFO (itmap/0) [IOProcessClient] (/onode-1.discovery.intern:_mnt_store) Starting client (__init__:308) 2019-04-13 21:40:12,362+0200 INFO (vmrecovery) [vds] VM recovered: afa2c17f-39e4-4168-aad7-2548dd19cc26 (recovery:97) 2019-04-13 21:40:12,363+0200 INFO (vmrecovery) [vds] recovery [1:8/16]: recovered domain afa2c17f-39e4-4168-aad7-2548dd19cc26 (recovery:123) 2019-04-13 21:40:12,393+0200 INFO (ioprocess/14455) [IOProcess] (/onode-1.discovery.intern:_mnt_store) Starting ioprocess (__init__:434) 2019-04-13 21:40:12,393+0200 INFO (itmap/1) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_vms) Starting client (__init__:308) 2019-04-13 21:40:12,408+0200 INFO (vmrecovery) [vds] VM recovered: 5a326302-86c1-41b5-ba58-793232299713 (recovery:97) 2019-04-13 21:40:12,409+0200 INFO (vmrecovery) [vds] recovery [1:9/16]: recovered domain 5a326302-86c1-41b5-ba58-793232299713 (recovery:123) 2019-04-13 21:40:12,445+0200 INFO (itmap/2) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_isos) Starting client (__init__:308) 2019-04-13 21:40:12,450+0200 INFO (ioprocess/14467) [IOProcess] (/nas.discovery.intern:_mnt_d2600_vms) Starting ioprocess (__init__:434) 2019-04-13 21:40:12,463+0200 INFO (vmrecovery) [vds] VM recovered: 6772f848-4874-4785-9bb0-88d40bfb71c4 (recovery:97) 2019-04-13 21:40:12,464+0200 INFO (vmrecovery) [vds] recovery [1:10/16]: recovered domain 6772f848-4874-4785-9bb0-88d40bfb71c4 (recovery:123) 2019-04-13 21:40:12,484+0200 INFO (itmap/3) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting client (__init__:308) 2019-04-13 21:40:12,487+0200 INFO (ioprocess/14475) [IOProcess] (/nas.discovery.intern:_mnt_d2600_isos) Starting ioprocess (__init__:434) 2019-04-13 21:40:12,504+0200 INFO (vmrecovery) [vds] VM recovered: 7b5e0fdd-8757-49ce-acf9-3206a372e1c9 (recovery:97) 2019-04-13 21:40:12,504+0200 INFO (vmrecovery) [vds] recovery [1:11/16]: recovered domain 7b5e0fdd-8757-49ce-acf9-3206a372e1c9 (recovery:123) 2019-04-13 21:40:12,524+0200 INFO (ioprocess/14483) [IOProcess] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting ioprocess (__init__:434) 2019-04-13 21:40:12,526+0200 INFO (vm/b5713713) [vdsm.api] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) from=internal, task_id=6d835c86-ef85-4c83-9056-cfbd1b046191 (api:52) 2019-04-13 21:40:12,526+0200 ERROR (vm/b5713713) [storage.TaskManager.Task] (Task='6d835c86-ef85-4c83-9056-cfbd1b046191') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getVolumeSize File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3059, in getVolumeSize dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) 2019-04-13 21:40:12,528+0200 INFO (vm/b5713713) [storage.TaskManager.Task] (Task='6d835c86-ef85-4c83-9056-cfbd1b046191') aborting: Task is aborted: "Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',)" - code 358 (task:1181) 2019-04-13 21:40:12,529+0200 ERROR (vm/b5713713) [storage.Dispatcher] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) (dispatcher:83) 2019-04-13 21:40:12,542+0200 INFO (vm/b5713713) [virt.vm] (vmId='b5713713-ac04-4a91-95b8-3c3f17eba32f') Changed state to Down: VM terminated with error (code=1) (vm:1675) 2019-04-13 21:40:12,542+0200 INFO (vm/28d7d5ff) [virt.vm] (vmId='28d7d5ff-2ca1-42c6-ae8d-065c721d3066') VM wrapper has started (vm:2746) 2019-04-13 21:40:12,547+0200 INFO (vm/28d7d5ff) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID='123e2cc8-8f24-4ddd-b401-68393535f555', options=None) from=internal, task_id=7c3a34be-7bd9-4f6b-ab84-741ad360d10e (api:48) 2019-04-13 21:40:12,555+0200 INFO (vm/b5713713) [virt.vm] (vmId='b5713713-ac04-4a91-95b8-3c3f17eba32f') Stopping connection (guestagent:455) 2019-04-13 21:40:12,556+0200 WARN (vm/b5713713) [vds] Not ready yet, ignoring event '|virt|VM_status|b5713713-ac04-4a91-95b8-3c3f17eba32f' args={'b5713713-ac04-4a91-95b8-3c3f17eba32f': {'status': 'Down', 'exitMessage': 'VM terminated with error', 'vmId': 'b5713713-ac04-4a91-95b8-3c3f17eba32f', 'exitReason': 1, 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:12,556+0200 WARN (vm/b5713713) [vds] Not ready yet, ignoring event '|virt|VM_status|b5713713-ac04-4a91-95b8-3c3f17eba32f' args={'b5713713-ac04-4a91-95b8-3c3f17eba32f': {'status': 'Down', 'displayInfo': [{'tlsPort': '-1', 'ipAddress': '0', 'type': 'vnc', 'port': '-1'}], 'hash': '8186054922530936182', 'exitMessage': 'VM terminated with error', 'cpuUser': '0.00', 'monitorResponse': '0', 'vmId': 'b5713713-ac04-4a91-95b8-3c3f17eba32f', 'exitReason': 1, 'cpuUsage': '0.00', 'elapsedTime': '1505231', 'cpuSys': '0.00', 'timeOffset': '0', 'clientIp': '', 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:12,558+0200 INFO (vmrecovery) [vds] VM recovered: 611486ee-b72f-4acc-8925-4d7b6df9af1d (recovery:97) 2019-04-13 21:40:12,559+0200 INFO (vmrecovery) [vds] recovery [1:12/16]: recovered domain 611486ee-b72f-4acc-8925-4d7b6df9af1d (recovery:123) 2019-04-13 21:40:12,562+0200 INFO (hsm/init) [storage.HSM] FINISH HSM init succeeded in 1.21 seconds (hsm:410) 2019-04-13 21:40:12,589+0200 INFO (vmrecovery) [vds] VM recovered: 40eaf7e2-659c-4790-8362-49fe12c5f61a (recovery:97) 2019-04-13 21:40:12,589+0200 INFO (vmrecovery) [vds] recovery [1:13/16]: recovered domain 40eaf7e2-659c-4790-8362-49fe12c5f61a (recovery:123) 2019-04-13 21:40:12,614+0200 INFO (vmrecovery) [vds] VM recovered: 139534f6-1741-4ded-8ea6-ea4e3ecdad1f (recovery:97) 2019-04-13 21:40:12,614+0200 INFO (vmrecovery) [vds] recovery [1:14/16]: recovered domain 139534f6-1741-4ded-8ea6-ea4e3ecdad1f (recovery:123) 2019-04-13 21:40:12,640+0200 INFO (vmrecovery) [vds] VM recovered: be42701b-3f74-4c81-b6ef-5f7e7806554a (recovery:97) 2019-04-13 21:40:12,641+0200 INFO (vmrecovery) [vds] recovery [1:15/16]: recovered domain be42701b-3f74-4c81-b6ef-5f7e7806554a (recovery:123) 2019-04-13 21:40:12,666+0200 INFO (vmrecovery) [vds] VM recovered: 5dd5db29-143f-4d70-9469-4aaa163b7f9d (recovery:97) 2019-04-13 21:40:12,666+0200 INFO (vmrecovery) [vds] recovery [1:16/16]: recovered domain 5dd5db29-143f-4d70-9469-4aaa163b7f9d (recovery:123) 2019-04-13 21:40:12,667+0200 INFO (vmrecovery) [vds] recovery: waiting for 15 domains to go up (clientIF:703) 2019-04-13 21:40:12,708+0200 WARN (vm/0f8aff0a) [storage.LVM] Reloading VGs failed (vgs=['9c3f06cf-7475-448e-819b-f4f52fa7d782'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782']) (lvm:442) 2019-04-13 21:40:12,714+0200 INFO (vm/0f8aff0a) [vdsm.api] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) from=internal, task_id=9eab2e4f-d237-40e3-ac88-d3ca77c71337 (api:52) 2019-04-13 21:40:12,714+0200 ERROR (vm/0f8aff0a) [storage.TaskManager.Task] (Task='9eab2e4f-d237-40e3-ac88-d3ca77c71337') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getVolumeSize File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3059, in getVolumeSize dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) 2019-04-13 21:40:12,715+0200 INFO (vm/0f8aff0a) [storage.TaskManager.Task] (Task='9eab2e4f-d237-40e3-ac88-d3ca77c71337') aborting: Task is aborted: "Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',)" - code 358 (task:1181) 2019-04-13 21:40:12,715+0200 ERROR (vm/0f8aff0a) [storage.Dispatcher] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) (dispatcher:83) 2019-04-13 21:40:12,725+0200 INFO (vm/0f8aff0a) [virt.vm] (vmId='0f8aff0a-7915-4214-a37c-c9fee6518869') Changed state to Down: VM terminated with error (code=1) (vm:1675) 2019-04-13 21:40:12,728+0200 INFO (vm/dd24e11f) [virt.vm] (vmId='dd24e11f-40aa-47d2-9f0b-11cd7e83649d') VM wrapper has started (vm:2746) 2019-04-13 21:40:12,730+0200 INFO (vm/dd24e11f) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', options=None) from=internal, task_id=2e46058e-69b2-4132-8fcd-ee3869876dfd (api:48) 2019-04-13 21:40:12,741+0200 INFO (vm/0f8aff0a) [virt.vm] (vmId='0f8aff0a-7915-4214-a37c-c9fee6518869') Stopping connection (guestagent:455) 2019-04-13 21:40:12,742+0200 WARN (vm/0f8aff0a) [vds] Not ready yet, ignoring event '|virt|VM_status|0f8aff0a-7915-4214-a37c-c9fee6518869' args={'0f8aff0a-7915-4214-a37c-c9fee6518869': {'status': 'Down', 'exitMessage': 'VM terminated with error', 'vmId': '0f8aff0a-7915-4214-a37c-c9fee6518869', 'exitReason': 1, 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:12,743+0200 WARN (vm/0f8aff0a) [vds] Not ready yet, ignoring event '|virt|VM_status|0f8aff0a-7915-4214-a37c-c9fee6518869' args={'0f8aff0a-7915-4214-a37c-c9fee6518869': {'status': 'Down', 'displayInfo': [{'tlsPort': '-1', 'ipAddress': '0', 'type': 'vnc', 'port': '-1'}], 'hash': '-6342556332583565497', 'exitMessage': 'VM terminated with error', 'cpuUser': '0.00', 'monitorResponse': '0', 'vmId': '0f8aff0a-7915-4214-a37c-c9fee6518869', 'exitReason': 1, 'cpuUsage': '0.00', 'elapsedTime': '1727586', 'cpuSys': '0.00', 'timeOffset': '0', 'clientIp': '', 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:12,869+0200 WARN (vm/c8df4609) [storage.LVM] Reloading VGs failed (vgs=['9c3f06cf-7475-448e-819b-f4f52fa7d782'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782']) (lvm:442) 2019-04-13 21:40:12,875+0200 INFO (vm/c8df4609) [vdsm.api] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) from=internal, task_id=2efcded7-b885-4814-ad9b-f26e692d1bcf (api:52) 2019-04-13 21:40:12,875+0200 ERROR (vm/c8df4609) [storage.TaskManager.Task] (Task='2efcded7-b885-4814-ad9b-f26e692d1bcf') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getVolumeSize File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3059, in getVolumeSize dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) 2019-04-13 21:40:12,876+0200 INFO (vm/c8df4609) [storage.TaskManager.Task] (Task='2efcded7-b885-4814-ad9b-f26e692d1bcf') aborting: Task is aborted: "Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',)" - code 358 (task:1181) 2019-04-13 21:40:12,876+0200 ERROR (vm/c8df4609) [storage.Dispatcher] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) (dispatcher:83) 2019-04-13 21:40:12,886+0200 INFO (vm/c8df4609) [virt.vm] (vmId='c8df4609-fe26-4b73-9137-57724aff3ca4') Changed state to Down: VM terminated with error (code=1) (vm:1675) 2019-04-13 21:40:12,886+0200 INFO (vm/cb6ca3d9) [virt.vm] (vmId='cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6') VM wrapper has started (vm:2746) 2019-04-13 21:40:12,891+0200 INFO (vm/cb6ca3d9) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=7389bcbb-cd87-45a7-acda-055a574f78b7 (api:48) 2019-04-13 21:40:12,903+0200 INFO (vm/c8df4609) [virt.vm] (vmId='c8df4609-fe26-4b73-9137-57724aff3ca4') Stopping connection (guestagent:455) 2019-04-13 21:40:12,904+0200 WARN (vm/c8df4609) [vds] Not ready yet, ignoring event '|virt|VM_status|c8df4609-fe26-4b73-9137-57724aff3ca4' args={'c8df4609-fe26-4b73-9137-57724aff3ca4': {'status': 'Down', 'exitMessage': 'VM terminated with error', 'vmId': 'c8df4609-fe26-4b73-9137-57724aff3ca4', 'exitReason': 1, 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:12,905+0200 WARN (vm/c8df4609) [vds] Not ready yet, ignoring event '|virt|VM_status|c8df4609-fe26-4b73-9137-57724aff3ca4' args={'c8df4609-fe26-4b73-9137-57724aff3ca4': {'status': 'Down', 'displayInfo': [{'tlsPort': '-1', 'ipAddress': '0', 'type': 'vnc', 'port': '-1'}], 'hash': '-5816284415883541435', 'exitMessage': 'VM terminated with error', 'cpuUser': '0.00', 'monitorResponse': '0', 'vmId': 'c8df4609-fe26-4b73-9137-57724aff3ca4', 'exitReason': 1, 'cpuUsage': '0.00', 'elapsedTime': '1727587', 'cpuSys': '0.00', 'timeOffset': '0', 'clientIp': '', 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:13,029+0200 WARN (vm/a40d738c) [storage.LVM] Reloading VGs failed (vgs=['9c3f06cf-7475-448e-819b-f4f52fa7d782'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782']) (lvm:442) 2019-04-13 21:40:13,035+0200 INFO (vm/a40d738c) [vdsm.api] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) from=internal, task_id=4c73bac8-09e2-4f4c-9e4f-6235251c17e3 (api:52) 2019-04-13 21:40:13,035+0200 ERROR (vm/a40d738c) [storage.TaskManager.Task] (Task='4c73bac8-09e2-4f4c-9e4f-6235251c17e3') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getVolumeSize File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3059, in getVolumeSize dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) 2019-04-13 21:40:13,036+0200 INFO (vm/a40d738c) [storage.TaskManager.Task] (Task='4c73bac8-09e2-4f4c-9e4f-6235251c17e3') aborting: Task is aborted: "Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',)" - code 358 (task:1181) 2019-04-13 21:40:13,036+0200 ERROR (vm/a40d738c) [storage.Dispatcher] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) (dispatcher:83) 2019-04-13 21:40:13,049+0200 INFO (vm/a40d738c) [virt.vm] (vmId='a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3') Changed state to Down: VM terminated with error (code=1) (vm:1675) 2019-04-13 21:40:13,048+0200 INFO (vm/afa2c17f) [virt.vm] (vmId='afa2c17f-39e4-4168-aad7-2548dd19cc26') VM wrapper has started (vm:2746) 2019-04-13 21:40:13,053+0200 INFO (vm/afa2c17f) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID='2b136ed7-8b9a-45c1-9edc-64ee96c87c43', options=None) from=internal, task_id=12401873-7fb9-4872-9cb7-b356debd911b (api:48) 2019-04-13 21:40:13,068+0200 INFO (vm/a40d738c) [virt.vm] (vmId='a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3') Stopping connection (guestagent:455) 2019-04-13 21:40:13,069+0200 WARN (vm/a40d738c) [vds] Not ready yet, ignoring event '|virt|VM_status|a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3' args={'a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3': {'status': 'Down', 'exitMessage': 'VM terminated with error', 'vmId': 'a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3', 'exitReason': 1, 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:13,069+0200 WARN (vm/a40d738c) [vds] Not ready yet, ignoring event '|virt|VM_status|a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3' args={'a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3': {'status': 'Down', 'displayInfo': [{'tlsPort': '-1', 'ipAddress': '0', 'type': 'vnc', 'port': '-1'}], 'hash': '8190221830248719195', 'exitMessage': 'VM terminated with error', 'cpuUser': '0.00', 'monitorResponse': '0', 'vmId': 'a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3', 'exitReason': 1, 'cpuUsage': '0.00', 'elapsedTime': '1727586', 'cpuSys': '0.00', 'timeOffset': '0', 'clientIp': '', 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:13,194+0200 WARN (vm/28d7d5ff) [storage.LVM] Reloading VGs failed (vgs=['9c3f06cf-7475-448e-819b-f4f52fa7d782'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782']) (lvm:442) 2019-04-13 21:40:13,200+0200 INFO (vm/28d7d5ff) [vdsm.api] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) from=internal, task_id=7c3a34be-7bd9-4f6b-ab84-741ad360d10e (api:52) 2019-04-13 21:40:13,200+0200 ERROR (vm/28d7d5ff) [storage.TaskManager.Task] (Task='7c3a34be-7bd9-4f6b-ab84-741ad360d10e') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getVolumeSize File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3059, in getVolumeSize dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) 2019-04-13 21:40:13,201+0200 INFO (vm/28d7d5ff) [storage.TaskManager.Task] (Task='7c3a34be-7bd9-4f6b-ab84-741ad360d10e') aborting: Task is aborted: "Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',)" - code 358 (task:1181) 2019-04-13 21:40:13,201+0200 ERROR (vm/28d7d5ff) [storage.Dispatcher] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) (dispatcher:83) 2019-04-13 21:40:13,253+0200 INFO (vm/5a326302) [virt.vm] (vmId='5a326302-86c1-41b5-ba58-793232299713') VM wrapper has started (vm:2746) 2019-04-13 21:40:13,254+0200 INFO (vm/28d7d5ff) [virt.vm] (vmId='28d7d5ff-2ca1-42c6-ae8d-065c721d3066') Changed state to Down: VM terminated with error (code=1) (vm:1675) 2019-04-13 21:40:13,257+0200 INFO (vm/5a326302) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', options=None) from=internal, task_id=e9e4d2f3-7cec-4b04-b9fd-f441d2cf515e (api:48) 2019-04-13 21:40:13,270+0200 INFO (vm/28d7d5ff) [virt.vm] (vmId='28d7d5ff-2ca1-42c6-ae8d-065c721d3066') Stopping connection (guestagent:455) 2019-04-13 21:40:13,270+0200 WARN (vm/28d7d5ff) [vds] Not ready yet, ignoring event '|virt|VM_status|28d7d5ff-2ca1-42c6-ae8d-065c721d3066' args={'28d7d5ff-2ca1-42c6-ae8d-065c721d3066': {'status': 'Down', 'exitMessage': 'VM terminated with error', 'vmId': '28d7d5ff-2ca1-42c6-ae8d-065c721d3066', 'exitReason': 1, 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:13,271+0200 WARN (vm/28d7d5ff) [vds] Not ready yet, ignoring event '|virt|VM_status|28d7d5ff-2ca1-42c6-ae8d-065c721d3066' args={'28d7d5ff-2ca1-42c6-ae8d-065c721d3066': {'status': 'Down', 'displayInfo': [{'tlsPort': '-1', 'ipAddress': '0', 'type': 'vnc', 'port': '-1'}], 'hash': '-7582957048910546443', 'exitMessage': 'VM terminated with error', 'cpuUser': '0.00', 'monitorResponse': '0', 'vmId': '28d7d5ff-2ca1-42c6-ae8d-065c721d3066', 'exitReason': 1, 'cpuUsage': '0.00', 'elapsedTime': '1727535', 'cpuSys': '0.00', 'timeOffset': '0', 'clientIp': '', 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:13,399+0200 WARN (vm/dd24e11f) [storage.LVM] Reloading VGs failed (vgs=['9c3f06cf-7475-448e-819b-f4f52fa7d782'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782']) (lvm:442) 2019-04-13 21:40:13,404+0200 INFO (vm/dd24e11f) [vdsm.api] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) from=internal, task_id=2e46058e-69b2-4132-8fcd-ee3869876dfd (api:52) 2019-04-13 21:40:13,405+0200 ERROR (vm/dd24e11f) [storage.TaskManager.Task] (Task='2e46058e-69b2-4132-8fcd-ee3869876dfd') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getVolumeSize File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3059, in getVolumeSize dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) 2019-04-13 21:40:13,405+0200 INFO (vm/dd24e11f) [storage.TaskManager.Task] (Task='2e46058e-69b2-4132-8fcd-ee3869876dfd') aborting: Task is aborted: "Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',)" - code 358 (task:1181) 2019-04-13 21:40:13,406+0200 ERROR (vm/dd24e11f) [storage.Dispatcher] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) (dispatcher:83) 2019-04-13 21:40:13,415+0200 INFO (vm/dd24e11f) [virt.vm] (vmId='dd24e11f-40aa-47d2-9f0b-11cd7e83649d') Changed state to Down: VM terminated with error (code=1) (vm:1675) 2019-04-13 21:40:13,415+0200 INFO (vm/6772f848) [virt.vm] (vmId='6772f848-4874-4785-9bb0-88d40bfb71c4') VM wrapper has started (vm:2746) 2019-04-13 21:40:13,420+0200 INFO (vm/6772f848) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', options=None) from=internal, task_id=aa183bcd-9dac-47e7-9cc4-cdff924d4339 (api:48) 2019-04-13 21:40:13,432+0200 INFO (vm/dd24e11f) [virt.vm] (vmId='dd24e11f-40aa-47d2-9f0b-11cd7e83649d') Stopping connection (guestagent:455) 2019-04-13 21:40:13,432+0200 WARN (vm/dd24e11f) [vds] Not ready yet, ignoring event '|virt|VM_status|dd24e11f-40aa-47d2-9f0b-11cd7e83649d' args={'dd24e11f-40aa-47d2-9f0b-11cd7e83649d': {'status': 'Down', 'exitMessage': 'VM terminated with error', 'vmId': 'dd24e11f-40aa-47d2-9f0b-11cd7e83649d', 'exitReason': 1, 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:13,433+0200 WARN (vm/dd24e11f) [vds] Not ready yet, ignoring event '|virt|VM_status|dd24e11f-40aa-47d2-9f0b-11cd7e83649d' args={'dd24e11f-40aa-47d2-9f0b-11cd7e83649d': {'status': 'Down', 'displayInfo': [{'tlsPort': '-1', 'ipAddress': '0', 'type': 'vnc', 'port': '-1'}], 'hash': '-5185357934340142596', 'exitMessage': 'VM terminated with error', 'cpuUser': '0.00', 'monitorResponse': '0', 'vmId': 'dd24e11f-40aa-47d2-9f0b-11cd7e83649d', 'exitReason': 1, 'cpuUsage': '0.00', 'elapsedTime': '1726560', 'cpuSys': '0.00', 'timeOffset': '0', 'clientIp': '', 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:13,562+0200 WARN (vm/cb6ca3d9) [storage.LVM] Reloading VGs failed (vgs=['9c3f06cf-7475-448e-819b-f4f52fa7d782'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782']) (lvm:442) 2019-04-13 21:40:13,567+0200 INFO (vm/cb6ca3d9) [vdsm.api] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) from=internal, task_id=7389bcbb-cd87-45a7-acda-055a574f78b7 (api:52) 2019-04-13 21:40:13,567+0200 ERROR (vm/cb6ca3d9) [storage.TaskManager.Task] (Task='7389bcbb-cd87-45a7-acda-055a574f78b7') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getVolumeSize File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3059, in getVolumeSize dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) 2019-04-13 21:40:13,568+0200 INFO (vm/cb6ca3d9) [storage.TaskManager.Task] (Task='7389bcbb-cd87-45a7-acda-055a574f78b7') aborting: Task is aborted: "Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',)" - code 358 (task:1181) 2019-04-13 21:40:13,568+0200 ERROR (vm/cb6ca3d9) [storage.Dispatcher] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) (dispatcher:83) 2019-04-13 21:40:13,577+0200 INFO (vm/cb6ca3d9) [virt.vm] (vmId='cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6') Changed state to Down: VM terminated with error (code=1) (vm:1675) 2019-04-13 21:40:13,576+0200 INFO (vm/7b5e0fdd) [virt.vm] (vmId='7b5e0fdd-8757-49ce-acf9-3206a372e1c9') VM wrapper has started (vm:2746) 2019-04-13 21:40:13,582+0200 INFO (vm/7b5e0fdd) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID='3f6c31c6-ec38-4979-a617-55537123ee61', options=None) from=internal, task_id=f6647767-9c19-4a15-8a86-78f38013d882 (api:48) 2019-04-13 21:40:13,595+0200 INFO (vm/cb6ca3d9) [virt.vm] (vmId='cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6') Stopping connection (guestagent:455) 2019-04-13 21:40:13,596+0200 WARN (vm/cb6ca3d9) [vds] Not ready yet, ignoring event '|virt|VM_status|cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6' args={'cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6': {'status': 'Down', 'exitMessage': 'VM terminated with error', 'vmId': 'cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6', 'exitReason': 1, 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:13,597+0200 WARN (vm/cb6ca3d9) [vds] Not ready yet, ignoring event '|virt|VM_status|cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6' args={'cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6': {'status': 'Down', 'displayInfo': [{'tlsPort': '-1', 'ipAddress': '0', 'type': 'vnc', 'port': '-1'}], 'hash': '762232370642670139', 'exitMessage': 'VM terminated with error', 'cpuUser': '0.00', 'monitorResponse': '0', 'vmId': 'cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6', 'exitReason': 1, 'cpuUsage': '0.00', 'elapsedTime': '1727513', 'cpuSys': '0.00', 'timeOffset': '0', 'clientIp': '', 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:13,668+0200 INFO (vmrecovery) [vds] recovery: waiting for 9 domains to go up (clientIF:703) 2019-04-13 21:40:13,722+0200 WARN (vm/afa2c17f) [storage.LVM] Reloading VGs failed (vgs=['9c3f06cf-7475-448e-819b-f4f52fa7d782'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782']) (lvm:442) 2019-04-13 21:40:13,728+0200 INFO (vm/afa2c17f) [vdsm.api] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) from=internal, task_id=12401873-7fb9-4872-9cb7-b356debd911b (api:52) 2019-04-13 21:40:13,728+0200 ERROR (vm/afa2c17f) [storage.TaskManager.Task] (Task='12401873-7fb9-4872-9cb7-b356debd911b') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getVolumeSize File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3059, in getVolumeSize dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) 2019-04-13 21:40:13,729+0200 INFO (vm/afa2c17f) [storage.TaskManager.Task] (Task='12401873-7fb9-4872-9cb7-b356debd911b') aborting: Task is aborted: "Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',)" - code 358 (task:1181) 2019-04-13 21:40:13,729+0200 ERROR (vm/afa2c17f) [storage.Dispatcher] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) (dispatcher:83) 2019-04-13 21:40:13,738+0200 INFO (vm/afa2c17f) [virt.vm] (vmId='afa2c17f-39e4-4168-aad7-2548dd19cc26') Changed state to Down: VM terminated with error (code=1) (vm:1675) 2019-04-13 21:40:13,738+0200 INFO (vm/611486ee) [virt.vm] (vmId='611486ee-b72f-4acc-8925-4d7b6df9af1d') VM wrapper has started (vm:2746) 2019-04-13 21:40:13,744+0200 INFO (vm/611486ee) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID='a884b4ce-381f-4653-be14-a31d15181f48', options=None) from=internal, task_id=12a0bf0b-13a2-40d8-abea-ff148885b773 (api:48) 2019-04-13 21:40:13,757+0200 INFO (vm/afa2c17f) [virt.vm] (vmId='afa2c17f-39e4-4168-aad7-2548dd19cc26') Stopping connection (guestagent:455) 2019-04-13 21:40:13,758+0200 WARN (vm/afa2c17f) [vds] Not ready yet, ignoring event '|virt|VM_status|afa2c17f-39e4-4168-aad7-2548dd19cc26' args={'afa2c17f-39e4-4168-aad7-2548dd19cc26': {'status': 'Down', 'exitMessage': 'VM terminated with error', 'vmId': 'afa2c17f-39e4-4168-aad7-2548dd19cc26', 'exitReason': 1, 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:13,758+0200 WARN (vm/afa2c17f) [vds] Not ready yet, ignoring event '|virt|VM_status|afa2c17f-39e4-4168-aad7-2548dd19cc26' args={'afa2c17f-39e4-4168-aad7-2548dd19cc26': {'status': 'Down', 'displayInfo': [{'tlsPort': '-1', 'ipAddress': '0', 'type': 'vnc', 'port': '-1'}], 'hash': '6376216043112237238', 'exitMessage': 'VM terminated with error', 'cpuUser': '0.00', 'monitorResponse': '0', 'vmId': 'afa2c17f-39e4-4168-aad7-2548dd19cc26', 'exitReason': 1, 'cpuUsage': '0.00', 'elapsedTime': '1727536', 'cpuSys': '0.00', 'timeOffset': '0', 'clientIp': '', 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:13,884+0200 WARN (vm/5a326302) [storage.LVM] Reloading VGs failed (vgs=['9c3f06cf-7475-448e-819b-f4f52fa7d782'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782']) (lvm:442) 2019-04-13 21:40:13,890+0200 INFO (vm/5a326302) [vdsm.api] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) from=internal, task_id=e9e4d2f3-7cec-4b04-b9fd-f441d2cf515e (api:52) 2019-04-13 21:40:13,890+0200 ERROR (vm/5a326302) [storage.TaskManager.Task] (Task='e9e4d2f3-7cec-4b04-b9fd-f441d2cf515e') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getVolumeSize File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3059, in getVolumeSize dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) 2019-04-13 21:40:13,890+0200 INFO (vm/5a326302) [storage.TaskManager.Task] (Task='e9e4d2f3-7cec-4b04-b9fd-f441d2cf515e') aborting: Task is aborted: "Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',)" - code 358 (task:1181) 2019-04-13 21:40:13,890+0200 ERROR (vm/5a326302) [storage.Dispatcher] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) (dispatcher:83) 2019-04-13 21:40:13,899+0200 INFO (vm/5a326302) [virt.vm] (vmId='5a326302-86c1-41b5-ba58-793232299713') Changed state to Down: VM terminated with error (code=1) (vm:1675) 2019-04-13 21:40:13,902+0200 INFO (vm/40eaf7e2) [virt.vm] (vmId='40eaf7e2-659c-4790-8362-49fe12c5f61a') VM wrapper has started (vm:2746) 2019-04-13 21:40:13,905+0200 INFO (vm/40eaf7e2) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', options=None) from=internal, task_id=1339877f-9188-4b3c-8783-538677fda539 (api:48) 2019-04-13 21:40:13,919+0200 INFO (vm/5a326302) [virt.vm] (vmId='5a326302-86c1-41b5-ba58-793232299713') Stopping connection (guestagent:455) 2019-04-13 21:40:13,919+0200 WARN (vm/5a326302) [vds] Not ready yet, ignoring event '|virt|VM_status|5a326302-86c1-41b5-ba58-793232299713' args={'5a326302-86c1-41b5-ba58-793232299713': {'status': 'Down', 'exitMessage': 'VM terminated with error', 'vmId': '5a326302-86c1-41b5-ba58-793232299713', 'exitReason': 1, 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:13,920+0200 WARN (vm/5a326302) [vds] Not ready yet, ignoring event '|virt|VM_status|5a326302-86c1-41b5-ba58-793232299713' args={'5a326302-86c1-41b5-ba58-793232299713': {'status': 'Down', 'displayInfo': [{'tlsPort': '-1', 'ipAddress': '0', 'type': 'vnc', 'port': '-1'}], 'hash': '-6532966778361301246', 'exitMessage': 'VM terminated with error', 'cpuUser': '0.00', 'monitorResponse': '0', 'vmId': '5a326302-86c1-41b5-ba58-793232299713', 'exitReason': 1, 'cpuUsage': '0.00', 'elapsedTime': '1727587', 'cpuSys': '0.00', 'timeOffset': '0', 'clientIp': '', 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:13,952+0200 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::ffff:192.168.0.19:51130 (protocoldetector:61) 2019-04-13 21:40:14,045+0200 WARN (vm/6772f848) [storage.LVM] Reloading VGs failed (vgs=['9c3f06cf-7475-448e-819b-f4f52fa7d782'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782']) (lvm:442) 2019-04-13 21:40:14,051+0200 INFO (vm/6772f848) [vdsm.api] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) from=internal, task_id=aa183bcd-9dac-47e7-9cc4-cdff924d4339 (api:52) 2019-04-13 21:40:14,051+0200 ERROR (vm/6772f848) [storage.TaskManager.Task] (Task='aa183bcd-9dac-47e7-9cc4-cdff924d4339') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getVolumeSize File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3059, in getVolumeSize dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) 2019-04-13 21:40:14,051+0200 INFO (vm/6772f848) [storage.TaskManager.Task] (Task='aa183bcd-9dac-47e7-9cc4-cdff924d4339') aborting: Task is aborted: "Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',)" - code 358 (task:1181) 2019-04-13 21:40:14,051+0200 ERROR (vm/6772f848) [storage.Dispatcher] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) (dispatcher:83) 2019-04-13 21:40:14,064+0200 INFO (vm/139534f6) [virt.vm] (vmId='139534f6-1741-4ded-8ea6-ea4e3ecdad1f') VM wrapper has started (vm:2746) 2019-04-13 21:40:14,066+0200 INFO (vm/139534f6) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', options=None) from=internal, task_id=47fbb832-751f-47bd-9a83-7834fffffb5e (api:48) 2019-04-13 21:40:14,067+0200 INFO (vm/6772f848) [virt.vm] (vmId='6772f848-4874-4785-9bb0-88d40bfb71c4') Changed state to Down: VM terminated with error (code=1) (vm:1675) 2019-04-13 21:40:14,070+0200 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::ffff:192.168.0.19:51130 (protocoldetector:125) 2019-04-13 21:40:14,070+0200 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompserver:95) 2019-04-13 21:40:14,071+0200 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompserver:124) 2019-04-13 21:40:14,072+0200 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompserver:124) 2019-04-13 21:40:14,073+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] In recovery, ignoring 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'59ef3a18-002f-02d1-0220-000000000124'} (__init__:324) 2019-04-13 21:40:14,074+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 99) in 0.00 seconds (__init__:312) 2019-04-13 21:40:14,074+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] In recovery, ignoring 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'59ef3a18-002f-02d1-0220-000000000124'} (__init__:324) 2019-04-13 21:40:14,074+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 99) in 0.00 seconds (__init__:312) 2019-04-13 21:40:14,074+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] In recovery, ignoring 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'59ef3a18-002f-02d1-0220-000000000124'} (__init__:324) 2019-04-13 21:40:14,074+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 99) in 0.00 seconds (__init__:312) 2019-04-13 21:40:14,077+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] In recovery, ignoring 'Host.getCapabilities' in bridge with {} (__init__:324) 2019-04-13 21:40:14,077+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getCapabilities failed (error 99) in 0.00 seconds (__init__:312) 2019-04-13 21:40:14,087+0200 INFO (vm/6772f848) [virt.vm] (vmId='6772f848-4874-4785-9bb0-88d40bfb71c4') Stopping connection (guestagent:455) 2019-04-13 21:40:14,087+0200 WARN (vm/6772f848) [vds] Not ready yet, ignoring event '|virt|VM_status|6772f848-4874-4785-9bb0-88d40bfb71c4' args={'6772f848-4874-4785-9bb0-88d40bfb71c4': {'status': 'Down', 'exitMessage': 'VM terminated with error', 'vmId': '6772f848-4874-4785-9bb0-88d40bfb71c4', 'exitReason': 1, 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:14,088+0200 WARN (vm/6772f848) [vds] Not ready yet, ignoring event '|virt|VM_status|6772f848-4874-4785-9bb0-88d40bfb71c4' args={'6772f848-4874-4785-9bb0-88d40bfb71c4': {'status': 'Down', 'displayInfo': [{'tlsPort': '-1', 'ipAddress': '0', 'type': 'vnc', 'port': '-1'}], 'hash': '-2105251786720994683', 'exitMessage': 'VM terminated with error', 'cpuUser': '0.00', 'monitorResponse': '0', 'vmId': '6772f848-4874-4785-9bb0-88d40bfb71c4', 'exitReason': 1, 'cpuUsage': '0.00', 'elapsedTime': '1727536', 'cpuSys': '0.00', 'timeOffset': '0', 'clientIp': '', 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:14,131+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] In recovery, ignoring 'Host.getAllTasksStatuses' in bridge with {} (__init__:324) 2019-04-13 21:40:14,131+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksStatuses failed (error 99) in 0.00 seconds (__init__:312) 2019-04-13 21:40:14,217+0200 WARN (vm/7b5e0fdd) [storage.LVM] Reloading VGs failed (vgs=['9c3f06cf-7475-448e-819b-f4f52fa7d782'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782']) (lvm:442) 2019-04-13 21:40:14,223+0200 INFO (vm/7b5e0fdd) [vdsm.api] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) from=internal, task_id=f6647767-9c19-4a15-8a86-78f38013d882 (api:52) 2019-04-13 21:40:14,223+0200 ERROR (vm/7b5e0fdd) [storage.TaskManager.Task] (Task='f6647767-9c19-4a15-8a86-78f38013d882') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getVolumeSize File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3059, in getVolumeSize dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) 2019-04-13 21:40:14,223+0200 INFO (vm/7b5e0fdd) [storage.TaskManager.Task] (Task='f6647767-9c19-4a15-8a86-78f38013d882') aborting: Task is aborted: "Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',)" - code 358 (task:1181) 2019-04-13 21:40:14,223+0200 ERROR (vm/7b5e0fdd) [storage.Dispatcher] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) (dispatcher:83) 2019-04-13 21:40:14,234+0200 INFO (vm/be42701b) [virt.vm] (vmId='be42701b-3f74-4c81-b6ef-5f7e7806554a') VM wrapper has started (vm:2746) 2019-04-13 21:40:14,236+0200 INFO (vm/be42701b) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID='ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', options=None) from=internal, task_id=1e2c09b8-7f41-48ff-8510-e7cdb8c5d7d7 (api:48) 2019-04-13 21:40:14,237+0200 INFO (vm/7b5e0fdd) [virt.vm] (vmId='7b5e0fdd-8757-49ce-acf9-3206a372e1c9') Changed state to Down: VM terminated with error (code=1) (vm:1675) 2019-04-13 21:40:14,255+0200 INFO (vm/7b5e0fdd) [virt.vm] (vmId='7b5e0fdd-8757-49ce-acf9-3206a372e1c9') Stopping connection (guestagent:455) 2019-04-13 21:40:14,255+0200 WARN (vm/7b5e0fdd) [vds] Not ready yet, ignoring event '|virt|VM_status|7b5e0fdd-8757-49ce-acf9-3206a372e1c9' args={'7b5e0fdd-8757-49ce-acf9-3206a372e1c9': {'status': 'Down', 'exitMessage': 'VM terminated with error', 'vmId': '7b5e0fdd-8757-49ce-acf9-3206a372e1c9', 'exitReason': 1, 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:14,256+0200 WARN (vm/7b5e0fdd) [vds] Not ready yet, ignoring event '|virt|VM_status|7b5e0fdd-8757-49ce-acf9-3206a372e1c9' args={'7b5e0fdd-8757-49ce-acf9-3206a372e1c9': {'status': 'Down', 'displayInfo': [{'tlsPort': '-1', 'ipAddress': '0', 'type': 'vnc', 'port': '-1'}], 'hash': '7762599171768731360', 'exitMessage': 'VM terminated with error', 'cpuUser': '0.00', 'monitorResponse': '0', 'vmId': '7b5e0fdd-8757-49ce-acf9-3206a372e1c9', 'exitReason': 1, 'cpuUsage': '0.00', 'elapsedTime': '1726194', 'cpuSys': '0.00', 'timeOffset': '0', 'clientIp': '', 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:14,379+0200 WARN (vm/611486ee) [storage.LVM] Reloading VGs failed (vgs=['9c3f06cf-7475-448e-819b-f4f52fa7d782'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782']) (lvm:442) 2019-04-13 21:40:14,385+0200 INFO (vm/611486ee) [vdsm.api] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) from=internal, task_id=12a0bf0b-13a2-40d8-abea-ff148885b773 (api:52) 2019-04-13 21:40:14,385+0200 ERROR (vm/611486ee) [storage.TaskManager.Task] (Task='12a0bf0b-13a2-40d8-abea-ff148885b773') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getVolumeSize File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3059, in getVolumeSize dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) 2019-04-13 21:40:14,385+0200 INFO (vm/611486ee) [storage.TaskManager.Task] (Task='12a0bf0b-13a2-40d8-abea-ff148885b773') aborting: Task is aborted: "Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',)" - code 358 (task:1181) 2019-04-13 21:40:14,385+0200 ERROR (vm/611486ee) [storage.Dispatcher] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) (dispatcher:83) 2019-04-13 21:40:14,396+0200 INFO (vm/5dd5db29) [virt.vm] (vmId='5dd5db29-143f-4d70-9469-4aaa163b7f9d') VM wrapper has started (vm:2746) 2019-04-13 21:40:14,398+0200 INFO (vm/5dd5db29) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID='67a4dd71-69ff-49a8-b8a4-c7101c75e105', options=None) from=internal, task_id=71b89391-16f3-42cc-bf2d-64a9dc9b575a (api:48) 2019-04-13 21:40:14,398+0200 INFO (vm/611486ee) [virt.vm] (vmId='611486ee-b72f-4acc-8925-4d7b6df9af1d') Changed state to Down: VM terminated with error (code=1) (vm:1675) 2019-04-13 21:40:14,411+0200 INFO (vm/611486ee) [virt.vm] (vmId='611486ee-b72f-4acc-8925-4d7b6df9af1d') Stopping connection (guestagent:455) 2019-04-13 21:40:14,411+0200 WARN (vm/611486ee) [vds] Not ready yet, ignoring event '|virt|VM_status|611486ee-b72f-4acc-8925-4d7b6df9af1d' args={'611486ee-b72f-4acc-8925-4d7b6df9af1d': {'status': 'Down', 'exitMessage': 'VM terminated with error', 'vmId': '611486ee-b72f-4acc-8925-4d7b6df9af1d', 'exitReason': 1, 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:14,412+0200 WARN (vm/611486ee) [vds] Not ready yet, ignoring event '|virt|VM_status|611486ee-b72f-4acc-8925-4d7b6df9af1d' args={'611486ee-b72f-4acc-8925-4d7b6df9af1d': {'status': 'Down', 'displayInfo': [{'tlsPort': '-1', 'ipAddress': '0', 'type': 'vnc', 'port': '-1'}], 'hash': '3591134256995562820', 'exitMessage': 'VM terminated with error', 'cpuUser': '0.00', 'monitorResponse': '0', 'vmId': '611486ee-b72f-4acc-8925-4d7b6df9af1d', 'exitReason': 1, 'cpuUsage': '0.00', 'elapsedTime': '1727588', 'cpuSys': '0.00', 'timeOffset': '0', 'clientIp': '', 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:14,539+0200 WARN (vm/40eaf7e2) [storage.LVM] Reloading VGs failed (vgs=['9c3f06cf-7475-448e-819b-f4f52fa7d782'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782']) (lvm:442) 2019-04-13 21:40:14,544+0200 INFO (vm/40eaf7e2) [vdsm.api] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) from=internal, task_id=1339877f-9188-4b3c-8783-538677fda539 (api:52) 2019-04-13 21:40:14,545+0200 ERROR (vm/40eaf7e2) [storage.TaskManager.Task] (Task='1339877f-9188-4b3c-8783-538677fda539') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getVolumeSize File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3059, in getVolumeSize dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) 2019-04-13 21:40:14,545+0200 INFO (vm/40eaf7e2) [storage.TaskManager.Task] (Task='1339877f-9188-4b3c-8783-538677fda539') aborting: Task is aborted: "Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',)" - code 358 (task:1181) 2019-04-13 21:40:14,545+0200 ERROR (vm/40eaf7e2) [storage.Dispatcher] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) (dispatcher:83) 2019-04-13 21:40:14,556+0200 INFO (vm/40eaf7e2) [virt.vm] (vmId='40eaf7e2-659c-4790-8362-49fe12c5f61a') Changed state to Down: VM terminated with error (code=1) (vm:1675) 2019-04-13 21:40:14,573+0200 INFO (vm/40eaf7e2) [virt.vm] (vmId='40eaf7e2-659c-4790-8362-49fe12c5f61a') Stopping connection (guestagent:455) 2019-04-13 21:40:14,574+0200 WARN (vm/40eaf7e2) [vds] Not ready yet, ignoring event '|virt|VM_status|40eaf7e2-659c-4790-8362-49fe12c5f61a' args={'40eaf7e2-659c-4790-8362-49fe12c5f61a': {'status': 'Down', 'exitMessage': 'VM terminated with error', 'vmId': '40eaf7e2-659c-4790-8362-49fe12c5f61a', 'exitReason': 1, 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:14,574+0200 WARN (vm/40eaf7e2) [vds] Not ready yet, ignoring event '|virt|VM_status|40eaf7e2-659c-4790-8362-49fe12c5f61a' args={'40eaf7e2-659c-4790-8362-49fe12c5f61a': {'status': 'Down', 'displayInfo': [{'tlsPort': '-1', 'ipAddress': '0', 'type': 'vnc', 'port': '-1'}], 'hash': '532753917418810717', 'exitMessage': 'VM terminated with error', 'cpuUser': '0.00', 'monitorResponse': '0', 'vmId': '40eaf7e2-659c-4790-8362-49fe12c5f61a', 'exitReason': 1, 'cpuUsage': '0.00', 'elapsedTime': '1727588', 'cpuSys': '0.00', 'timeOffset': '0', 'clientIp': '', 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:14,669+0200 INFO (vmrecovery) [vds] recovery: waiting for 3 domains to go up (clientIF:703) 2019-04-13 21:40:14,697+0200 WARN (vm/139534f6) [storage.LVM] Reloading VGs failed (vgs=['9c3f06cf-7475-448e-819b-f4f52fa7d782'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782']) (lvm:442) 2019-04-13 21:40:14,703+0200 INFO (vm/139534f6) [vdsm.api] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) from=internal, task_id=47fbb832-751f-47bd-9a83-7834fffffb5e (api:52) 2019-04-13 21:40:14,703+0200 ERROR (vm/139534f6) [storage.TaskManager.Task] (Task='47fbb832-751f-47bd-9a83-7834fffffb5e') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getVolumeSize File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3059, in getVolumeSize dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) 2019-04-13 21:40:14,703+0200 INFO (vm/139534f6) [storage.TaskManager.Task] (Task='47fbb832-751f-47bd-9a83-7834fffffb5e') aborting: Task is aborted: "Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',)" - code 358 (task:1181) 2019-04-13 21:40:14,703+0200 ERROR (vm/139534f6) [storage.Dispatcher] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) (dispatcher:83) 2019-04-13 21:40:14,714+0200 INFO (vm/139534f6) [virt.vm] (vmId='139534f6-1741-4ded-8ea6-ea4e3ecdad1f') Changed state to Down: VM terminated with error (code=1) (vm:1675) 2019-04-13 21:40:14,734+0200 INFO (vm/139534f6) [virt.vm] (vmId='139534f6-1741-4ded-8ea6-ea4e3ecdad1f') Stopping connection (guestagent:455) 2019-04-13 21:40:14,734+0200 WARN (vm/139534f6) [vds] Not ready yet, ignoring event '|virt|VM_status|139534f6-1741-4ded-8ea6-ea4e3ecdad1f' args={'139534f6-1741-4ded-8ea6-ea4e3ecdad1f': {'status': 'Down', 'exitMessage': 'VM terminated with error', 'vmId': '139534f6-1741-4ded-8ea6-ea4e3ecdad1f', 'exitReason': 1, 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:14,734+0200 WARN (vm/139534f6) [vds] Not ready yet, ignoring event '|virt|VM_status|139534f6-1741-4ded-8ea6-ea4e3ecdad1f' args={'139534f6-1741-4ded-8ea6-ea4e3ecdad1f': {'status': 'Down', 'displayInfo': [{'tlsPort': '-1', 'ipAddress': '0', 'type': 'vnc', 'port': '-1'}], 'hash': '-4692113255477588237', 'exitMessage': 'VM terminated with error', 'cpuUser': '0.00', 'monitorResponse': '0', 'vmId': '139534f6-1741-4ded-8ea6-ea4e3ecdad1f', 'exitReason': 1, 'cpuUsage': '0.00', 'elapsedTime': '1727589', 'cpuSys': '0.00', 'timeOffset': '0', 'clientIp': '', 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:14,854+0200 WARN (vm/be42701b) [storage.LVM] Reloading VGs failed (vgs=['9c3f06cf-7475-448e-819b-f4f52fa7d782'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782']) (lvm:442) 2019-04-13 21:40:14,860+0200 INFO (vm/be42701b) [vdsm.api] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) from=internal, task_id=1e2c09b8-7f41-48ff-8510-e7cdb8c5d7d7 (api:52) 2019-04-13 21:40:14,860+0200 ERROR (vm/be42701b) [storage.TaskManager.Task] (Task='1e2c09b8-7f41-48ff-8510-e7cdb8c5d7d7') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getVolumeSize File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3059, in getVolumeSize dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) 2019-04-13 21:40:14,860+0200 INFO (vm/be42701b) [storage.TaskManager.Task] (Task='1e2c09b8-7f41-48ff-8510-e7cdb8c5d7d7') aborting: Task is aborted: "Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',)" - code 358 (task:1181) 2019-04-13 21:40:14,861+0200 ERROR (vm/be42701b) [storage.Dispatcher] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) (dispatcher:83) 2019-04-13 21:40:14,872+0200 INFO (vm/be42701b) [virt.vm] (vmId='be42701b-3f74-4c81-b6ef-5f7e7806554a') Changed state to Down: VM terminated with error (code=1) (vm:1675) 2019-04-13 21:40:14,889+0200 INFO (vm/be42701b) [virt.vm] (vmId='be42701b-3f74-4c81-b6ef-5f7e7806554a') Stopping connection (guestagent:455) 2019-04-13 21:40:14,890+0200 WARN (vm/be42701b) [vds] Not ready yet, ignoring event '|virt|VM_status|be42701b-3f74-4c81-b6ef-5f7e7806554a' args={'be42701b-3f74-4c81-b6ef-5f7e7806554a': {'status': 'Down', 'exitMessage': 'VM terminated with error', 'vmId': 'be42701b-3f74-4c81-b6ef-5f7e7806554a', 'exitReason': 1, 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:14,890+0200 WARN (vm/be42701b) [vds] Not ready yet, ignoring event '|virt|VM_status|be42701b-3f74-4c81-b6ef-5f7e7806554a' args={'be42701b-3f74-4c81-b6ef-5f7e7806554a': {'status': 'Down', 'displayInfo': [{'tlsPort': '-1', 'ipAddress': '0', 'type': 'vnc', 'port': '-1'}], 'hash': '-6140325523410103396', 'exitMessage': 'VM terminated with error', 'cpuUser': '0.00', 'monitorResponse': '0', 'vmId': 'be42701b-3f74-4c81-b6ef-5f7e7806554a', 'exitReason': 1, 'cpuUsage': '0.00', 'elapsedTime': '1727685', 'cpuSys': '0.00', 'timeOffset': '0', 'clientIp': '', 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:15,019+0200 WARN (vm/5dd5db29) [storage.LVM] Reloading VGs failed (vgs=['9c3f06cf-7475-448e-819b-f4f52fa7d782'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782']) (lvm:442) 2019-04-13 21:40:15,025+0200 INFO (vm/5dd5db29) [vdsm.api] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) from=internal, task_id=71b89391-16f3-42cc-bf2d-64a9dc9b575a (api:52) 2019-04-13 21:40:15,025+0200 ERROR (vm/5dd5db29) [storage.TaskManager.Task] (Task='71b89391-16f3-42cc-bf2d-64a9dc9b575a') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getVolumeSize File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3059, in getVolumeSize dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) 2019-04-13 21:40:15,025+0200 INFO (vm/5dd5db29) [storage.TaskManager.Task] (Task='71b89391-16f3-42cc-bf2d-64a9dc9b575a') aborting: Task is aborted: "Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',)" - code 358 (task:1181) 2019-04-13 21:40:15,025+0200 ERROR (vm/5dd5db29) [storage.Dispatcher] FINISH getVolumeSize error=Storage domain does not exist: ('9c3f06cf-7475-448e-819b-f4f52fa7d782',) (dispatcher:83) 2019-04-13 21:40:15,336+0200 INFO (vm/5dd5db29) [virt.vm] (vmId='5dd5db29-143f-4d70-9469-4aaa163b7f9d') Changed state to Down: VM terminated with error (code=1) (vm:1675) 2019-04-13 21:40:15,354+0200 INFO (vm/5dd5db29) [virt.vm] (vmId='5dd5db29-143f-4d70-9469-4aaa163b7f9d') Stopping connection (guestagent:455) 2019-04-13 21:40:15,354+0200 WARN (vm/5dd5db29) [vds] Not ready yet, ignoring event '|virt|VM_status|5dd5db29-143f-4d70-9469-4aaa163b7f9d' args={'5dd5db29-143f-4d70-9469-4aaa163b7f9d': {'status': 'Down', 'exitMessage': 'VM terminated with error', 'vmId': '5dd5db29-143f-4d70-9469-4aaa163b7f9d', 'exitReason': 1, 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:15,355+0200 WARN (vm/5dd5db29) [vds] Not ready yet, ignoring event '|virt|VM_status|5dd5db29-143f-4d70-9469-4aaa163b7f9d' args={'5dd5db29-143f-4d70-9469-4aaa163b7f9d': {'status': 'Down', 'displayInfo': [{'tlsPort': '-1', 'ipAddress': '0', 'type': 'vnc', 'port': '-1'}], 'hash': '4658040420000682773', 'exitMessage': 'VM terminated with error', 'cpuUser': '0.00', 'monitorResponse': '0', 'vmId': '5dd5db29-143f-4d70-9469-4aaa163b7f9d', 'exitReason': 1, 'cpuUsage': '0.00', 'elapsedTime': '1727685', 'cpuSys': '0.00', 'timeOffset': '0', 'clientIp': '', 'exitCode': 1}} (clientIF:212) 2019-04-13 21:40:15,671+0200 INFO (vmrecovery) [vdsm.api] START getConnectedStoragePoolsList(options=None) from=internal, task_id=68c23e8b-4916-4c97-9c3d-acb75b8a8350 (api:48) 2019-04-13 21:40:15,671+0200 INFO (vmrecovery) [vdsm.api] FINISH getConnectedStoragePoolsList return={'poollist': []} from=internal, task_id=68c23e8b-4916-4c97-9c3d-acb75b8a8350 (api:54) 2019-04-13 21:40:15,671+0200 INFO (vmrecovery) [vds] recovery: waiting for storage pool to go up (clientIF:709) 2019-04-13 21:40:16,217+0200 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:58472 (protocoldetector:61) 2019-04-13 21:40:16,229+0200 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::1:58472 (protocoldetector:125) 2019-04-13 21:40:16,230+0200 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompserver:95) 2019-04-13 21:40:16,230+0200 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompserver:124) 2019-04-13 21:40:16,271+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:40:16,278+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:40:16,279+0200 INFO (jsonrpc/6) [throttled] Current getAllVmStats: {'be42701b-3f74-4c81-b6ef-5f7e7806554a': 'Down', '28d7d5ff-2ca1-42c6-ae8d-065c721d3066': 'Down', 'dd24e11f-40aa-47d2-9f0b-11cd7e83649d': 'Down', '40eaf7e2-659c-4790-8362-49fe12c5f61a': 'Down', '0f8aff0a-7915-4214-a37c-c9fee6518869': 'Down', '5dd5db29-143f-4d70-9469-4aaa163b7f9d': 'Down', 'a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3': 'Down', '139534f6-1741-4ded-8ea6-ea4e3ecdad1f': 'Down', '7b5e0fdd-8757-49ce-acf9-3206a372e1c9': 'Down', 'b5713713-ac04-4a91-95b8-3c3f17eba32f': 'Down', '6772f848-4874-4785-9bb0-88d40bfb71c4': 'Down', 'afa2c17f-39e4-4168-aad7-2548dd19cc26': 'Down', '611486ee-b72f-4acc-8925-4d7b6df9af1d': 'Down', 'c8df4609-fe26-4b73-9137-57724aff3ca4': 'Down', 'cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6': 'Down', '5a326302-86c1-41b5-ba58-793232299713': 'Down'} (throttledlog:104) 2019-04-13 21:40:16,279+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:40:16,280+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:40:17,081+0200 INFO (jsonrpc/7) [api.host] START getCapabilities() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:40:21,073+0200 INFO (vmrecovery) [vdsm.api] START getConnectedStoragePoolsList(options=None) from=internal, task_id=308bb7c0-3e93-4ada-a154-71825a0e8897 (api:48) 2019-04-13 21:40:21,073+0200 INFO (vmrecovery) [vdsm.api] FINISH getConnectedStoragePoolsList return={'poollist': []} from=internal, task_id=308bb7c0-3e93-4ada-a154-71825a0e8897 (api:54) 2019-04-13 21:40:21,073+0200 INFO (vmrecovery) [vds] recovery: waiting for storage pool to go up (clientIF:709) 2019-04-13 21:40:29,311+0200 INFO (vmrecovery) [vdsm.api] START getConnectedStoragePoolsList(options=None) from=internal, task_id=5f4b3f51-799c-4035-90a3-5f32b8789abb (api:48) 2019-04-13 21:40:29,311+0200 INFO (vmrecovery) [vdsm.api] FINISH getConnectedStoragePoolsList return={'poollist': []} from=internal, task_id=5f4b3f51-799c-4035-90a3-5f32b8789abb (api:54) 2019-04-13 21:40:29,311+0200 INFO (vmrecovery) [vds] recovery: waiting for storage pool to go up (clientIF:709) 2019-04-13 21:40:32,936+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:40:32,938+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:40:32,939+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:40:32,970+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.setKsmTune succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:40:35,345+0200 INFO (vmrecovery) [vdsm.api] START getConnectedStoragePoolsList(options=None) from=internal, task_id=4595edc8-0e9e-49c9-b70a-c785804b8860 (api:48) 2019-04-13 21:40:35,345+0200 INFO (vmrecovery) [vdsm.api] FINISH getConnectedStoragePoolsList return={'poollist': []} from=internal, task_id=4595edc8-0e9e-49c9-b70a-c785804b8860 (api:54) 2019-04-13 21:40:35,345+0200 INFO (vmrecovery) [vds] recovery: waiting for storage pool to go up (clientIF:709) 2019-04-13 21:40:35,396+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=b8a0dca3-a60f-49e3-8ef2-d49dc3edc7b1 (api:48) 2019-04-13 21:40:35,396+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=b8a0dca3-a60f-49e3-8ef2-d49dc3edc7b1 (api:54) 2019-04-13 21:40:40,592+0200 INFO (vmrecovery) [vdsm.api] START getConnectedStoragePoolsList(options=None) from=internal, task_id=45ef4178-0338-4cc3-b089-a86277d21bb4 (api:48) 2019-04-13 21:40:40,592+0200 INFO (vmrecovery) [vdsm.api] FINISH getConnectedStoragePoolsList return={'poollist': []} from=internal, task_id=45ef4178-0338-4cc3-b089-a86277d21bb4 (api:54) 2019-04-13 21:40:40,592+0200 INFO (vmrecovery) [vds] recovery: waiting for storage pool to go up (clientIF:709) 2019-04-13 21:40:41,261+0200 INFO (jsonrpc/7) [root] managedvolume not supported: Managed Volume Not Supported. Missing package os-brick.: ('Cannot import os_brick',) (caps:149) 2019-04-13 21:40:41,941+0200 INFO (jsonrpc/7) [root] /usr/libexec/vdsm/hooks/after_get_caps/50_openstacknet: rc=0 err= (hooks:114) 2019-04-13 21:40:42,212+0200 INFO (jsonrpc/7) [root] /usr/libexec/vdsm/hooks/after_get_caps/openstacknet_utils.py: rc=0 err= (hooks:114) 2019-04-13 21:40:42,568+0200 INFO (jsonrpc/7) [root] /usr/libexec/vdsm/hooks/after_get_caps/ovirt_provider_ovn_hook: rc=0 err= (hooks:114) 2019-04-13 21:40:42,570+0200 INFO (jsonrpc/7) [api.host] FINISH getCapabilities return={'status': {'message': 'Done', 'code': 0}, 'info': {u'HBAInventory': {u'iSCSI': [{u'InitiatorName': u'iqn.1994-05.com.redhat:5ee45db2f0f'}], u'FC': [{u'wwpn': u'21000024ff2c1d40', u'wwnn': u'20000024ff2c1d40', u'model': u'QLE2562 - QLogic 8Gb FC Dual-port HBA for System x'}, {u'wwpn': u'21000024ff2c1d41', u'wwnn': u'20000024ff2c1d41', u'model': u'QLE2562 - QLogic 8Gb FC Dual-port HBA for System x'}]}, u'packages2': {u'kernel': {u'release': u'957.10.1.el7.x86_64', u'version': u'3.10.0'}, u'glusterfs-rdma': {u'release': u'2.el7', u'version': u'5.3'}, u'glusterfs-fuse': {u'release': u'2.el7', u'version': u'5.3'}, u'spice-server': {u'release': u'6.el7_6.1', u'version': u'0.14.0'}, u'librbd1': {u'release': u'4.el7', u'version': u'10.2.5'}, u'vdsm': {u'release': u'1.el7', u'version': u'4.30.11'}, u'qemu-kvm': {u'release': u'18.el7_6.3.1', u'version': u'2.12.0'}, u'openvswitch': {u'release': u'3.el7', u'version': u'2.10.1'}, u'libvirt': {u'release': u'10.el7_6.6', u'version': u'4.5.0'}, u'ovirt-hosted-engine-ha': {u'release': u'1.el7', u'version': u'2.3.1'}, u'qemu-img': {u'release': u'18.el7_6.3.1', u'version': u'2.12.0'}, u'mom': {u'release': u'1.el7.centos', u'version': u'0.5.12'}, u'glusterfs': {u'release': u'2.el7', u'version': u'5.3'}, u'glusterfs-cli': {u'release': u'2.el7', u'version': u'5.3'}, u'glusterfs-server': {u'release': u'2.el7', u'version': u'5.3'}, u'glusterfs-geo-replication': {u'release': u'2.el7', u'version': u'5.3'}}, u'numaNodeDistance': {u'1': [21, 10], u'0': [10, 21]}, u'cpuModel': u'Intel(R) Xeon(R) CPU X5660 @ 2.80GHz', u'nestedVirtualization': False, u'liveMerge': u'true', u'hooks': {u'after_vm_start': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'ea0a5a715da8c1badbcda28e8b8fa00e'}}, u'after_network_setup': {u'30_ethtool_options': {u'md5': u'ce1fbad7aa0389e3b06231219140bf0d'}}, u'before_vm_start': {u'50_hostedengine': {u'md5': u'95c810cdcfe4195302a59574a5148289'}, u'50_vhostmd': {u'md5': u'9206bc390bcbf208b06a8e899581be2d'}}, u'after_device_migrate_destination': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'6226fbc4d1602994828a3904fc1b875d'}}, u'before_device_migrate_destination': {u'50_vmfex': {u'md5': u'49caba1a5faadd8efacef966f79bc30a'}}, u'before_device_create': {u'50_vmfex': {u'md5': u'49caba1a5faadd8efacef966f79bc30a'}, u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'ec8d5d7ba063a109f749cd63c7a2b58d'}, u'20_ovirt_provider_ovn_vhostuser_hook': {u'md5': u'a8af653b7386c138b2e6e9738bd6b62c'}, u'10_ovirt_provider_ovn_hook': {u'md5': u'73822988042847bab1ea832a6b9fa837'}}, u'after_get_caps': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'5c3a9ab6e06e039bdd220c0216e45809'}, u'ovirt_provider_ovn_hook': {u'md5': u'4c4b1d2d5460e6a65114ae36cb775df6'}}, u'after_nic_hotplug': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'6226fbc4d1602994828a3904fc1b875d'}}, u'before_vm_migrate_destination': {u'50_vhostmd': {u'md5': u'9206bc390bcbf208b06a8e899581be2d'}}, u'after_device_create': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'6226fbc4d1602994828a3904fc1b875d'}}, u'before_vm_dehibernate': {u'50_vhostmd': {u'md5': u'9206bc390bcbf208b06a8e899581be2d'}}, u'before_nic_hotplug': {u'50_vmfex': {u'md5': u'49caba1a5faadd8efacef966f79bc30a'}, u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'ec8d5d7ba063a109f749cd63c7a2b58d'}, u'20_ovirt_provider_ovn_vhostuser_hook': {u'md5': u'a8af653b7386c138b2e6e9738bd6b62c'}, u'10_ovirt_provider_ovn_hook': {u'md5': u'73822988042847bab1ea832a6b9fa837'}}, u'before_network_setup': {u'50_fcoe': {u'md5': u'28c352339c8beef1e1b05c67d106d062'}}, u'after_vm_destroy': {u'delete_vhostuserclient_hook': {u'md5': u'c2f279cc9483a3f842f6c29df13994c1'}, u'50_vhostmd': {u'md5': u'bdf4802c0521cf1bae08f2b90a9559cf'}}}, u'supportsIPv6': True, u'realtimeKernel': False, u'vmTypes': [u'kvm'], u'liveSnapshot': u'true', u'cpuThreads': u'24', u'kdumpStatus': 1, u'networks': {u'ovirtmgmt': {u'iface': u'ovirtmgmt', u'ipv6autoconf': True, u'addr': u'192.168.0.35', u'dhcpv6': False, u'ipv6addrs': [], u'switch': u'legacy', u'bridged': True, u'southbound': u'bond0', u'dhcpv4': False, u'netmask': u'255.255.255.0', u'ipv4defaultroute': True, u'stp': u'off', u'ipv4addrs': [u'192.168.0.35/24'], u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u'192.168.0.1', u'ports': [u'bond0']}, u'streamingnet': {u'iface': u'streamingnet', u'ipv6autoconf': True, u'addr': u'', u'dhcpv6': False, u'ipv6addrs': [], u'switch': u'legacy', u'bridged': True, u'southbound': u'eno4', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'stp': u'off', u'ipv4addrs': [], u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u'', u'ports': [u'eno4']}}, u'kernelArgs': u'BOOT_IMAGE=/vmlinuz-3.10.0-957.10.1.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=de_DE.UTF-8 intel_iommu=on', u'domain_versions': [0, 2, 3, 4, 5], u'bridges': {u'ovirtmgmt': {u'ipv6autoconf': True, u'addr': u'192.168.0.35', u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'255.255.255.0', u'ipv4defaultroute': True, u'stp': u'off', u'ipv4addrs': [u'192.168.0.35/24'], u'ipv6gateway': u'::', u'gateway': u'192.168.0.1', u'opts': {u'multicast_last_member_count': u'2', u'vlan_protocol': u'0x8100', u'hash_elasticity': u'4', u'multicast_query_response_interval': u'1000', u'group_fwd_mask': u'0x0', u'multicast_snooping': u'1', u'multicast_startup_query_interval': u'3125', u'hello_timer': u'0', u'multicast_querier_interval': u'25500', u'max_age': u'2000', u'hash_max': u'512', u'stp_state': u'0', u'topology_change_detected': u'0', u'priority': u'32768', u'multicast_igmp_version': u'2', u'multicast_membership_interval': u'26000', u'root_path_cost': u'0', u'root_port': u'0', u'multicast_stats_enabled': u'0', u'multicast_startup_query_count': u'2', u'nf_call_iptables': u'0', u'vlan_stats_enabled': u'0', u'hello_time': u'200', u'topology_change': u'0', u'bridge_id': u'8000.e41f13b59416', u'topology_change_timer': u'0', u'ageing_time': u'30000', u'nf_call_ip6tables': u'0', u'multicast_mld_version': u'1', u'gc_timer': u'26934', u'root_id': u'8000.e41f13b59416', u'nf_call_arptables': u'0', u'group_addr': u'1:80:c2:0:0:0', u'multicast_last_member_interval': u'100', u'default_pvid': u'1', u'multicast_query_interval': u'12500', u'multicast_query_use_ifaddr': u'0', u'tcn_timer': u'0', u'multicast_router': u'1', u'vlan_filtering': u'0', u'multicast_querier': u'0', u'forward_delay': u'0'}, u'ports': [u'bond0']}, u'streamingnet': {u'ipv6autoconf': True, u'addr': u'', u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'stp': u'off', u'ipv4addrs': [], u'ipv6gateway': u'::', u'gateway': u'', u'opts': {u'multicast_last_member_count': u'2', u'vlan_protocol': u'0x8100', u'hash_elasticity': u'4', u'multicast_query_response_interval': u'1000', u'group_fwd_mask': u'0x0', u'multicast_snooping': u'1', u'multicast_startup_query_interval': u'3125', u'hello_timer': u'0', u'multicast_querier_interval': u'25500', u'max_age': u'2000', u'hash_max': u'512', u'stp_state': u'0', u'topology_change_detected': u'0', u'priority': u'32768', u'multicast_igmp_version': u'2', u'multicast_membership_interval': u'26000', u'root_path_cost': u'0', u'root_port': u'0', u'multicast_stats_enabled': u'0', u'multicast_startup_query_count': u'2', u'nf_call_iptables': u'0', u'vlan_stats_enabled': u'0', u'hello_time': u'200', u'topology_change': u'0', u'bridge_id': u'8000.5cf3fc3918de', u'topology_change_timer': u'0', u'ageing_time': u'30000', u'nf_call_ip6tables': u'0', u'multicast_mld_version': u'1', u'gc_timer': u'27446', u'root_id': u'8000.5cf3fc3918de', u'nf_call_arptables': u'0', u'group_addr': u'1:80:c2:0:0:0', u'multicast_last_member_interval': u'100', u'default_pvid': u'1', u'multicast_query_interval': u'12500', u'multicast_query_use_ifaddr': u'0', u'tcn_timer': u'0', u'multicast_router': u'1', u'vlan_filtering': u'0', u'multicast_querier': u'0', u'forward_delay': u'0'}, u'ports': [u'eno4']}}, u'uuid': u'2650319A-1C29-11E0-8699-E41F13B59414', u'onlineCpus': u'0,1,2,3,4,5,12,13,14,15,16,17,6,7,8,9,10,11,18,19,20,21,22,23', u'nameservers': [u'192.168.0.1'], u'nics': {u'eno1': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e4:1f:13:b5:94:14', u'ipv6gateway': u'::', u'gateway': u''}, u'eno2': {u'permhwaddr': u'e4:1f:13:b5:94:16', u'ipv6autoconf': False, u'addr': u'', u'speed': 1000, u'dhcpv6': False, u'ipv6addrs': [], u'ad_aggregator_id': u'1', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e4:1f:13:b5:94:16', u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u''}, u'eno3': {u'permhwaddr': u'5c:f3:fc:39:18:dc', u'ipv6autoconf': False, u'addr': u'', u'speed': 1000, u'dhcpv6': False, u'ipv6addrs': [], u'ad_aggregator_id': u'1', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e4:1f:13:b5:94:16', u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u''}, u'eno4': {u'ipv6autoconf': False, u'addr': u'', u'speed': 1000, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'5c:f3:fc:39:18:de', u'ipv6gateway': u'::', u'gateway': u''}, u'ens3f3': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'00:1b:21:8a:a6:cb', u'ipv6gateway': u'::', u'gateway': u''}, u'ens3f2': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'00:1b:21:8a:a6:ca', u'ipv6gateway': u'::', u'gateway': u''}, u'enp0s29f0u2': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e6:1f:13:a8:94:17', u'ipv6gateway': u'::', u'gateway': u''}, u'ens3f0': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'00:1b:21:8a:a6:c8', u'ipv6gateway': u'::', u'gateway': u''}, u'ens3f1': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'00:1b:21:8a:a6:c9', u'ipv6gateway': u'::', u'gateway': u''}}, u'software_revision': u'1', u'hostdevPassthrough': u'true', u'clusterLevels': [u'4.1', u'4.2', u'4.3'], u'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,eagerfpu,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,popcnt,aes,lahf_lm,epb,ssbd,ibrs,ibpb,stibp,tpr_shadow,vnmi,flexpriority,ept,vpid,dtherm,arat,spec_ctrl,intel_stibp,flush_l1d,model_Opteron_G2,model_kvm32,model_coreduo,model_Conroe,model_Nehalem,model_Westmere-IBRS,model_Opteron_G1,model_core2duo,model_Nehalem-IBRS,model_qemu32,model_Penryn,model_pentium2,model_pentium3,model_qemu64,model_Westmere,model_kvm64,model_pentium,model_486', u'kernelFeatures': {u'RETP': 1, u'IBRS': 0, u'PTI': 1}, u'ISCSIInitiatorName': u'iqn.1994-05.com.redhat:5ee45db2f0f', u'netConfigDirty': u'False', u'selinux': {u'mode': u'1'}, u'autoNumaBalancing': 1, u'reservedMem': u'321', u'bondings': {u'bond0': {u'ipv6autoconf': False, u'ad_partner_mac': u'ec:b1:d7:11:34:40', u'speed': 2000, u'dhcpv6': False, u'ipv6addrs': [], u'netmask': u'', u'active_slave': u'', u'ad_aggregator_id': u'1', u'dhcpv4': False, u'switch': u'legacy', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e4:1f:13:b5:94:16', u'slaves': [u'eno2', u'eno3'], u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u'', u'opts': {u'mode': u'4', u'xmit_hash_policy': u'2'}, u'addr': u''}}, u'software_version': u'4.30.11', u'supportedENGINEs': [u'4.1', u'4.2', u'4.3'], u'vncEncrypted': True, u'backupEnabled': False, u'cpuSpeed': u'2793.000', u'numaNodes': {u'1': {u'totalMemory': u'98304', u'cpus': [6, 7, 8, 9, 10, 11, 18, 19, 20, 21, 22, 23]}, u'0': {u'totalMemory': u'98291', u'cpus': [0, 1, 2, 3, 4, 5, 12, 13, 14, 15, 16, 17]}}, u'cpuSockets': u'2', u'vlans': {}, u'version_name': u'Snow Man', u'cpuCores': u'12', u'hostedEngineDeployed': False, u'hugepages': [1048576, 2048], u'guestOverhead': u'65', u'additionalFeatures': [u'libgfapi_supported', u'GLUSTER_SNAPSHOT', u'GLUSTER_GEO_REPLICATION', u'GLUSTER_BRICK_MANAGEMENT'], u'kvmEnabled': u'true', u'memSize': u'193330', u'emulatedMachines': [u'pc-i440fx-rhel7.1.0', u'pc-q35-rhel7.3.0', u'rhel6.3.0', u'pc-i440fx-rhel7.5.0', u'pc-i440fx-rhel7.0.0', u'rhel6.1.0', u'pc-q35-rhel7.6.0', u'pc-i440fx-rhel7.4.0', u'rhel6.6.0', u'pc-q35-rhel7.5.0', u'rhel6.2.0', u'pc', u'pc-i440fx-rhel7.3.0', u'q35', u'pc-i440fx-rhel7.2.0', u'rhel6.4.0', u'pc-q35-rhel7.4.0', u'pc-i440fx-rhel7.6.0', u'rhel6.0.0', u'rhel6.5.0'], u'rngSources': [u'hwrng', u'random'], u'operatingSystem': {u'release': u'6.1810.2.el7.centos', u'pretty_name': u'CentOS Linux 7 (Core)', u'version': u'7', u'name': u'RHEL'}}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:40:42,581+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getCapabilities succeeded in 25.50 seconds (__init__:312) 2019-04-13 21:40:42,718+0200 INFO (jsonrpc/2) [api.host] START getHardwareInfo() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:40:42,769+0200 INFO (jsonrpc/2) [api.host] FINISH getHardwareInfo return={'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'System x3650 M3 -[7945L2G]-', 'systemSerialNumber': '99L8940', 'systemFamily': 'System x', 'systemVersion': '00', 'systemUUID': '2650319a-1c29-11e0-8699-e41f13b59414', 'systemManufacturer': 'IBM'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:40:42,769+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getHardwareInfo succeeded in 0.05 seconds (__init__:312) 2019-04-13 21:40:42,960+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,51130, flow_id=5fa1e25 (api:48) 2019-04-13 21:40:42,982+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, flow_id=5fa1e25, task_id=0505b925-3627-4566-986e-fcc8aa791195 (api:48) 2019-04-13 21:40:42,982+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={} from=::ffff:192.168.0.19,51130, flow_id=5fa1e25, task_id=0505b925-3627-4566-986e-fcc8aa791195 (api:54) 2019-04-13 21:40:42,983+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, flow_id=5fa1e25, task_id=fb53c418-6d31-4b65-866b-d3569260d3da (api:48) 2019-04-13 21:40:42,983+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, flow_id=5fa1e25, task_id=fb53c418-6d31-4b65-866b-d3569260d3da (api:54) 2019-04-13 21:40:42,993+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.81', 'nodeIndex': 1, 'cpuSys': '0.46', 'cpuIdle': '98.73'}, '21': {'cpuUser': '0.52', 'nodeIndex': 1, 'cpuSys': '0.06', 'cpuIdle': '99.42'}, '22': {'cpuUser': '0.06', 'nodeIndex': 1, 'cpuSys': '0.06', 'cpuIdle': '99.88'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '7.51', 'nodeIndex': 0, 'cpuSys': '3.06', 'cpuIdle': '89.43'}, '0': {'cpuUser': '0.29', 'nodeIndex': 0, 'cpuSys': '0.23', 'cpuIdle': '99.48'}, '3': {'cpuUser': '0.46', 'nodeIndex': 0, 'cpuSys': '0.29', 'cpuIdle': '99.25'}, '2': {'cpuUser': '0.52', 'nodeIndex': 0, 'cpuSys': '0.46', 'cpuIdle': '99.02'}, '5': {'cpuUser': '0.40', 'nodeIndex': 0, 'cpuSys': '0.29', 'cpuIdle': '99.31'}, '4': {'cpuUser': '1.39', 'nodeIndex': 0, 'cpuSys': '0.29', 'cpuIdle': '98.32'}, '7': {'cpuUser': '0.17', 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.43'}, '6': {'cpuUser': '0.12', 'nodeIndex': 1, 'cpuSys': '0.29', 'cpuIdle': '99.59'}, '9': {'cpuUser': '0.64', 'nodeIndex': 1, 'cpuSys': '0.29', 'cpuIdle': '99.07'}, '8': {'cpuUser': '0.23', 'nodeIndex': 1, 'cpuSys': '0.35', 'cpuIdle': '99.42'}, '11': {'cpuUser': '0.12', 'nodeIndex': 1, 'cpuSys': '0.06', 'cpuIdle': '99.82'}, '10': {'cpuUser': '0.17', 'nodeIndex': 1, 'cpuSys': '0.23', 'cpuIdle': '99.60'}, '13': {'cpuUser': '0.29', 'nodeIndex': 0, 'cpuSys': '0.58', 'cpuIdle': '99.13'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.06', 'cpuIdle': '99.94'}, '14': {'cpuUser': '0.06', 'nodeIndex': 0, 'cpuSys': '0.06', 'cpuIdle': '99.88'}, '17': {'cpuUser': '0.06', 'nodeIndex': 0, 'cpuSys': '0.06', 'cpuIdle': '99.88'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '9.76', 'nodeIndex': 1, 'cpuSys': '1.15', 'cpuIdle': '89.09'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95669'}, '0': {'memPercent': 3, 'memFree': '95514'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '1', 'storageDomains': {}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184442.981489, 'rx': '1736', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '12807', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184442.981489, 'rx': '14086', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184442.981489, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '24391', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184442.981489, 'rx': '11558', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '17361', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184442.981489, 'rx': '17361', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184442.981489, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '4053', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184442.981489, 'rx': '12114', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '8754', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184442.981489, 'rx': '1972', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184442.981489, 'rx': '8564', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184442.981489, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184442.981489, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184442.981489, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184442.981489, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184442.981489, 'rx': '5750', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '42', 'ksmPages': 100, 'elapsedTime': '30.98', 'cpuLoad': '0.71', 'cpuSys': '0.36', 'diskStats': {'/var/log': {'free': '47810'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47810'}}, 'cpuUserVdsmd': '7.79', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': True, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191831, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:40:42 GMT', 'cpuUser': '0.98', 'memFree': 191575, 'cpuIdle': '98.66', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '2.14'}} from=::ffff:192.168.0.19,51130, flow_id=5fa1e25 (api:54) 2019-04-13 21:40:42,996+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:40:43,384+0200 INFO (jsonrpc/4) [vdsm.api] START connectStorageServer(domType=1, spUUID=u'59ef3a18-002f-02d1-0220-000000000124', conList=[{u'tpgt': u'1', u'id': u'2286311a-c1c9-4dda-bea5-9c78651c6e36', u'connection': u'onode-1.discovery.intern:/mnt/store', u'iqn': u'', u'user': u'', u'ipv6_enabled': u'false', u'protocol_version': u'auto', u'password': '********', u'port': u''}], options=None) from=::ffff:192.168.0.19,51130, flow_id=344934b8, task_id=a95bb281-d381-4f24-865c-14fb1583f2e9 (api:48) 2019-04-13 21:40:43,386+0200 INFO (jsonrpc/4) [storage.StorageServer.MountConnection] Creating directory u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store' (storageServer:168) 2019-04-13 21:40:43,386+0200 INFO (jsonrpc/4) [storage.fileUtils] Creating directory: /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store mode: None (fileUtils:199) 2019-04-13 21:40:43,386+0200 INFO (jsonrpc/4) [storage.Mount] mounting onode-1.discovery.intern:/mnt/store at /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store (mount:204) 2019-04-13 21:40:43,727+0200 INFO (jsonrpc/4) [IOProcessClient] (Global) Starting client (__init__:308) 2019-04-13 21:40:43,738+0200 INFO (ioprocess/14781) [IOProcess] (Global) Starting ioprocess (__init__:434) 2019-04-13 21:40:43,754+0200 INFO (jsonrpc/4) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 0, 'id': u'2286311a-c1c9-4dda-bea5-9c78651c6e36'}]} from=::ffff:192.168.0.19,51130, flow_id=344934b8, task_id=a95bb281-d381-4f24-865c-14fb1583f2e9 (api:54) 2019-04-13 21:40:43,755+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 0.37 seconds (__init__:312) 2019-04-13 21:40:43,780+0200 INFO (jsonrpc/5) [vdsm.api] START connectStorageServer(domType=1, spUUID=u'59ef3a18-002f-02d1-0220-000000000124', conList=[{u'tpgt': u'1', u'id': u'b4f4bcf2-e104-468e-b330-ed731ec8ef6c', u'connection': u'nas.discovery.intern:/mnt/d2600/vms', u'iqn': u'', u'user': u'', u'ipv6_enabled': u'false', u'protocol_version': u'auto', u'password': '********', u'port': u''}], options=None) from=::ffff:192.168.0.19,51130, flow_id=344934b8, task_id=721bca1b-2e83-4a96-951f-c7381b0e73b7 (api:48) 2019-04-13 21:40:43,782+0200 INFO (jsonrpc/5) [storage.StorageServer.MountConnection] Creating directory u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms' (storageServer:168) 2019-04-13 21:40:43,782+0200 INFO (jsonrpc/5) [storage.fileUtils] Creating directory: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms mode: None (fileUtils:199) 2019-04-13 21:40:43,782+0200 INFO (jsonrpc/5) [storage.Mount] mounting nas.discovery.intern:/mnt/d2600/vms at /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms (mount:204) 2019-04-13 21:40:44,004+0200 INFO (jsonrpc/5) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 0, 'id': u'b4f4bcf2-e104-468e-b330-ed731ec8ef6c'}]} from=::ffff:192.168.0.19,51130, flow_id=344934b8, task_id=721bca1b-2e83-4a96-951f-c7381b0e73b7 (api:54) 2019-04-13 21:40:44,005+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 0.22 seconds (__init__:312) 2019-04-13 21:40:44,021+0200 INFO (jsonrpc/6) [vdsm.api] START connectStorageServer(domType=1, spUUID=u'59ef3a18-002f-02d1-0220-000000000124', conList=[{u'tpgt': u'1', u'id': u'07dd9ff2-afa1-4a33-b401-935ea5b8c9db', u'connection': u'nas.discovery.intern:/mnt/d2600/isos', u'iqn': u'', u'user': u'', u'ipv6_enabled': u'false', u'protocol_version': u'auto', u'password': '********', u'port': u''}], options=None) from=::ffff:192.168.0.19,51130, flow_id=344934b8, task_id=c88c0f98-4bfd-44d4-b009-085a46abb4a1 (api:48) 2019-04-13 21:40:44,023+0200 INFO (jsonrpc/6) [storage.StorageServer.MountConnection] Creating directory u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos' (storageServer:168) 2019-04-13 21:40:44,023+0200 INFO (jsonrpc/6) [storage.fileUtils] Creating directory: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos mode: None (fileUtils:199) 2019-04-13 21:40:44,023+0200 INFO (jsonrpc/6) [storage.Mount] mounting nas.discovery.intern:/mnt/d2600/isos at /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos (mount:204) 2019-04-13 21:40:44,135+0200 INFO (jsonrpc/6) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 0, 'id': u'07dd9ff2-afa1-4a33-b401-935ea5b8c9db'}]} from=::ffff:192.168.0.19,51130, flow_id=344934b8, task_id=c88c0f98-4bfd-44d4-b009-085a46abb4a1 (api:54) 2019-04-13 21:40:44,136+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 0.11 seconds (__init__:312) 2019-04-13 21:40:44,151+0200 INFO (jsonrpc/1) [vdsm.api] START connectStorageServer(domType=1, spUUID=u'59ef3a18-002f-02d1-0220-000000000124', conList=[{u'tpgt': u'1', u'id': u'858731bf-2fb1-45a7-a693-235f675a667b', u'connection': u'nas.discovery.intern:/mnt/d2600/sicherung/vmBackups', u'iqn': u'', u'user': u'', u'ipv6_enabled': u'false', u'protocol_version': u'auto', u'password': '********', u'port': u''}], options=None) from=::ffff:192.168.0.19,51130, flow_id=344934b8, task_id=03c62510-357f-4fe5-b7bd-0be5cd0ab7ac (api:48) 2019-04-13 21:40:44,153+0200 INFO (jsonrpc/1) [storage.StorageServer.MountConnection] Creating directory u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups' (storageServer:168) 2019-04-13 21:40:44,153+0200 INFO (jsonrpc/1) [storage.fileUtils] Creating directory: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups mode: None (fileUtils:199) 2019-04-13 21:40:44,153+0200 INFO (jsonrpc/1) [storage.Mount] mounting nas.discovery.intern:/mnt/d2600/sicherung/vmBackups at /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups (mount:204) 2019-04-13 21:40:44,264+0200 INFO (jsonrpc/1) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 0, 'id': u'858731bf-2fb1-45a7-a693-235f675a667b'}]} from=::ffff:192.168.0.19,51130, flow_id=344934b8, task_id=03c62510-357f-4fe5-b7bd-0be5cd0ab7ac (api:54) 2019-04-13 21:40:44,264+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 0.11 seconds (__init__:312) 2019-04-13 21:40:44,346+0200 INFO (jsonrpc/0) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,51130, flow_id=344934b8, task_id=f7940b97-87c9-4db3-8fc5-093a2931e592 (api:48) 2019-04-13 21:40:44,347+0200 INFO (jsonrpc/0) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 21:40:44,347+0200 INFO (jsonrpc/0) [storage.StoragePool] updating pool 59ef3a18-002f-02d1-0220-000000000124 backend from type NoneType instance 0x7fee1f1c5dc0 to type StoragePoolMemoryBackend instance 0x7feda80762b8 (sp:157) 2019-04-13 21:40:44,347+0200 INFO (jsonrpc/0) [storage.StoragePool] Connect host #1 to the storage pool 59ef3a18-002f-02d1-0220-000000000124 with master domain: 6a4cde72-6a6a-499e-9b6e-d490d91ca19e (ver = 47) (sp:688) 2019-04-13 21:40:44,473+0200 INFO (jsonrpc/7) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,51130, task_id=7339c05c-2a31-4d4d-b07c-339b2c1ed8cd (api:48) 2019-04-13 21:40:44,600+0200 INFO (jsonrpc/0) [IOProcessClient] (6a4cde72-6a6a-499e-9b6e-d490d91ca19e) Starting client (__init__:308) 2019-04-13 21:40:44,612+0200 INFO (ioprocess/14873) [IOProcess] (6a4cde72-6a6a-499e-9b6e-d490d91ca19e) Starting ioprocess (__init__:434) 2019-04-13 21:40:44,641+0200 INFO (jsonrpc/0) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:40:44,641+0200 INFO (jsonrpc/0) [storage.Monitor] Start monitoring 9c3f06cf-7475-448e-819b-f4f52fa7d782 (monitor:183) 2019-04-13 21:40:44,642+0200 INFO (jsonrpc/0) [storage.Monitor] Start monitoring 42bb6772-12f9-4bf2-9245-43caa41c5a2a (monitor:183) 2019-04-13 21:40:44,644+0200 INFO (jsonrpc/0) [storage.Monitor] Start monitoring 93f02ede-ec5c-4641-8e97-2949f98e856d (monitor:183) 2019-04-13 21:40:44,645+0200 INFO (jsonrpc/0) [storage.Monitor] Start monitoring 6a4cde72-6a6a-499e-9b6e-d490d91ca19e (monitor:183) 2019-04-13 21:40:44,647+0200 INFO (jsonrpc/0) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 21:40:44,648+0200 INFO (jsonrpc/0) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 21:40:44,654+0200 INFO (monitor/6a4cde7) [storage.check] Start checking u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/metadata' (check:121) 2019-04-13 21:40:44,683+0200 INFO (monitor/6a4cde7) [storage.Monitor] Domain 6a4cde72-6a6a-499e-9b6e-d490d91ca19e became VALID (monitor:470) 2019-04-13 21:40:44,685+0200 INFO (monitor/9c3f06c) [IOProcessClient] (9c3f06cf-7475-448e-819b-f4f52fa7d782) Starting client (__init__:308) 2019-04-13 21:40:44,690+0200 INFO (monitor/6a4cde7) [storage.SANLock] Acquiring host id for domain 6a4cde72-6a6a-499e-9b6e-d490d91ca19e (id=1, async=True) (clusterlock:294) 2019-04-13 21:40:44,702+0200 INFO (event/1) [vds] vmContainerLock acquired (clientIF:243) 2019-04-13 21:40:44,704+0200 INFO (monitor/42bb677) [IOProcessClient] (42bb6772-12f9-4bf2-9245-43caa41c5a2a) Starting client (__init__:308) 2019-04-13 21:40:44,710+0200 INFO (ioprocess/14900) [IOProcess] (9c3f06cf-7475-448e-819b-f4f52fa7d782) Starting ioprocess (__init__:434) 2019-04-13 21:40:44,722+0200 INFO (monitor/93f02ed) [IOProcessClient] (93f02ede-ec5c-4641-8e97-2949f98e856d) Starting client (__init__:308) 2019-04-13 21:40:44,727+0200 INFO (ioprocess/14912) [IOProcess] (42bb6772-12f9-4bf2-9245-43caa41c5a2a) Starting ioprocess (__init__:434) 2019-04-13 21:40:44,737+0200 INFO (ioprocess/14919) [IOProcess] (93f02ede-ec5c-4641-8e97-2949f98e856d) Starting ioprocess (__init__:434) 2019-04-13 21:40:44,741+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:40:44,741+0200 INFO (monitor/42bb677) [storage.check] Start checking u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/dom_md/metadata' (check:121) 2019-04-13 21:40:44,748+0200 INFO (monitor/42bb677) [storage.Monitor] Host id for domain 42bb6772-12f9-4bf2-9245-43caa41c5a2a successfully acquired (id: 1) (monitor:448) 2019-04-13 21:40:44,753+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:40:44,754+0200 INFO (monitor/9c3f06c) [storage.check] Start checking u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/dom_md/metadata' (check:121) 2019-04-13 21:40:44,762+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:40:44,764+0200 INFO (monitor/93f02ed) [storage.check] Start checking u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d/dom_md/metadata' (check:121) 2019-04-13 21:40:44,767+0200 INFO (monitor/93f02ed) [storage.Monitor] Host id for domain 93f02ede-ec5c-4641-8e97-2949f98e856d successfully acquired (id: 1) (monitor:448) 2019-04-13 21:40:44,783+0200 INFO (check/loop) [storage.Monitor] Domain 42bb6772-12f9-4bf2-9245-43caa41c5a2a became VALID (monitor:470) 2019-04-13 21:40:44,784+0200 INFO (check/loop) [storage.Monitor] Domain 9c3f06cf-7475-448e-819b-f4f52fa7d782 became VALID (monitor:470) 2019-04-13 21:40:44,786+0200 INFO (check/loop) [storage.Monitor] Domain 93f02ede-ec5c-4641-8e97-2949f98e856d became VALID (monitor:470) 2019-04-13 21:40:44,786+0200 INFO (event/5) [vds] vmContainerLock acquired (clientIF:243) 2019-04-13 21:40:44,788+0200 INFO (event/8) [vds] vmContainerLock acquired (clientIF:243) 2019-04-13 21:40:44,789+0200 INFO (event/10) [vds] vmContainerLock acquired (clientIF:243) 2019-04-13 21:40:44,835+0200 WARN (jsonrpc/0) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 21:40:44,840+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 21:40:44,840+0200 INFO (jsonrpc/0) [storage.StoragePool] Creating symlink from /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1247) 2019-04-13 21:40:44,840+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 21:40:44,841+0200 INFO (jsonrpc/0) [storage.StoragePool] Creating symlink from /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1247) 2019-04-13 21:40:44,841+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 21:40:44,841+0200 INFO (jsonrpc/0) [storage.StoragePool] Creating symlink from /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1247) 2019-04-13 21:40:44,841+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 21:40:44,841+0200 INFO (jsonrpc/0) [storage.StoragePool] Creating symlink from /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1247) 2019-04-13 21:40:44,841+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 21:40:44,841+0200 INFO (jsonrpc/0) [storage.StoragePool] Creating symlink from /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1247) 2019-04-13 21:40:44,870+0200 INFO (jsonrpc/0) [storage.MailBox.HsmMailMonitor] HSM_MailMonitor sending mail to SPM - ['/usr/bin/dd', 'of=/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd/dom_md/inbox', 'iflag=fullblock', 'oflag=direct', 'conv=notrunc', 'bs=4096', 'count=1', 'seek=1'] (mailbox:381) 2019-04-13 21:40:44,907+0200 INFO (jsonrpc/0) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,51130, flow_id=344934b8, task_id=f7940b97-87c9-4db3-8fc5-093a2931e592 (api:54) 2019-04-13 21:40:44,908+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.57 seconds (__init__:312) 2019-04-13 21:40:44,909+0200 INFO (jsonrpc/7) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 21:40:44,924+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,51130, flow_id=344934b8 (api:48) 2019-04-13 21:40:44,946+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, flow_id=344934b8, task_id=8c82ea6d-1c3f-462a-a983-99ef3bbdb909 (api:48) 2019-04-13 21:40:44,947+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000603988', 'lastCheck': '0.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000445815', 'lastCheck': '0.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000537318', 'lastCheck': '0.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000308135', 'lastCheck': '0.3', 'valid': True}} from=::ffff:192.168.0.19,51130, flow_id=344934b8, task_id=8c82ea6d-1c3f-462a-a983-99ef3bbdb909 (api:54) 2019-04-13 21:40:44,947+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, flow_id=344934b8, task_id=091ae183-3e37-4c58-8ea0-868ef5a4a0b5 (api:48) 2019-04-13 21:40:44,948+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, flow_id=344934b8, task_id=091ae183-3e37-4c58-8ea0-868ef5a4a0b5 (api:54) 2019-04-13 21:40:44,957+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.81', 'nodeIndex': 1, 'cpuSys': '0.46', 'cpuIdle': '98.73'}, '21': {'cpuUser': '0.52', 'nodeIndex': 1, 'cpuSys': '0.06', 'cpuIdle': '99.42'}, '22': {'cpuUser': '0.06', 'nodeIndex': 1, 'cpuSys': '0.06', 'cpuIdle': '99.88'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '7.51', 'nodeIndex': 0, 'cpuSys': '3.06', 'cpuIdle': '89.43'}, '0': {'cpuUser': '0.29', 'nodeIndex': 0, 'cpuSys': '0.23', 'cpuIdle': '99.48'}, '3': {'cpuUser': '0.46', 'nodeIndex': 0, 'cpuSys': '0.29', 'cpuIdle': '99.25'}, '2': {'cpuUser': '0.52', 'nodeIndex': 0, 'cpuSys': '0.46', 'cpuIdle': '99.02'}, '5': {'cpuUser': '0.40', 'nodeIndex': 0, 'cpuSys': '0.29', 'cpuIdle': '99.31'}, '4': {'cpuUser': '1.39', 'nodeIndex': 0, 'cpuSys': '0.29', 'cpuIdle': '98.32'}, '7': {'cpuUser': '0.17', 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.43'}, '6': {'cpuUser': '0.12', 'nodeIndex': 1, 'cpuSys': '0.29', 'cpuIdle': '99.59'}, '9': {'cpuUser': '0.64', 'nodeIndex': 1, 'cpuSys': '0.29', 'cpuIdle': '99.07'}, '8': {'cpuUser': '0.23', 'nodeIndex': 1, 'cpuSys': '0.35', 'cpuIdle': '99.42'}, '11': {'cpuUser': '0.12', 'nodeIndex': 1, 'cpuSys': '0.06', 'cpuIdle': '99.82'}, '10': {'cpuUser': '0.17', 'nodeIndex': 1, 'cpuSys': '0.23', 'cpuIdle': '99.60'}, '13': {'cpuUser': '0.29', 'nodeIndex': 0, 'cpuSys': '0.58', 'cpuIdle': '99.13'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.06', 'cpuIdle': '99.94'}, '14': {'cpuUser': '0.06', 'nodeIndex': 0, 'cpuSys': '0.06', 'cpuIdle': '99.88'}, '17': {'cpuUser': '0.06', 'nodeIndex': 0, 'cpuSys': '0.06', 'cpuIdle': '99.88'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '9.76', 'nodeIndex': 1, 'cpuSys': '1.15', 'cpuIdle': '89.09'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95669'}, '0': {'memPercent': 3, 'memFree': '95514'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 16, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000603988', 'lastCheck': '0.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000537318', 'lastCheck': '0.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000308135', 'lastCheck': '0.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000445815', 'lastCheck': '0.2', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184444.945953, 'rx': '1736', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '56732', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184444.945953, 'rx': '63660', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184444.945953, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '89483', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184444.945953, 'rx': '88507', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '71513', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184444.945953, 'rx': '71513', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184444.945953, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '4733', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184444.945953, 'rx': '61560', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '51999', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184444.945953, 'rx': '2100', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184444.945953, 'rx': '8820', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184444.945953, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184444.945953, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184444.945953, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184444.945953, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184444.945953, 'rx': '5934', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '42', 'ksmPages': 100, 'elapsedTime': '32.94', 'cpuLoad': '0.71', 'cpuSys': '0.36', 'diskStats': {'/var/log': {'free': '47810'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47810'}}, 'cpuUserVdsmd': '7.79', 'netConfigDirty': 'False', 'memCommitted': 47104, 'ksmState': True, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191813, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:40:44 GMT', 'cpuUser': '0.98', 'memFree': 191557, 'cpuIdle': '98.66', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '2.14'}} from=::ffff:192.168.0.19,51130, flow_id=344934b8 (api:54) 2019-04-13 21:40:44,960+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.04 seconds (__init__:312) 2019-04-13 21:40:45,215+0200 INFO (jsonrpc/7) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:40:45,216+0200 INFO (jsonrpc/7) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 21:40:45,216+0200 INFO (jsonrpc/7) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 21:40:45,221+0200 INFO (event/2) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 21:40:45,221+0200 INFO (event/2) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 21:40:45,352+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.setMOMPolicyParameters succeeded in 0.07 seconds (__init__:312) 2019-04-13 21:40:45,363+0200 WARN (jsonrpc/7) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 21:40:45,368+0200 INFO (jsonrpc/7) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 21:40:45,368+0200 INFO (jsonrpc/7) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 21:40:45,368+0200 INFO (jsonrpc/7) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 21:40:45,368+0200 INFO (jsonrpc/7) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 21:40:45,368+0200 INFO (jsonrpc/7) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 21:40:45,368+0200 INFO (jsonrpc/7) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,51130, task_id=7339c05c-2a31-4d4d-b07c-339b2c1ed8cd (api:54) 2019-04-13 21:40:45,369+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.90 seconds (__init__:312) 2019-04-13 21:40:45,385+0200 INFO (jsonrpc/4) [api.host] START hostdevListByCaps(caps=None) from=::ffff:192.168.0.19,51130, flow_id=7e5fcf82 (api:48) 2019-04-13 21:40:45,390+0200 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=7ecc694b-7797-4a12-8bc8-3e566b95d046 (api:48) 2019-04-13 21:40:45,423+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=7ecc694b-7797-4a12-8bc8-3e566b95d046 (api:52) 2019-04-13 21:40:45,424+0200 ERROR (jsonrpc/5) [storage.TaskManager.Task] (Task='7ecc694b-7797-4a12-8bc8-3e566b95d046') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:40:45,424+0200 INFO (jsonrpc/5) [storage.TaskManager.Task] (Task='7ecc694b-7797-4a12-8bc8-3e566b95d046') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:40:45,425+0200 ERROR (jsonrpc/5) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:40:45,425+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.03 seconds (__init__:312) 2019-04-13 21:40:45,467+0200 INFO (event/6) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:40:45,468+0200 INFO (event/6) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 21:40:45,470+0200 INFO (event/9) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:40:45,470+0200 INFO (event/9) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 21:40:45,478+0200 INFO (event/11) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:40:45,479+0200 INFO (event/11) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 21:40:45,489+0200 INFO (jsonrpc/4) [api.host] FINISH hostdevListByCaps return={'status': {'message': 'Done', 'code': 0}, 'deviceList': {'net_eno1_e4_1f_13_b5_94_14': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'eno1', 'parent': 'pci_0000_0b_00_0'}}, 'pci_0000_00_1d_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB UHCI Controller #3', 'vendor': 'Intel Corporation', 'product_id': '0x3a36', 'parent': 'computer', 'address': {'slot': '29', 'bus': '0', 'domain': '0', 'function': '2'}, 'iommu_group': '14', 'vendor_id': '0x8086', 'driver': 'uhci_hcd'}}, 'pci_0000_00_1d_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB UHCI Controller #2', 'vendor': 'Intel Corporation', 'product_id': '0x3a35', 'parent': 'computer', 'address': {'slot': '29', 'bus': '0', 'domain': '0', 'function': '1'}, 'iommu_group': '14', 'vendor_id': '0x8086', 'driver': 'uhci_hcd'}}, 'pci_0000_00_1d_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB UHCI Controller #1', 'vendor': 'Intel Corporation', 'product_id': '0x3a34', 'parent': 'computer', 'address': {'slot': '29', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '14', 'vendor_id': '0x8086', 'driver': 'uhci_hcd'}}, 'pci_0000_00_1d_7': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB2 EHCI Controller #1', 'vendor': 'Intel Corporation', 'product_id': '0x3a3a', 'parent': 'computer', 'address': {'slot': '29', 'bus': '0', 'domain': '0', 'function': '7'}, 'iommu_group': '14', 'vendor_id': '0x8086', 'driver': 'ehci-pci'}}, 'scsi_0_2_1_0': {'params': {'product': 'ServeRAID M5015', 'vendor': 'IBM', 'parent': 'scsi_target0_2_1', 'driver': 'sd', 'capability': 'scsi', 'is_assignable': 'true', 'udev_path': '/dev/sg1', 'address': {'bus': '2', 'host': '0', 'target': '1', 'lun': '0'}}}, 'pci_0000_00_15_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 Trusted Execution Technology Registers', 'vendor': 'Intel Corporation', 'product_id': '0x342f', 'parent': 'computer', 'address': {'slot': '21', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '10', 'vendor_id': '0x8086'}}, 'usb_2_5_1_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'usbhid', 'parent': 'usb_2_5_1'}}, 'block_sda_3600605b002f69270217fb60a16a05ccf': {'params': {'capability': 'storage', 'is_assignable': 'true', 'product': 'ServeRAID M5015', 'vendor': 'IBM', 'parent': 'scsi_0_2_0_0'}}, 'pci_0000_10_00_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'NetXtreme II BCM5709 Gigabit Ethernet', 'vendor': 'Broadcom Inc. and subsidiaries', 'product_id': '0x1639', 'parent': 'pci_0000_00_02_0', 'address': {'slot': '0', 'bus': '16', 'domain': '0', 'function': '1'}, 'iommu_group': '18', 'vendor_id': '0x14e4', 'driver': 'bnx2'}}, 'usb_usb2': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '2.0 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0002', 'parent': 'pci_0000_00_1d_7', 'address': {'device': '1', 'bus': '2'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'pci_0000_fe_06_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control', 'vendor': 'Intel Corporation', 'product_id': '0x2db3', 'parent': 'computer', 'address': {'slot': '6', 'bus': '254', 'domain': '0', 'function': '3'}, 'iommu_group': '29', 'vendor_id': '0x8086'}}, 'pci_0000_fe_06_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Rank', 'vendor': 'Intel Corporation', 'product_id': '0x2db2', 'parent': 'computer', 'address': {'slot': '6', 'bus': '254', 'domain': '0', 'function': '2'}, 'iommu_group': '29', 'vendor_id': '0x8086'}}, 'pci_0000_fe_06_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Address', 'vendor': 'Intel Corporation', 'product_id': '0x2db1', 'parent': 'computer', 'address': {'slot': '6', 'bus': '254', 'domain': '0', 'function': '1'}, 'iommu_group': '29', 'vendor_id': '0x8086'}}, 'pci_0000_00_11_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500 Routing & Protocol Layer Register Port 1', 'vendor': 'Intel Corporation', 'product_id': '0x3428', 'parent': 'computer', 'address': {'slot': '17', 'bus': '0', 'domain': '0', 'function': '1'}, 'iommu_group': '8', 'vendor_id': '0x8086'}}, 'usb_2_6_4': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '4-Port USB 2.0 Hub', 'vendor': 'Realtek Semiconductor Corp.', 'product_id': '0x5411', 'parent': 'usb_2_6', 'address': {'device': '6', 'bus': '2'}, 'vendor_id': '0x0bda', 'driver': 'usb'}}, 'net_ens3f1_00_1b_21_8a_a6_c9': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'ens3f1', 'parent': 'pci_0000_24_00_1'}}, 'net_enp0s29f0u2_e6_1f_13_a8_94_17': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'enp0s29f0u2', 'parent': 'usb_5_2_1_0'}}, 'pci_0000_00_1f_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JIB (ICH10) LPC Interface Controller', 'vendor': 'Intel Corporation', 'product_id': '0x3a18', 'parent': 'computer', 'address': {'slot': '31', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '16', 'vendor_id': '0x8086', 'driver': 'lpc_ich'}}, 'pci_0000_00_1f_5': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) 2 port SATA IDE Controller #2', 'vendor': 'Intel Corporation', 'product_id': '0x3a26', 'parent': 'computer', 'address': {'slot': '31', 'bus': '0', 'domain': '0', 'function': '5'}, 'iommu_group': '16', 'vendor_id': '0x8086', 'driver': 'ata_piix'}}, 'pci_0000_24_00_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82580 Gigabit Network Connection', 'vendor': 'Intel Corporation', 'product_id': '0x150e', 'parent': 'pci_0000_00_09_0', 'address': {'slot': '0', 'bus': '36', 'domain': '0', 'function': '2'}, 'iommu_group': '22', 'vendor_id': '0x8086', 'driver': 'igb'}}, 'pci_0000_24_00_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82580 Gigabit Network Connection', 'vendor': 'Intel Corporation', 'product_id': '0x150e', 'parent': 'pci_0000_00_09_0', 'address': {'slot': '0', 'bus': '36', 'domain': '0', 'function': '3'}, 'iommu_group': '23', 'vendor_id': '0x8086', 'driver': 'igb'}}, 'pci_0000_00_1f_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) SMBus Controller', 'vendor': 'Intel Corporation', 'product_id': '0x3a30', 'parent': 'computer', 'address': {'slot': '31', 'bus': '0', 'domain': '0', 'function': '3'}, 'iommu_group': '16', 'vendor_id': '0x8086', 'driver': 'i801_smbus'}}, 'pci_0000_00_1f_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) 4 port SATA IDE Controller #1', 'vendor': 'Intel Corporation', 'product_id': '0x3a20', 'parent': 'computer', 'address': {'slot': '31', 'bus': '0', 'domain': '0', 'function': '2'}, 'iommu_group': '16', 'vendor_id': '0x8086', 'driver': 'ata_piix'}}, 'pci_0000_24_00_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82580 Gigabit Network Connection', 'vendor': 'Intel Corporation', 'product_id': '0x150e', 'parent': 'pci_0000_00_09_0', 'address': {'slot': '0', 'bus': '36', 'domain': '0', 'function': '1'}, 'iommu_group': '21', 'vendor_id': '0x8086', 'driver': 'igb'}}, 'drm_card0': {'params': {'capability': 'drm', 'is_assignable': 'true', 'parent': 'pci_0000_07_00_0'}}, 'pci_0000_00_1e_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '82801 PCI Bridge', 'vendor': 'Intel Corporation', 'product_id': '0x244e', 'parent': 'computer', 'address': {'slot': '30', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '15', 'vendor_id': '0x8086'}}, 'computer': {'params': {'capability': 'system', 'is_assignable': 'true', 'product': 'System x3650 M3 -[7945L2G]-'}}, 'pci_0000_07_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'MGA G200EV', 'vendor': 'Matrox Electronics Systems Ltd.', 'product_id': '0x0530', 'parent': 'pci_0000_06_00_0', 'address': {'slot': '0', 'bus': '7', 'domain': '0', 'function': '0'}, 'iommu_group': '13', 'vendor_id': '0x102b', 'driver': 'mgag200'}}, 'pci_0000_00_1a_7': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB2 EHCI Controller #2', 'vendor': 'Intel Corporation', 'product_id': '0x3a3c', 'parent': 'computer', 'address': {'slot': '26', 'bus': '0', 'domain': '0', 'function': '7'}, 'iommu_group': '12', 'vendor_id': '0x8086', 'driver': 'ehci-pci'}}, 'pci_0000_00_1a_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB UHCI Controller #4', 'vendor': 'Intel Corporation', 'product_id': '0x3a37', 'parent': 'computer', 'address': {'slot': '26', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '12', 'vendor_id': '0x8086', 'driver': 'uhci_hcd'}}, 'pci_0000_00_1a_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB UHCI Controller #5', 'vendor': 'Intel Corporation', 'product_id': '0x3a38', 'parent': 'computer', 'address': {'slot': '26', 'bus': '0', 'domain': '0', 'function': '1'}, 'iommu_group': '12', 'vendor_id': '0x8086', 'driver': 'uhci_hcd'}}, 'pci_0000_ff_05_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Address', 'vendor': 'Intel Corporation', 'product_id': '0x2da9', 'parent': 'computer', 'address': {'slot': '5', 'bus': '255', 'domain': '0', 'function': '1'}, 'iommu_group': '34', 'vendor_id': '0x8086'}}, 'pci_0000_ff_05_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Control', 'vendor': 'Intel Corporation', 'product_id': '0x2da8', 'parent': 'computer', 'address': {'slot': '5', 'bus': '255', 'domain': '0', 'function': '0'}, 'iommu_group': '34', 'vendor_id': '0x8086'}}, 'pci_0000_ff_05_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control', 'vendor': 'Intel Corporation', 'product_id': '0x2dab', 'parent': 'computer', 'address': {'slot': '5', 'bus': '255', 'domain': '0', 'function': '3'}, 'iommu_group': '34', 'vendor_id': '0x8086'}}, 'pci_0000_ff_05_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Rank', 'vendor': 'Intel Corporation', 'product_id': '0x2daa', 'parent': 'computer', 'address': {'slot': '5', 'bus': '255', 'domain': '0', 'function': '2'}, 'iommu_group': '34', 'vendor_id': '0x8086'}}, 'pci_0000_fe_04_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Address', 'vendor': 'Intel Corporation', 'product_id': '0x2da1', 'parent': 'computer', 'address': {'slot': '4', 'bus': '254', 'domain': '0', 'function': '1'}, 'iommu_group': '27', 'vendor_id': '0x8086'}}, 'usb_3_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb3'}}, 'net_eno3_e4_1f_13_b5_94_16': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'eno3', 'parent': 'pci_0000_10_00_0'}}, 'pci_0000_00_1c_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '82801JI (ICH10 Family) PCI Express Root Port 1', 'vendor': 'Intel Corporation', 'product_id': '0x3a40', 'parent': 'computer', 'address': {'slot': '28', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '13', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'pci_0000_00_1c_4': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '82801JI (ICH10 Family) PCI Express Root Port 5', 'vendor': 'Intel Corporation', 'product_id': '0x3a48', 'parent': 'computer', 'address': {'slot': '28', 'bus': '0', 'domain': '0', 'function': '4'}, 'iommu_group': '13', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'usb_6_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb6'}}, 'pci_0000_ff_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QuickPath Architecture Generic Non-core Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2c70', 'parent': 'computer', 'address': {'slot': '0', 'bus': '255', 'domain': '0', 'function': '0'}, 'iommu_group': '30', 'vendor_id': '0x8086'}}, 'pci_0000_ff_00_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QuickPath Architecture System Address Decoder', 'vendor': 'Intel Corporation', 'product_id': '0x2d81', 'parent': 'computer', 'address': {'slot': '0', 'bus': '255', 'domain': '0', 'function': '1'}, 'iommu_group': '30', 'vendor_id': '0x8086'}}, 'net_eno4_5c_f3_fc_39_18_de': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'eno4', 'parent': 'pci_0000_10_00_1'}}, 'net_ens3f3_00_1b_21_8a_a6_cb': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'ens3f3', 'parent': 'pci_0000_24_00_3'}}, 'usb_4_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb4'}}, 'pci_0000_fe_04_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Control', 'vendor': 'Intel Corporation', 'product_id': '0x2da0', 'parent': 'computer', 'address': {'slot': '4', 'bus': '254', 'domain': '0', 'function': '0'}, 'iommu_group': '27', 'vendor_id': '0x8086'}}, 'pci_0000_fe_04_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control', 'vendor': 'Intel Corporation', 'product_id': '0x2da3', 'parent': 'computer', 'address': {'slot': '4', 'bus': '254', 'domain': '0', 'function': '3'}, 'iommu_group': '27', 'vendor_id': '0x8086'}}, 'pci_0000_fe_04_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Rank', 'vendor': 'Intel Corporation', 'product_id': '0x2da2', 'parent': 'computer', 'address': {'slot': '4', 'bus': '254', 'domain': '0', 'function': '2'}, 'iommu_group': '27', 'vendor_id': '0x8086'}}, 'usb_2_5_3_1_1_1': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'usbhid', 'parent': 'usb_2_5_3_1'}}, 'usb_1_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb1'}}, 'usb_2_5_3_1_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'usbhid', 'parent': 'usb_2_5_3_1'}}, 'pci_0000_ff_06_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Rank', 'vendor': 'Intel Corporation', 'product_id': '0x2db2', 'parent': 'computer', 'address': {'slot': '6', 'bus': '255', 'domain': '0', 'function': '2'}, 'iommu_group': '35', 'vendor_id': '0x8086'}}, 'pci_0000_ff_06_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control', 'vendor': 'Intel Corporation', 'product_id': '0x2db3', 'parent': 'computer', 'address': {'slot': '6', 'bus': '255', 'domain': '0', 'function': '3'}, 'iommu_group': '35', 'vendor_id': '0x8086'}}, 'pci_0000_ff_06_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Control', 'vendor': 'Intel Corporation', 'product_id': '0x2db0', 'parent': 'computer', 'address': {'slot': '6', 'bus': '255', 'domain': '0', 'function': '0'}, 'iommu_group': '35', 'vendor_id': '0x8086'}}, 'pci_0000_ff_06_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Address', 'vendor': 'Intel Corporation', 'product_id': '0x2db1', 'parent': 'computer', 'address': {'slot': '6', 'bus': '255', 'domain': '0', 'function': '1'}, 'iommu_group': '35', 'vendor_id': '0x8086'}}, 'pci_0000_fe_02_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Mirror Port Link 1', 'vendor': 'Intel Corporation', 'product_id': '0x2d93', 'parent': 'computer', 'address': {'slot': '2', 'bus': '254', 'domain': '0', 'function': '3'}, 'iommu_group': '25', 'vendor_id': '0x8086'}}, 'pci_0000_fe_02_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Mirror Port Link 0', 'vendor': 'Intel Corporation', 'product_id': '0x2d92', 'parent': 'computer', 'address': {'slot': '2', 'bus': '254', 'domain': '0', 'function': '2'}, 'iommu_group': '25', 'vendor_id': '0x8086'}}, 'pci_0000_fe_02_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Physical 0', 'vendor': 'Intel Corporation', 'product_id': '0x2d91', 'parent': 'computer', 'address': {'slot': '2', 'bus': '254', 'domain': '0', 'function': '1'}, 'iommu_group': '25', 'vendor_id': '0x8086'}}, 'pci_0000_fe_02_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Link 0', 'vendor': 'Intel Corporation', 'product_id': '0x2d90', 'parent': 'computer', 'address': {'slot': '2', 'bus': '254', 'domain': '0', 'function': '0'}, 'iommu_group': '25', 'vendor_id': '0x8086'}}, 'pci_0000_fe_02_5': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Physical 1', 'vendor': 'Intel Corporation', 'product_id': '0x2d95', 'parent': 'computer', 'address': {'slot': '2', 'bus': '254', 'domain': '0', 'function': '5'}, 'iommu_group': '25', 'vendor_id': '0x8086'}}, 'net_ens3f2_00_1b_21_8a_a6_ca': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'ens3f2', 'parent': 'pci_0000_24_00_2'}}, 'scsi_0_2_0_0': {'params': {'product': 'ServeRAID M5015', 'vendor': 'IBM', 'parent': 'scsi_target0_2_0', 'driver': 'sd', 'capability': 'scsi', 'is_assignable': 'true', 'udev_path': '/dev/sg0', 'address': {'bus': '2', 'host': '0', 'target': '0', 'lun': '0'}}}, 'usb_5_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb5'}}, 'pci_0000_ff_02_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Mirror Port Link 0', 'vendor': 'Intel Corporation', 'product_id': '0x2d92', 'parent': 'computer', 'address': {'slot': '2', 'bus': '255', 'domain': '0', 'function': '2'}, 'iommu_group': '31', 'vendor_id': '0x8086'}}, 'pci_0000_00_01_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '5520/5500/X58 I/O Hub PCI Express Root Port 1', 'vendor': 'Intel Corporation', 'product_id': '0x3408', 'parent': 'computer', 'address': {'slot': '1', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '1', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'pci_0000_ff_02_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Mirror Port Link 1', 'vendor': 'Intel Corporation', 'product_id': '0x2d93', 'parent': 'computer', 'address': {'slot': '2', 'bus': '255', 'domain': '0', 'function': '3'}, 'iommu_group': '31', 'vendor_id': '0x8086'}}, 'pci_0000_0b_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'NetXtreme II BCM5709 Gigabit Ethernet', 'vendor': 'Broadcom Inc. and subsidiaries', 'product_id': '0x1639', 'parent': 'pci_0000_00_01_0', 'address': {'slot': '0', 'bus': '11', 'domain': '0', 'function': '0'}, 'iommu_group': '17', 'vendor_id': '0x14e4', 'driver': 'bnx2'}}, 'usb_7_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb7'}}, 'usb_2_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb2'}}, 'pci_0000_0b_00_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'NetXtreme II BCM5709 Gigabit Ethernet', 'vendor': 'Broadcom Inc. and subsidiaries', 'product_id': '0x1639', 'parent': 'pci_0000_00_01_0', 'address': {'slot': '0', 'bus': '11', 'domain': '0', 'function': '1'}, 'iommu_group': '17', 'vendor_id': '0x14e4', 'driver': 'bnx2'}}, 'pci_0000_00_14_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 I/O Hub Throttle Registers', 'vendor': 'Intel Corporation', 'product_id': '0x3438', 'parent': 'computer', 'address': {'slot': '20', 'bus': '0', 'domain': '0', 'function': '3'}, 'iommu_group': '9', 'vendor_id': '0x8086'}}, 'pci_0000_00_14_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 I/O Hub Control Status and RAS Registers', 'vendor': 'Intel Corporation', 'product_id': '0x3423', 'parent': 'computer', 'address': {'slot': '20', 'bus': '0', 'domain': '0', 'function': '2'}, 'iommu_group': '9', 'vendor_id': '0x8086'}}, 'pci_0000_00_14_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers', 'vendor': 'Intel Corporation', 'product_id': '0x3422', 'parent': 'computer', 'address': {'slot': '20', 'bus': '0', 'domain': '0', 'function': '1'}, 'iommu_group': '9', 'vendor_id': '0x8086'}}, 'pci_0000_00_14_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 I/O Hub System Management Registers', 'vendor': 'Intel Corporation', 'product_id': '0x342e', 'parent': 'computer', 'address': {'slot': '20', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '9', 'vendor_id': '0x8086', 'driver': 'i7core_edac'}}, 'pci_0000_ff_04_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Control', 'vendor': 'Intel Corporation', 'product_id': '0x2da0', 'parent': 'computer', 'address': {'slot': '4', 'bus': '255', 'domain': '0', 'function': '0'}, 'iommu_group': '33', 'vendor_id': '0x8086'}}, 'pci_0000_ff_04_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Address', 'vendor': 'Intel Corporation', 'product_id': '0x2da1', 'parent': 'computer', 'address': {'slot': '4', 'bus': '255', 'domain': '0', 'function': '1'}, 'iommu_group': '33', 'vendor_id': '0x8086'}}, 'pci_0000_ff_04_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Rank', 'vendor': 'Intel Corporation', 'product_id': '0x2da2', 'parent': 'computer', 'address': {'slot': '4', 'bus': '255', 'domain': '0', 'function': '2'}, 'iommu_group': '33', 'vendor_id': '0x8086'}}, 'pci_0000_ff_04_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control', 'vendor': 'Intel Corporation', 'product_id': '0x2da3', 'parent': 'computer', 'address': {'slot': '4', 'bus': '255', 'domain': '0', 'function': '3'}, 'iommu_group': '33', 'vendor_id': '0x8086'}}, 'pci_0000_00_05_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '5520/X58 I/O Hub PCI Express Root Port 5', 'vendor': 'Intel Corporation', 'product_id': '0x340c', 'parent': 'computer', 'address': {'slot': '5', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '4', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'pci_0000_ff_02_4': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Link 1', 'vendor': 'Intel Corporation', 'product_id': '0x2d94', 'parent': 'computer', 'address': {'slot': '2', 'bus': '255', 'domain': '0', 'function': '4'}, 'iommu_group': '31', 'vendor_id': '0x8086'}}, 'pci_0000_ff_02_5': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Physical 1', 'vendor': 'Intel Corporation', 'product_id': '0x2d95', 'parent': 'computer', 'address': {'slot': '2', 'bus': '255', 'domain': '0', 'function': '5'}, 'iommu_group': '31', 'vendor_id': '0x8086'}}, 'pci_0000_fe_00_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QuickPath Architecture System Address Decoder', 'vendor': 'Intel Corporation', 'product_id': '0x2d81', 'parent': 'computer', 'address': {'slot': '0', 'bus': '254', 'domain': '0', 'function': '1'}, 'iommu_group': '24', 'vendor_id': '0x8086'}}, 'pci_0000_fe_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QuickPath Architecture Generic Non-core Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2c70', 'parent': 'computer', 'address': {'slot': '0', 'bus': '254', 'domain': '0', 'function': '0'}, 'iommu_group': '24', 'vendor_id': '0x8086'}}, 'pci_0000_ff_02_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Link 0', 'vendor': 'Intel Corporation', 'product_id': '0x2d90', 'parent': 'computer', 'address': {'slot': '2', 'bus': '255', 'domain': '0', 'function': '0'}, 'iommu_group': '31', 'vendor_id': '0x8086'}}, 'pci_0000_ff_02_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Physical 0', 'vendor': 'Intel Corporation', 'product_id': '0x2d91', 'parent': 'computer', 'address': {'slot': '2', 'bus': '255', 'domain': '0', 'function': '1'}, 'iommu_group': '31', 'vendor_id': '0x8086'}}, 'pci_0000_00_16_5': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x342a', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '5'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_00_16_4': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x3429', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '4'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_00_16_7': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x342c', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '7'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_00_16_6': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x342b', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '6'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_00_16_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x3431', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '1'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_00_16_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x3430', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_00_16_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x3433', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '3'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_01_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'MegaRAID SAS 2108 [Liberator]', 'vendor': 'LSI Logic / Symbios Logic', 'product_id': '0x0079', 'parent': 'pci_0000_00_1c_0', 'address': {'slot': '0', 'bus': '1', 'domain': '0', 'function': '0'}, 'iommu_group': '13', 'vendor_id': '0x1000', 'driver': 'megaraid_sas'}}, 'scsi_target0_2_1': {'params': {'capability': 'scsi_target', 'is_assignable': 'true', 'parent': 'scsi_host0'}}, 'pci_0000_00_03_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '5520/5500/X58 I/O Hub PCI Express Root Port 3', 'vendor': 'Intel Corporation', 'product_id': '0x340a', 'parent': 'computer', 'address': {'slot': '3', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '3', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'pci_0000_fe_05_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Control', 'vendor': 'Intel Corporation', 'product_id': '0x2da8', 'parent': 'computer', 'address': {'slot': '5', 'bus': '254', 'domain': '0', 'function': '0'}, 'iommu_group': '28', 'vendor_id': '0x8086'}}, 'pci_0000_fe_05_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Address', 'vendor': 'Intel Corporation', 'product_id': '0x2da9', 'parent': 'computer', 'address': {'slot': '5', 'bus': '254', 'domain': '0', 'function': '1'}, 'iommu_group': '28', 'vendor_id': '0x8086'}}, 'pci_0000_fe_05_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Rank', 'vendor': 'Intel Corporation', 'product_id': '0x2daa', 'parent': 'computer', 'address': {'slot': '5', 'bus': '254', 'domain': '0', 'function': '2'}, 'iommu_group': '28', 'vendor_id': '0x8086'}}, 'pci_0000_fe_05_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control', 'vendor': 'Intel Corporation', 'product_id': '0x2dab', 'parent': 'computer', 'address': {'slot': '5', 'bus': '254', 'domain': '0', 'function': '3'}, 'iommu_group': '28', 'vendor_id': '0x8086'}}, 'pci_0000_00_11_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500 Physical and Link Layer Registers Port 1', 'vendor': 'Intel Corporation', 'product_id': '0x3427', 'parent': 'computer', 'address': {'slot': '17', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '8', 'vendor_id': '0x8086'}}, 'scsi_generic_sg1': {'params': {'capability': 'scsi_generic', 'is_assignable': 'true', 'udev_path': '/dev/sg1', 'parent': 'scsi_0_2_1_0'}}, 'scsi_generic_sg0': {'params': {'capability': 'scsi_generic', 'is_assignable': 'true', 'udev_path': '/dev/sg0', 'parent': 'scsi_0_2_0_0'}}, 'pci_0000_00_16_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x3432', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '2'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_fe_03_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller RAS Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2d9a', 'parent': 'computer', 'address': {'slot': '3', 'bus': '254', 'domain': '0', 'function': '2'}, 'iommu_group': '26', 'vendor_id': '0x8086'}}, 'pci_0000_fe_03_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2d98', 'parent': 'computer', 'address': {'slot': '3', 'bus': '254', 'domain': '0', 'function': '0'}, 'iommu_group': '26', 'vendor_id': '0x8086'}}, 'pci_0000_fe_03_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Target Address Decoder', 'vendor': 'Intel Corporation', 'product_id': '0x2d99', 'parent': 'computer', 'address': {'slot': '3', 'bus': '254', 'domain': '0', 'function': '1'}, 'iommu_group': '26', 'vendor_id': '0x8086'}}, 'pci_0000_fe_03_4': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Test Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2d9c', 'parent': 'computer', 'address': {'slot': '3', 'bus': '254', 'domain': '0', 'function': '4'}, 'iommu_group': '26', 'vendor_id': '0x8086'}}, 'usb_usb7': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '1.1 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0001', 'parent': 'pci_0000_00_1d_2', 'address': {'device': '1', 'bus': '7'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'usb_usb6': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '1.1 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0001', 'parent': 'pci_0000_00_1d_1', 'address': {'device': '1', 'bus': '6'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'usb_usb5': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '1.1 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0001', 'parent': 'pci_0000_00_1d_0', 'address': {'device': '1', 'bus': '5'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'usb_usb4': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '1.1 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0001', 'parent': 'pci_0000_00_1a_1', 'address': {'device': '1', 'bus': '4'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'usb_usb3': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '1.1 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0001', 'parent': 'pci_0000_00_1a_0', 'address': {'device': '1', 'bus': '3'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'pci_0000_10_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'NetXtreme II BCM5709 Gigabit Ethernet', 'vendor': 'Broadcom Inc. and subsidiaries', 'product_id': '0x1639', 'parent': 'pci_0000_00_02_0', 'address': {'slot': '0', 'bus': '16', 'domain': '0', 'function': '0'}, 'iommu_group': '18', 'vendor_id': '0x14e4', 'driver': 'bnx2'}}, 'usb_usb1': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '2.0 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0002', 'parent': 'pci_0000_00_1a_7', 'address': {'device': '1', 'bus': '1'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'usb_2_5_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_2_5'}}, 'usb_2_5_3_1': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'CS682 2-Port USB 2.0 DVI KVM Switch', 'vendor': 'ATEN International Co., Ltd', 'product_id': '0x2213', 'parent': 'usb_2_5_3', 'address': {'device': '9', 'bus': '2'}, 'vendor_id': '0x0557', 'driver': 'usb'}}, 'usb_2_6': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'Hub', 'vendor': 'Terminus Technology Inc.', 'product_id': '0x0101', 'parent': 'usb_usb2', 'address': {'device': '4', 'bus': '2'}, 'vendor_id': '0x1a40', 'driver': 'usb'}}, 'usb_2_5': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'Hub', 'vendor': 'Genesys Logic, Inc.', 'product_id': '0x0608', 'parent': 'usb_usb2', 'address': {'device': '3', 'bus': '2'}, 'vendor_id': '0x05e3', 'driver': 'usb'}}, 'usb_2_5_1': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'Uninterruptible Power Supply', 'vendor': 'American Power Conversion', 'product_id': '0x0002', 'parent': 'usb_2_5', 'address': {'device': '5', 'bus': '2'}, 'vendor_id': '0x051d', 'driver': 'usb'}}, 'pci_0000_fe_06_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Control', 'vendor': 'Intel Corporation', 'product_id': '0x2db0', 'parent': 'computer', 'address': {'slot': '6', 'bus': '254', 'domain': '0', 'function': '0'}, 'iommu_group': '29', 'vendor_id': '0x8086'}}, 'usb_2_5_3': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'Hub', 'vendor': 'ATEN International Co., Ltd', 'product_id': '0x8021', 'parent': 'usb_2_5', 'address': {'device': '8', 'bus': '2'}, 'vendor_id': '0x0557', 'driver': 'usb'}}, 'usb_2_5_2': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'HASP copy protection dongle', 'vendor': 'Aladdin Knowledge Systems', 'product_id': '0x0001', 'parent': 'usb_2_5', 'address': {'device': '7', 'bus': '2'}, 'vendor_id': '0x0529', 'driver': 'usb'}}, 'pci_0000_00_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520 I/O Hub to ESI Port', 'vendor': 'Intel Corporation', 'product_id': '0x3406', 'parent': 'computer', 'address': {'slot': '0', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '0', 'vendor_id': '0x8086'}}, 'pci_0000_00_09_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '7500/5520/5500/X58 I/O Hub PCI Express Root Port 9', 'vendor': 'Intel Corporation', 'product_id': '0x3410', 'parent': 'computer', 'address': {'slot': '9', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '6', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'pci_0000_1f_00_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'ISP2532-based 8Gb Fibre Channel to PCI Express HBA', 'vendor': 'QLogic Corp.', 'product_id': '0x2532', 'parent': 'pci_0000_00_07_0', 'address': {'slot': '0', 'bus': '31', 'domain': '0', 'function': '1'}, 'iommu_group': '19', 'vendor_id': '0x1077', 'driver': 'qla2xxx'}}, 'pci_0000_1f_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'ISP2532-based 8Gb Fibre Channel to PCI Express HBA', 'vendor': 'QLogic Corp.', 'product_id': '0x2532', 'parent': 'pci_0000_00_07_0', 'address': {'slot': '0', 'bus': '31', 'domain': '0', 'function': '0'}, 'iommu_group': '19', 'vendor_id': '0x1077', 'driver': 'qla2xxx'}}, 'pci_0000_24_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82580 Gigabit Network Connection', 'vendor': 'Intel Corporation', 'product_id': '0x150e', 'parent': 'pci_0000_00_09_0', 'address': {'slot': '0', 'bus': '36', 'domain': '0', 'function': '0'}, 'iommu_group': '20', 'vendor_id': '0x8086', 'driver': 'igb'}}, 'usb_2_5_2_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'parent': 'usb_2_5_2'}}, 'pci_0000_00_07_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '5520/5500/X58 I/O Hub PCI Express Root Port 7', 'vendor': 'Intel Corporation', 'product_id': '0x340e', 'parent': 'computer', 'address': {'slot': '7', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '5', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'net_eno2_e4_1f_13_b5_94_16': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'eno2', 'parent': 'pci_0000_0b_00_1'}}, 'usb_2_5_3_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_2_5_3'}}, 'usb_5_2_1_1': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'cdc_ether', 'parent': 'usb_5_2'}}, 'usb_5_2_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'cdc_ether', 'parent': 'usb_5_2'}}, 'pci_0000_fe_02_4': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Link 1', 'vendor': 'Intel Corporation', 'product_id': '0x2d94', 'parent': 'computer', 'address': {'slot': '2', 'bus': '254', 'domain': '0', 'function': '4'}, 'iommu_group': '25', 'vendor_id': '0x8086'}}, 'usb_2_6_4_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_2_6_4'}}, 'pci_0000_00_02_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '5520/5500/X58 I/O Hub PCI Express Root Port 2', 'vendor': 'Intel Corporation', 'product_id': '0x3409', 'parent': 'computer', 'address': {'slot': '2', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '2', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'pci_0000_ff_03_4': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Test Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2d9c', 'parent': 'computer', 'address': {'slot': '3', 'bus': '255', 'domain': '0', 'function': '4'}, 'iommu_group': '32', 'vendor_id': '0x8086'}}, 'pci_0000_ff_03_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller RAS Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2d9a', 'parent': 'computer', 'address': {'slot': '3', 'bus': '255', 'domain': '0', 'function': '2'}, 'iommu_group': '32', 'vendor_id': '0x8086'}}, 'pci_0000_ff_03_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Target Address Decoder', 'vendor': 'Intel Corporation', 'product_id': '0x2d99', 'parent': 'computer', 'address': {'slot': '3', 'bus': '255', 'domain': '0', 'function': '1'}, 'iommu_group': '32', 'vendor_id': '0x8086'}}, 'pci_0000_ff_03_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2d98', 'parent': 'computer', 'address': {'slot': '3', 'bus': '255', 'domain': '0', 'function': '0'}, 'iommu_group': '32', 'vendor_id': '0x8086'}}, 'pci_0000_00_10_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 Routing and Protocol Layer Registers Port 0', 'vendor': 'Intel Corporation', 'product_id': '0x3426', 'parent': 'computer', 'address': {'slot': '16', 'bus': '0', 'domain': '0', 'function': '1'}, 'iommu_group': '7', 'vendor_id': '0x8086'}}, 'pci_0000_00_10_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 Physical and Link Layer Registers Port 0', 'vendor': 'Intel Corporation', 'product_id': '0x3425', 'parent': 'computer', 'address': {'slot': '16', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '7', 'vendor_id': '0x8086'}}, 'pci_0000_06_00_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': 'VSC452 [SuperBMC]', 'vendor': 'Vitesse Semiconductor', 'product_id': '0x0452', 'parent': 'pci_0000_00_1c_4', 'address': {'slot': '0', 'bus': '6', 'domain': '0', 'function': '0'}, 'iommu_group': '13', 'vendor_id': '0x101b'}}, 'scsi_host3': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_00_1f_5'}}, 'scsi_host2': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_00_1f_2'}}, 'scsi_host1': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_00_1f_2'}}, 'scsi_host0': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_01_00_0'}}, 'scsi_host6': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_1f_00_1'}}, 'scsi_host5': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_1f_00_0'}}, 'scsi_host4': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_00_1f_5'}}, 'usb_2_6_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_2_6'}}, 'scsi_target0_2_0': {'params': {'capability': 'scsi_target', 'is_assignable': 'true', 'parent': 'scsi_host0'}}, 'net_lo_00_00_00_00_00_00': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'lo', 'parent': 'computer'}}, 'net_ens3f0_00_1b_21_8a_a6_c8': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'ens3f0', 'parent': 'pci_0000_24_00_0'}}, 'usb_5_2': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'RNDIS/CDC ETHER', 'vendor': 'IBM Corp.', 'product_id': '0x4010', 'parent': 'usb_usb5', 'address': {'device': '2', 'bus': '5'}, 'vendor_id': '0x04b3', 'driver': 'usb'}}, 'block_sdb_3600605b002f6927021946993801eb9a6': {'params': {'capability': 'storage', 'is_assignable': 'true', 'product': 'ServeRAID M5015', 'vendor': 'IBM', 'parent': 'scsi_0_2_1_0'}}}} from=::ffff:192.168.0.19,51130, flow_id=7e5fcf82 (api:54) 2019-04-13 21:40:45,498+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.hostdevListByCaps succeeded in 0.12 seconds (__init__:312) 2019-04-13 21:40:45,596+0200 INFO (vmrecovery) [vdsm.api] START getConnectedStoragePoolsList(options=None) from=internal, task_id=41ee77f1-e6b5-4f80-90f3-04f076fa80cf (api:48) 2019-04-13 21:40:45,596+0200 INFO (vmrecovery) [vdsm.api] FINISH getConnectedStoragePoolsList return={'poollist': ['59ef3a18-002f-02d1-0220-000000000124']} from=internal, task_id=41ee77f1-e6b5-4f80-90f3-04f076fa80cf (api:54) 2019-04-13 21:40:45,596+0200 INFO (vmrecovery) [vds] recovery [1/16]: preparing paths for domain be42701b-3f74-4c81-b6ef-5f7e7806554a (clientIF:722) 2019-04-13 21:40:45,598+0200 INFO (vmrecovery) [vds] prepared volume path: (clientIF:499) 2019-04-13 21:40:45,598+0200 INFO (vmrecovery) [vdsm.api] START prepareImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='04dc2b89-95cd-49d4-aee2-6927359e8ca1', leafUUID='ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', allowIllegal=False) from=internal, task_id=8d782509-b5be-4d89-a657-2f8489d3014d (api:48) 2019-04-13 21:40:45,965+0200 INFO (vmrecovery) [vdsm.api] FINISH prepareImage error=Cannot prepare illegal volume: (u'149564f0-7406-4bed-a091-8d965f603283',) from=internal, task_id=8d782509-b5be-4d89-a657-2f8489d3014d (api:52) 2019-04-13 21:40:45,965+0200 ERROR (vmrecovery) [storage.TaskManager.Task] (Task='8d782509-b5be-4d89-a657-2f8489d3014d') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in prepareImage File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3204, in prepareImage raise se.prepareIllegalVolumeError(volUUID) prepareIllegalVolumeError: Cannot prepare illegal volume: (u'149564f0-7406-4bed-a091-8d965f603283',) 2019-04-13 21:40:45,965+0200 INFO (vmrecovery) [storage.TaskManager.Task] (Task='8d782509-b5be-4d89-a657-2f8489d3014d') aborting: Task is aborted: "Cannot prepare illegal volume: (u'149564f0-7406-4bed-a091-8d965f603283',)" - code 227 (task:1181) 2019-04-13 21:40:45,965+0200 ERROR (vmrecovery) [storage.Dispatcher] FINISH prepareImage error=Cannot prepare illegal volume: (u'149564f0-7406-4bed-a091-8d965f603283',) (dispatcher:83) 2019-04-13 21:40:45,965+0200 ERROR (vmrecovery) [vds] recovery [1/16]: failed for vm be42701b-3f74-4c81-b6ef-5f7e7806554a (clientIF:727) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 723, in _preparePathsForRecoveredVMs vm_obj.preparePaths() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1003, in preparePaths self._preparePathsForDrives(drives) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1011, in _preparePathsForDrives drive['path'] = self.cif.prepareVolumePath(drive, self.id) File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 415, in prepareVolumePath raise vm.VolumeError(drive) VolumeError: Bad volume specification {'index': 0, 'domainID': '9c3f06cf-7475-448e-819b-f4f52fa7d782', 'name': 'sda', 'alias': 'ua-04dc2b89-95cd-49d4-aee2-6927359e8ca1', 'bootOrder': '1', 'format': 'raw', 'poolID': '59ef3a18-002f-02d1-0220-000000000124', 'volumeID': 'ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', 'diskType': 'file', 'discard': False, 'specParams': {}, 'iface': 'scsi', 'cache': 'none', 'propagateErrors': 'off', 'address': {'bus': '0', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'device': 'disk', 'path': '/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/04dc2b89-95cd-49d4-aee2-6927359e8ca1/ff358cbb-ba3d-4fe5-8ad9-7ca4406489a1', 'serial': '04dc2b89-95cd-49d4-aee2-6927359e8ca1', 'imageID': '04dc2b89-95cd-49d4-aee2-6927359e8ca1', 'type': 'disk'} 2019-04-13 21:40:45,967+0200 INFO (vmrecovery) [vds] recovery [2/16]: preparing paths for domain 28d7d5ff-2ca1-42c6-ae8d-065c721d3066 (clientIF:722) 2019-04-13 21:40:45,968+0200 INFO (vmrecovery) [vds] prepared volume path: (clientIF:499) 2019-04-13 21:40:45,968+0200 INFO (vmrecovery) [vdsm.api] START prepareImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', leafUUID='123e2cc8-8f24-4ddd-b401-68393535f555', allowIllegal=False) from=internal, task_id=a79e5812-fa01-49fd-8f53-addd4ef9ab2f (api:48) 2019-04-13 21:40:45,997+0200 INFO (vmrecovery) [storage.StorageDomain] Fixing permissions on /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/4a9cd960-b3db-4e18-a7b7-a5bde23abea5/123e2cc8-8f24-4ddd-b401-68393535f555 (fileSD:629) 2019-04-13 21:40:45,998+0200 INFO (vmrecovery) [storage.StorageDomain] Creating domain run directory u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782' (fileSD:583) 2019-04-13 21:40:45,998+0200 INFO (vmrecovery) [storage.fileUtils] Creating directory: /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782 mode: None (fileUtils:199) 2019-04-13 21:40:45,998+0200 INFO (vmrecovery) [storage.StorageDomain] Creating symlink from /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/4a9cd960-b3db-4e18-a7b7-a5bde23abea5 to /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/4a9cd960-b3db-4e18-a7b7-a5bde23abea5 (fileSD:586) 2019-04-13 21:40:46,004+0200 INFO (vmrecovery) [vdsm.api] FINISH prepareImage return={'info': {'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/4a9cd960-b3db-4e18-a7b7-a5bde23abea5/123e2cc8-8f24-4ddd-b401-68393535f555', 'type': 'file'}, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/4a9cd960-b3db-4e18-a7b7-a5bde23abea5/123e2cc8-8f24-4ddd-b401-68393535f555', 'imgVolumesInfo': [{'domainID': '9c3f06cf-7475-448e-819b-f4f52fa7d782', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/4a9cd960-b3db-4e18-a7b7-a5bde23abea5/123e2cc8-8f24-4ddd-b401-68393535f555', 'volumeID': u'123e2cc8-8f24-4ddd-b401-68393535f555', 'leasePath': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/4a9cd960-b3db-4e18-a7b7-a5bde23abea5/123e2cc8-8f24-4ddd-b401-68393535f555.lease', 'imageID': '4a9cd960-b3db-4e18-a7b7-a5bde23abea5'}]} from=internal, task_id=a79e5812-fa01-49fd-8f53-addd4ef9ab2f (api:54) 2019-04-13 21:40:46,004+0200 INFO (vmrecovery) [vds] prepared volume path: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/4a9cd960-b3db-4e18-a7b7-a5bde23abea5/123e2cc8-8f24-4ddd-b401-68393535f555 (clientIF:499) 2019-04-13 21:40:46,005+0200 INFO (vmrecovery) [virt.vm] (vmId='28d7d5ff-2ca1-42c6-ae8d-065c721d3066') Enabling drive monitoring (drivemonitor:56) 2019-04-13 21:40:46,005+0200 INFO (vmrecovery) [vds] recovery [3/16]: preparing paths for domain dd24e11f-40aa-47d2-9f0b-11cd7e83649d (clientIF:722) 2019-04-13 21:40:46,005+0200 INFO (vmrecovery) [vds] prepared volume path: (clientIF:499) 2019-04-13 21:40:46,006+0200 INFO (vmrecovery) [vdsm.api] START prepareImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', leafUUID='50497a6a-62bd-4c4e-b537-ac5bf907b84b', allowIllegal=False) from=internal, task_id=be6df47e-9a7b-4cc0-acd9-2f91cabfa3a5 (api:48) 2019-04-13 21:40:46,034+0200 INFO (vmrecovery) [storage.StorageDomain] Fixing permissions on /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/ce44616c-3479-4636-83e3-c1c967a108d7/50497a6a-62bd-4c4e-b537-ac5bf907b84b (fileSD:629) 2019-04-13 21:40:46,035+0200 INFO (vmrecovery) [storage.StorageDomain] Creating domain run directory u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782' (fileSD:583) 2019-04-13 21:40:46,035+0200 INFO (vmrecovery) [storage.fileUtils] Creating directory: /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782 mode: None (fileUtils:199) 2019-04-13 21:40:46,035+0200 INFO (vmrecovery) [storage.StorageDomain] Creating symlink from /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/ce44616c-3479-4636-83e3-c1c967a108d7 to /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/ce44616c-3479-4636-83e3-c1c967a108d7 (fileSD:586) 2019-04-13 21:40:46,041+0200 INFO (vmrecovery) [vdsm.api] FINISH prepareImage return={'info': {'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/ce44616c-3479-4636-83e3-c1c967a108d7/50497a6a-62bd-4c4e-b537-ac5bf907b84b', 'type': 'file'}, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/ce44616c-3479-4636-83e3-c1c967a108d7/50497a6a-62bd-4c4e-b537-ac5bf907b84b', 'imgVolumesInfo': [{'domainID': '9c3f06cf-7475-448e-819b-f4f52fa7d782', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/ce44616c-3479-4636-83e3-c1c967a108d7/50497a6a-62bd-4c4e-b537-ac5bf907b84b', 'volumeID': u'50497a6a-62bd-4c4e-b537-ac5bf907b84b', 'leasePath': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/ce44616c-3479-4636-83e3-c1c967a108d7/50497a6a-62bd-4c4e-b537-ac5bf907b84b.lease', 'imageID': 'ce44616c-3479-4636-83e3-c1c967a108d7'}]} from=internal, task_id=be6df47e-9a7b-4cc0-acd9-2f91cabfa3a5 (api:54) 2019-04-13 21:40:46,041+0200 INFO (vmrecovery) [vds] prepared volume path: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/ce44616c-3479-4636-83e3-c1c967a108d7/50497a6a-62bd-4c4e-b537-ac5bf907b84b (clientIF:499) 2019-04-13 21:40:46,041+0200 INFO (vmrecovery) [virt.vm] (vmId='dd24e11f-40aa-47d2-9f0b-11cd7e83649d') Enabling drive monitoring (drivemonitor:56) 2019-04-13 21:40:46,041+0200 INFO (vmrecovery) [vds] recovery [4/16]: preparing paths for domain 40eaf7e2-659c-4790-8362-49fe12c5f61a (clientIF:722) 2019-04-13 21:40:46,042+0200 INFO (vmrecovery) [vds] prepared volume path: (clientIF:499) 2019-04-13 21:40:46,043+0200 INFO (vmrecovery) [vdsm.api] START prepareImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', leafUUID='66d4a17e-1aac-41c8-88e0-1a414071a2dd', allowIllegal=False) from=internal, task_id=6945a9bf-8654-4ef0-be8d-7633e0081362 (api:48) 2019-04-13 21:40:46,072+0200 INFO (vmrecovery) [storage.StorageDomain] Fixing permissions on /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/7cb20658-d49a-4a0b-b168-1680a56f8292/66d4a17e-1aac-41c8-88e0-1a414071a2dd (fileSD:629) 2019-04-13 21:40:46,073+0200 INFO (vmrecovery) [storage.StorageDomain] Creating domain run directory u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782' (fileSD:583) 2019-04-13 21:40:46,073+0200 INFO (vmrecovery) [storage.fileUtils] Creating directory: /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782 mode: None (fileUtils:199) 2019-04-13 21:40:46,073+0200 INFO (vmrecovery) [storage.StorageDomain] Creating symlink from /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/7cb20658-d49a-4a0b-b168-1680a56f8292 to /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/7cb20658-d49a-4a0b-b168-1680a56f8292 (fileSD:586) 2019-04-13 21:40:46,078+0200 INFO (vmrecovery) [vdsm.api] FINISH prepareImage return={'info': {'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/7cb20658-d49a-4a0b-b168-1680a56f8292/66d4a17e-1aac-41c8-88e0-1a414071a2dd', 'type': 'file'}, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/7cb20658-d49a-4a0b-b168-1680a56f8292/66d4a17e-1aac-41c8-88e0-1a414071a2dd', 'imgVolumesInfo': [{'domainID': '9c3f06cf-7475-448e-819b-f4f52fa7d782', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/7cb20658-d49a-4a0b-b168-1680a56f8292/66d4a17e-1aac-41c8-88e0-1a414071a2dd', 'volumeID': u'66d4a17e-1aac-41c8-88e0-1a414071a2dd', 'leasePath': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/7cb20658-d49a-4a0b-b168-1680a56f8292/66d4a17e-1aac-41c8-88e0-1a414071a2dd.lease', 'imageID': '7cb20658-d49a-4a0b-b168-1680a56f8292'}]} from=internal, task_id=6945a9bf-8654-4ef0-be8d-7633e0081362 (api:54) 2019-04-13 21:40:46,078+0200 INFO (vmrecovery) [vds] prepared volume path: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/7cb20658-d49a-4a0b-b168-1680a56f8292/66d4a17e-1aac-41c8-88e0-1a414071a2dd (clientIF:499) 2019-04-13 21:40:46,078+0200 INFO (vmrecovery) [virt.vm] (vmId='40eaf7e2-659c-4790-8362-49fe12c5f61a') Enabling drive monitoring (drivemonitor:56) 2019-04-13 21:40:46,078+0200 INFO (vmrecovery) [vds] recovery [5/16]: preparing paths for domain 0f8aff0a-7915-4214-a37c-c9fee6518869 (clientIF:722) 2019-04-13 21:40:46,079+0200 INFO (vmrecovery) [vds] prepared volume path: (clientIF:499) 2019-04-13 21:40:46,079+0200 INFO (vmrecovery) [vdsm.api] START prepareImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', leafUUID='ef6741d1-380a-4736-a924-f88ed68e1270', allowIllegal=False) from=internal, task_id=73101351-891d-46f7-a1ce-6162103bc7fe (api:48) 2019-04-13 21:40:46,105+0200 INFO (vmrecovery) [storage.StorageDomain] Fixing permissions on /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd/ef6741d1-380a-4736-a924-f88ed68e1270 (fileSD:629) 2019-04-13 21:40:46,106+0200 INFO (vmrecovery) [storage.StorageDomain] Creating domain run directory u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782' (fileSD:583) 2019-04-13 21:40:46,107+0200 INFO (vmrecovery) [storage.fileUtils] Creating directory: /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782 mode: None (fileUtils:199) 2019-04-13 21:40:46,107+0200 INFO (vmrecovery) [storage.StorageDomain] Creating symlink from /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd to /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd (fileSD:586) 2019-04-13 21:40:46,112+0200 INFO (vmrecovery) [vdsm.api] FINISH prepareImage return={'info': {'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd/ef6741d1-380a-4736-a924-f88ed68e1270', 'type': 'file'}, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd/ef6741d1-380a-4736-a924-f88ed68e1270', 'imgVolumesInfo': [{'domainID': '9c3f06cf-7475-448e-819b-f4f52fa7d782', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd/ef6741d1-380a-4736-a924-f88ed68e1270', 'volumeID': u'ef6741d1-380a-4736-a924-f88ed68e1270', 'leasePath': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd/ef6741d1-380a-4736-a924-f88ed68e1270.lease', 'imageID': '9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd'}]} from=internal, task_id=73101351-891d-46f7-a1ce-6162103bc7fe (api:54) 2019-04-13 21:40:46,112+0200 INFO (vmrecovery) [vds] prepared volume path: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd/ef6741d1-380a-4736-a924-f88ed68e1270 (clientIF:499) 2019-04-13 21:40:46,113+0200 INFO (vmrecovery) [virt.vm] (vmId='0f8aff0a-7915-4214-a37c-c9fee6518869') Enabling drive monitoring (drivemonitor:56) 2019-04-13 21:40:46,113+0200 INFO (vmrecovery) [vds] recovery [6/16]: preparing paths for domain 5dd5db29-143f-4d70-9469-4aaa163b7f9d (clientIF:722) 2019-04-13 21:40:46,114+0200 INFO (vmrecovery) [vds] prepared volume path: (clientIF:499) 2019-04-13 21:40:46,115+0200 INFO (vmrecovery) [vdsm.api] START prepareImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='22b8e06e-7cde-4dcb-8676-e0502f335cb7', leafUUID='67a4dd71-69ff-49a8-b8a4-c7101c75e105', allowIllegal=False) from=internal, task_id=5465f89e-c902-49df-8d9a-dda4f59bbab8 (api:48) 2019-04-13 21:40:46,154+0200 INFO (vmrecovery) [vdsm.api] FINISH prepareImage error=Cannot prepare illegal volume: (u'b7fb83ef-77ba-408b-951c-a655c708638b',) from=internal, task_id=5465f89e-c902-49df-8d9a-dda4f59bbab8 (api:52) 2019-04-13 21:40:46,154+0200 ERROR (vmrecovery) [storage.TaskManager.Task] (Task='5465f89e-c902-49df-8d9a-dda4f59bbab8') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in prepareImage File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3204, in prepareImage raise se.prepareIllegalVolumeError(volUUID) prepareIllegalVolumeError: Cannot prepare illegal volume: (u'b7fb83ef-77ba-408b-951c-a655c708638b',) 2019-04-13 21:40:46,154+0200 INFO (vmrecovery) [storage.TaskManager.Task] (Task='5465f89e-c902-49df-8d9a-dda4f59bbab8') aborting: Task is aborted: "Cannot prepare illegal volume: (u'b7fb83ef-77ba-408b-951c-a655c708638b',)" - code 227 (task:1181) 2019-04-13 21:40:46,155+0200 ERROR (vmrecovery) [storage.Dispatcher] FINISH prepareImage error=Cannot prepare illegal volume: (u'b7fb83ef-77ba-408b-951c-a655c708638b',) (dispatcher:83) 2019-04-13 21:40:46,155+0200 ERROR (vmrecovery) [vds] recovery [6/16]: failed for vm 5dd5db29-143f-4d70-9469-4aaa163b7f9d (clientIF:727) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 723, in _preparePathsForRecoveredVMs vm_obj.preparePaths() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1003, in preparePaths self._preparePathsForDrives(drives) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1011, in _preparePathsForDrives drive['path'] = self.cif.prepareVolumePath(drive, self.id) File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 415, in prepareVolumePath raise vm.VolumeError(drive) VolumeError: Bad volume specification {'index': 0, 'domainID': '9c3f06cf-7475-448e-819b-f4f52fa7d782', 'name': 'sda', 'alias': 'ua-22b8e06e-7cde-4dcb-8676-e0502f335cb7', 'bootOrder': '1', 'format': 'raw', 'poolID': '59ef3a18-002f-02d1-0220-000000000124', 'volumeID': '67a4dd71-69ff-49a8-b8a4-c7101c75e105', 'diskType': 'file', 'discard': False, 'specParams': {}, 'iface': 'scsi', 'cache': 'none', 'propagateErrors': 'off', 'address': {'bus': '0', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'device': 'disk', 'path': '/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/22b8e06e-7cde-4dcb-8676-e0502f335cb7/67a4dd71-69ff-49a8-b8a4-c7101c75e105', 'serial': '22b8e06e-7cde-4dcb-8676-e0502f335cb7', 'imageID': '22b8e06e-7cde-4dcb-8676-e0502f335cb7', 'type': 'disk'} 2019-04-13 21:40:46,155+0200 INFO (vmrecovery) [vds] recovery [7/16]: preparing paths for domain 139534f6-1741-4ded-8ea6-ea4e3ecdad1f (clientIF:722) 2019-04-13 21:40:46,156+0200 INFO (vmrecovery) [vds] prepared volume path: (clientIF:499) 2019-04-13 21:40:46,157+0200 INFO (vmrecovery) [vdsm.api] START prepareImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', leafUUID='04c6ab2a-994d-4002-9f55-5f32378864f1', allowIllegal=False) from=internal, task_id=edb9bc95-587a-4f10-881a-f457b5f021a7 (api:48) 2019-04-13 21:40:46,188+0200 INFO (vmrecovery) [storage.StorageDomain] Fixing permissions on /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/fc829fbd-ef00-4ae4-b06a-74a972251909/04c6ab2a-994d-4002-9f55-5f32378864f1 (fileSD:629) 2019-04-13 21:40:46,188+0200 INFO (vmrecovery) [storage.StorageDomain] Creating domain run directory u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782' (fileSD:583) 2019-04-13 21:40:46,189+0200 INFO (vmrecovery) [storage.fileUtils] Creating directory: /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782 mode: None (fileUtils:199) 2019-04-13 21:40:46,189+0200 INFO (vmrecovery) [storage.StorageDomain] Creating symlink from /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/fc829fbd-ef00-4ae4-b06a-74a972251909 to /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/fc829fbd-ef00-4ae4-b06a-74a972251909 (fileSD:586) 2019-04-13 21:40:46,194+0200 INFO (vmrecovery) [vdsm.api] FINISH prepareImage return={'info': {'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/fc829fbd-ef00-4ae4-b06a-74a972251909/04c6ab2a-994d-4002-9f55-5f32378864f1', 'type': 'file'}, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/fc829fbd-ef00-4ae4-b06a-74a972251909/04c6ab2a-994d-4002-9f55-5f32378864f1', 'imgVolumesInfo': [{'domainID': '9c3f06cf-7475-448e-819b-f4f52fa7d782', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/fc829fbd-ef00-4ae4-b06a-74a972251909/04c6ab2a-994d-4002-9f55-5f32378864f1', 'volumeID': u'04c6ab2a-994d-4002-9f55-5f32378864f1', 'leasePath': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/fc829fbd-ef00-4ae4-b06a-74a972251909/04c6ab2a-994d-4002-9f55-5f32378864f1.lease', 'imageID': 'fc829fbd-ef00-4ae4-b06a-74a972251909'}]} from=internal, task_id=edb9bc95-587a-4f10-881a-f457b5f021a7 (api:54) 2019-04-13 21:40:46,195+0200 INFO (vmrecovery) [vds] prepared volume path: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/fc829fbd-ef00-4ae4-b06a-74a972251909/04c6ab2a-994d-4002-9f55-5f32378864f1 (clientIF:499) 2019-04-13 21:40:46,195+0200 INFO (vmrecovery) [virt.vm] (vmId='139534f6-1741-4ded-8ea6-ea4e3ecdad1f') Enabling drive monitoring (drivemonitor:56) 2019-04-13 21:40:46,195+0200 INFO (vmrecovery) [vds] recovery [8/16]: preparing paths for domain 7b5e0fdd-8757-49ce-acf9-3206a372e1c9 (clientIF:722) 2019-04-13 21:40:46,196+0200 INFO (vmrecovery) [vds] prepared volume path: (clientIF:499) 2019-04-13 21:40:46,197+0200 INFO (vmrecovery) [vdsm.api] START prepareImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', leafUUID='3f6c31c6-ec38-4979-a617-55537123ee61', allowIllegal=False) from=internal, task_id=e1bcdc50-d1cd-4e48-9ce1-a211e8c3fae7 (api:48) 2019-04-13 21:40:46,218+0200 INFO (vmrecovery) [storage.StorageDomain] Fixing permissions on /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/7bb7b150-d992-40fe-bbc9-e3fb66f21c10/3f6c31c6-ec38-4979-a617-55537123ee61 (fileSD:629) 2019-04-13 21:40:46,219+0200 INFO (vmrecovery) [storage.StorageDomain] Creating domain run directory u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782' (fileSD:583) 2019-04-13 21:40:46,219+0200 INFO (vmrecovery) [storage.fileUtils] Creating directory: /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782 mode: None (fileUtils:199) 2019-04-13 21:40:46,219+0200 INFO (vmrecovery) [storage.StorageDomain] Creating symlink from /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/7bb7b150-d992-40fe-bbc9-e3fb66f21c10 to /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/7bb7b150-d992-40fe-bbc9-e3fb66f21c10 (fileSD:586) 2019-04-13 21:40:46,224+0200 INFO (vmrecovery) [vdsm.api] FINISH prepareImage return={'info': {'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/7bb7b150-d992-40fe-bbc9-e3fb66f21c10/3f6c31c6-ec38-4979-a617-55537123ee61', 'type': 'file'}, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/7bb7b150-d992-40fe-bbc9-e3fb66f21c10/3f6c31c6-ec38-4979-a617-55537123ee61', 'imgVolumesInfo': [{'domainID': '9c3f06cf-7475-448e-819b-f4f52fa7d782', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/7bb7b150-d992-40fe-bbc9-e3fb66f21c10/3f6c31c6-ec38-4979-a617-55537123ee61', 'volumeID': u'3f6c31c6-ec38-4979-a617-55537123ee61', 'leasePath': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/7bb7b150-d992-40fe-bbc9-e3fb66f21c10/3f6c31c6-ec38-4979-a617-55537123ee61.lease', 'imageID': '7bb7b150-d992-40fe-bbc9-e3fb66f21c10'}]} from=internal, task_id=e1bcdc50-d1cd-4e48-9ce1-a211e8c3fae7 (api:54) 2019-04-13 21:40:46,225+0200 INFO (vmrecovery) [vds] prepared volume path: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/7bb7b150-d992-40fe-bbc9-e3fb66f21c10/3f6c31c6-ec38-4979-a617-55537123ee61 (clientIF:499) 2019-04-13 21:40:46,225+0200 INFO (vmrecovery) [virt.vm] (vmId='7b5e0fdd-8757-49ce-acf9-3206a372e1c9') Enabling drive monitoring (drivemonitor:56) 2019-04-13 21:40:46,225+0200 INFO (vmrecovery) [vds] recovery [9/16]: preparing paths for domain b5713713-ac04-4a91-95b8-3c3f17eba32f (clientIF:722) 2019-04-13 21:40:46,226+0200 INFO (vmrecovery) [vds] prepared volume path: (clientIF:499) 2019-04-13 21:40:46,227+0200 INFO (vmrecovery) [vdsm.api] START prepareImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fd3b80fd-49ad-44ac-9efd-1328300582cd', leafUUID='a1d583c1-e135-4e3f-b6fa-0f29a5261366', allowIllegal=False) from=internal, task_id=dd8e05be-27ac-411f-8ee9-88f893505382 (api:48) 2019-04-13 21:40:46,252+0200 INFO (vmrecovery) [vdsm.api] FINISH prepareImage error=Cannot prepare illegal volume: (u'47c0f42e-8bda-4e3f-8337-870899238788',) from=internal, task_id=dd8e05be-27ac-411f-8ee9-88f893505382 (api:52) 2019-04-13 21:40:46,252+0200 ERROR (vmrecovery) [storage.TaskManager.Task] (Task='dd8e05be-27ac-411f-8ee9-88f893505382') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in prepareImage File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3204, in prepareImage raise se.prepareIllegalVolumeError(volUUID) prepareIllegalVolumeError: Cannot prepare illegal volume: (u'47c0f42e-8bda-4e3f-8337-870899238788',) 2019-04-13 21:40:46,253+0200 INFO (vmrecovery) [storage.TaskManager.Task] (Task='dd8e05be-27ac-411f-8ee9-88f893505382') aborting: Task is aborted: "Cannot prepare illegal volume: (u'47c0f42e-8bda-4e3f-8337-870899238788',)" - code 227 (task:1181) 2019-04-13 21:40:46,253+0200 ERROR (vmrecovery) [storage.Dispatcher] FINISH prepareImage error=Cannot prepare illegal volume: (u'47c0f42e-8bda-4e3f-8337-870899238788',) (dispatcher:83) 2019-04-13 21:40:46,253+0200 ERROR (vmrecovery) [vds] recovery [9/16]: failed for vm b5713713-ac04-4a91-95b8-3c3f17eba32f (clientIF:727) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 723, in _preparePathsForRecoveredVMs vm_obj.preparePaths() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1003, in preparePaths self._preparePathsForDrives(drives) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1011, in _preparePathsForDrives drive['path'] = self.cif.prepareVolumePath(drive, self.id) File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 415, in prepareVolumePath raise vm.VolumeError(drive) VolumeError: Bad volume specification {'index': 0, 'domainID': '9c3f06cf-7475-448e-819b-f4f52fa7d782', 'name': 'sda', 'alias': 'ua-fd3b80fd-49ad-44ac-9efd-1328300582cd', 'bootOrder': '1', 'format': 'raw', 'poolID': '59ef3a18-002f-02d1-0220-000000000124', 'volumeID': 'a1d583c1-e135-4e3f-b6fa-0f29a5261366', 'diskType': 'file', 'discard': False, 'specParams': {}, 'iface': 'scsi', 'cache': 'none', 'propagateErrors': 'off', 'address': {'bus': '0', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'device': 'disk', 'path': '/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/fd3b80fd-49ad-44ac-9efd-1328300582cd/a1d583c1-e135-4e3f-b6fa-0f29a5261366', 'serial': 'fd3b80fd-49ad-44ac-9efd-1328300582cd', 'imageID': 'fd3b80fd-49ad-44ac-9efd-1328300582cd', 'type': 'disk'} 2019-04-13 21:40:46,254+0200 INFO (vmrecovery) [vds] recovery [10/16]: preparing paths for domain 6772f848-4874-4785-9bb0-88d40bfb71c4 (clientIF:722) 2019-04-13 21:40:46,255+0200 INFO (vmrecovery) [vds] prepared volume path: (clientIF:499) 2019-04-13 21:40:46,255+0200 INFO (vmrecovery) [vdsm.api] START prepareImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', leafUUID='6d9eda60-5b17-47ba-84df-43c6cce2fe35', allowIllegal=False) from=internal, task_id=7a8899c7-b936-49e4-9554-db42987a354e (api:48) 2019-04-13 21:40:46,277+0200 INFO (vmrecovery) [storage.StorageDomain] Fixing permissions on /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/467e6c8c-7c9f-4fc1-866b-3f96940305a9/6d9eda60-5b17-47ba-84df-43c6cce2fe35 (fileSD:629) 2019-04-13 21:40:46,278+0200 INFO (vmrecovery) [storage.StorageDomain] Creating domain run directory u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782' (fileSD:583) 2019-04-13 21:40:46,278+0200 INFO (vmrecovery) [storage.fileUtils] Creating directory: /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782 mode: None (fileUtils:199) 2019-04-13 21:40:46,278+0200 INFO (vmrecovery) [storage.StorageDomain] Creating symlink from /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/467e6c8c-7c9f-4fc1-866b-3f96940305a9 to /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/467e6c8c-7c9f-4fc1-866b-3f96940305a9 (fileSD:586) 2019-04-13 21:40:46,283+0200 INFO (vmrecovery) [vdsm.api] FINISH prepareImage return={'info': {'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/467e6c8c-7c9f-4fc1-866b-3f96940305a9/6d9eda60-5b17-47ba-84df-43c6cce2fe35', 'type': 'file'}, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/467e6c8c-7c9f-4fc1-866b-3f96940305a9/6d9eda60-5b17-47ba-84df-43c6cce2fe35', 'imgVolumesInfo': [{'domainID': '9c3f06cf-7475-448e-819b-f4f52fa7d782', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/467e6c8c-7c9f-4fc1-866b-3f96940305a9/6d9eda60-5b17-47ba-84df-43c6cce2fe35', 'volumeID': u'6d9eda60-5b17-47ba-84df-43c6cce2fe35', 'leasePath': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/467e6c8c-7c9f-4fc1-866b-3f96940305a9/6d9eda60-5b17-47ba-84df-43c6cce2fe35.lease', 'imageID': '467e6c8c-7c9f-4fc1-866b-3f96940305a9'}]} from=internal, task_id=7a8899c7-b936-49e4-9554-db42987a354e (api:54) 2019-04-13 21:40:46,284+0200 INFO (vmrecovery) [vds] prepared volume path: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/467e6c8c-7c9f-4fc1-866b-3f96940305a9/6d9eda60-5b17-47ba-84df-43c6cce2fe35 (clientIF:499) 2019-04-13 21:40:46,284+0200 INFO (vmrecovery) [virt.vm] (vmId='6772f848-4874-4785-9bb0-88d40bfb71c4') Enabling drive monitoring (drivemonitor:56) 2019-04-13 21:40:46,284+0200 INFO (vmrecovery) [vds] recovery [11/16]: preparing paths for domain 611486ee-b72f-4acc-8925-4d7b6df9af1d (clientIF:722) 2019-04-13 21:40:46,285+0200 INFO (vmrecovery) [vds] prepared volume path: (clientIF:499) 2019-04-13 21:40:46,286+0200 INFO (vmrecovery) [vdsm.api] START prepareImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', leafUUID='a884b4ce-381f-4653-be14-a31d15181f48', allowIllegal=False) from=internal, task_id=84090143-bc9c-4467-8774-52af5798d39a (api:48) 2019-04-13 21:40:46,321+0200 INFO (vmrecovery) [storage.StorageDomain] Fixing permissions on /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/e5cfca12-c454-4f4f-a102-b4f54ff04082/a884b4ce-381f-4653-be14-a31d15181f48 (fileSD:629) 2019-04-13 21:40:46,322+0200 INFO (vmrecovery) [storage.StorageDomain] Creating domain run directory u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782' (fileSD:583) 2019-04-13 21:40:46,322+0200 INFO (vmrecovery) [storage.fileUtils] Creating directory: /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782 mode: None (fileUtils:199) 2019-04-13 21:40:46,323+0200 INFO (vmrecovery) [storage.StorageDomain] Creating symlink from /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/e5cfca12-c454-4f4f-a102-b4f54ff04082 to /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/e5cfca12-c454-4f4f-a102-b4f54ff04082 (fileSD:586) 2019-04-13 21:40:46,328+0200 INFO (vmrecovery) [vdsm.api] FINISH prepareImage return={'info': {'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/e5cfca12-c454-4f4f-a102-b4f54ff04082/a884b4ce-381f-4653-be14-a31d15181f48', 'type': 'file'}, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/e5cfca12-c454-4f4f-a102-b4f54ff04082/a884b4ce-381f-4653-be14-a31d15181f48', 'imgVolumesInfo': [{'domainID': '9c3f06cf-7475-448e-819b-f4f52fa7d782', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/e5cfca12-c454-4f4f-a102-b4f54ff04082/a884b4ce-381f-4653-be14-a31d15181f48', 'volumeID': u'a884b4ce-381f-4653-be14-a31d15181f48', 'leasePath': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/e5cfca12-c454-4f4f-a102-b4f54ff04082/a884b4ce-381f-4653-be14-a31d15181f48.lease', 'imageID': 'e5cfca12-c454-4f4f-a102-b4f54ff04082'}]} from=internal, task_id=84090143-bc9c-4467-8774-52af5798d39a (api:54) 2019-04-13 21:40:46,328+0200 INFO (vmrecovery) [vds] prepared volume path: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/e5cfca12-c454-4f4f-a102-b4f54ff04082/a884b4ce-381f-4653-be14-a31d15181f48 (clientIF:499) 2019-04-13 21:40:46,329+0200 INFO (vmrecovery) [virt.vm] (vmId='611486ee-b72f-4acc-8925-4d7b6df9af1d') Enabling drive monitoring (drivemonitor:56) 2019-04-13 21:40:46,329+0200 INFO (vmrecovery) [vds] recovery [12/16]: preparing paths for domain afa2c17f-39e4-4168-aad7-2548dd19cc26 (clientIF:722) 2019-04-13 21:40:46,330+0200 INFO (vmrecovery) [vds] prepared volume path: (clientIF:499) 2019-04-13 21:40:46,330+0200 INFO (vmrecovery) [vdsm.api] START prepareImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e4b540ec-1e39-4349-b79b-f446f0b74237', leafUUID='2b136ed7-8b9a-45c1-9edc-64ee96c87c43', allowIllegal=False) from=internal, task_id=83bc0008-983f-46b2-a1c5-731685e63fd8 (api:48) 2019-04-13 21:40:46,359+0200 INFO (vmrecovery) [vdsm.api] FINISH prepareImage error=Cannot prepare illegal volume: (u'940f5e42-77b4-4b95-ae11-62ecf1f3a5de',) from=internal, task_id=83bc0008-983f-46b2-a1c5-731685e63fd8 (api:52) 2019-04-13 21:40:46,359+0200 ERROR (vmrecovery) [storage.TaskManager.Task] (Task='83bc0008-983f-46b2-a1c5-731685e63fd8') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in prepareImage File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3204, in prepareImage raise se.prepareIllegalVolumeError(volUUID) prepareIllegalVolumeError: Cannot prepare illegal volume: (u'940f5e42-77b4-4b95-ae11-62ecf1f3a5de',) 2019-04-13 21:40:46,359+0200 INFO (vmrecovery) [storage.TaskManager.Task] (Task='83bc0008-983f-46b2-a1c5-731685e63fd8') aborting: Task is aborted: "Cannot prepare illegal volume: (u'940f5e42-77b4-4b95-ae11-62ecf1f3a5de',)" - code 227 (task:1181) 2019-04-13 21:40:46,360+0200 ERROR (vmrecovery) [storage.Dispatcher] FINISH prepareImage error=Cannot prepare illegal volume: (u'940f5e42-77b4-4b95-ae11-62ecf1f3a5de',) (dispatcher:83) 2019-04-13 21:40:46,360+0200 ERROR (vmrecovery) [vds] recovery [12/16]: failed for vm afa2c17f-39e4-4168-aad7-2548dd19cc26 (clientIF:727) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 723, in _preparePathsForRecoveredVMs vm_obj.preparePaths() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1003, in preparePaths self._preparePathsForDrives(drives) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1011, in _preparePathsForDrives drive['path'] = self.cif.prepareVolumePath(drive, self.id) File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 415, in prepareVolumePath raise vm.VolumeError(drive) VolumeError: Bad volume specification {'index': 0, 'domainID': '9c3f06cf-7475-448e-819b-f4f52fa7d782', 'name': 'sda', 'alias': 'ua-e4b540ec-1e39-4349-b79b-f446f0b74237', 'bootOrder': '1', 'format': 'raw', 'poolID': '59ef3a18-002f-02d1-0220-000000000124', 'volumeID': '2b136ed7-8b9a-45c1-9edc-64ee96c87c43', 'diskType': 'file', 'discard': False, 'specParams': {}, 'iface': 'scsi', 'cache': 'none', 'propagateErrors': 'off', 'address': {'bus': '0', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'device': 'disk', 'path': '/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/e4b540ec-1e39-4349-b79b-f446f0b74237/2b136ed7-8b9a-45c1-9edc-64ee96c87c43', 'serial': 'e4b540ec-1e39-4349-b79b-f446f0b74237', 'imageID': 'e4b540ec-1e39-4349-b79b-f446f0b74237', 'type': 'disk'} 2019-04-13 21:40:46,361+0200 INFO (vmrecovery) [vds] recovery [13/16]: preparing paths for domain a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3 (clientIF:722) 2019-04-13 21:40:46,362+0200 INFO (vmrecovery) [vds] prepared volume path: (clientIF:499) 2019-04-13 21:40:46,362+0200 INFO (vmrecovery) [vdsm.api] START prepareImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='614d3cdc-7f35-4c25-9b7b-711f0a5e6644', leafUUID='44b3ceca-f0db-43da-af29-aa681df146ac', allowIllegal=False) from=internal, task_id=bf298f7b-c0f4-4b4f-8ac6-3521a13c5571 (api:48) 2019-04-13 21:40:46,377+0200 INFO (vmrecovery) [vdsm.api] FINISH prepareImage error=Cannot prepare illegal volume: (u'8fcf0d6e-a32e-46f1-8601-2960508a6986',) from=internal, task_id=bf298f7b-c0f4-4b4f-8ac6-3521a13c5571 (api:52) 2019-04-13 21:40:46,378+0200 ERROR (vmrecovery) [storage.TaskManager.Task] (Task='bf298f7b-c0f4-4b4f-8ac6-3521a13c5571') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in prepareImage File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3204, in prepareImage raise se.prepareIllegalVolumeError(volUUID) prepareIllegalVolumeError: Cannot prepare illegal volume: (u'8fcf0d6e-a32e-46f1-8601-2960508a6986',) 2019-04-13 21:40:46,378+0200 INFO (vmrecovery) [storage.TaskManager.Task] (Task='bf298f7b-c0f4-4b4f-8ac6-3521a13c5571') aborting: Task is aborted: "Cannot prepare illegal volume: (u'8fcf0d6e-a32e-46f1-8601-2960508a6986',)" - code 227 (task:1181) 2019-04-13 21:40:46,379+0200 ERROR (vmrecovery) [storage.Dispatcher] FINISH prepareImage error=Cannot prepare illegal volume: (u'8fcf0d6e-a32e-46f1-8601-2960508a6986',) (dispatcher:83) 2019-04-13 21:40:46,379+0200 ERROR (vmrecovery) [vds] recovery [13/16]: failed for vm a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3 (clientIF:727) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 723, in _preparePathsForRecoveredVMs vm_obj.preparePaths() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1003, in preparePaths self._preparePathsForDrives(drives) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1011, in _preparePathsForDrives drive['path'] = self.cif.prepareVolumePath(drive, self.id) File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 415, in prepareVolumePath raise vm.VolumeError(drive) VolumeError: Bad volume specification {'index': 0, 'domainID': '9c3f06cf-7475-448e-819b-f4f52fa7d782', 'name': 'sda', 'alias': 'ua-614d3cdc-7f35-4c25-9b7b-711f0a5e6644', 'bootOrder': '1', 'format': 'raw', 'poolID': '59ef3a18-002f-02d1-0220-000000000124', 'volumeID': '44b3ceca-f0db-43da-af29-aa681df146ac', 'diskType': 'file', 'discard': False, 'specParams': {}, 'iface': 'scsi', 'cache': 'none', 'propagateErrors': 'off', 'address': {'bus': '0', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'device': 'disk', 'path': '/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/614d3cdc-7f35-4c25-9b7b-711f0a5e6644/44b3ceca-f0db-43da-af29-aa681df146ac', 'serial': '614d3cdc-7f35-4c25-9b7b-711f0a5e6644', 'imageID': '614d3cdc-7f35-4c25-9b7b-711f0a5e6644', 'type': 'disk'} 2019-04-13 21:40:46,379+0200 INFO (vmrecovery) [vds] recovery [14/16]: preparing paths for domain c8df4609-fe26-4b73-9137-57724aff3ca4 (clientIF:722) 2019-04-13 21:40:46,380+0200 INFO (vmrecovery) [vds] prepared volume path: (clientIF:499) 2019-04-13 21:40:46,381+0200 INFO (vmrecovery) [vdsm.api] START prepareImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', leafUUID='2b36748a-d4c6-4fd8-aad2-81549e665188', allowIllegal=False) from=internal, task_id=6cb9f7a6-0f7a-47c1-b19e-2fbcdf0c42ba (api:48) 2019-04-13 21:40:46,410+0200 INFO (vmrecovery) [storage.StorageDomain] Fixing permissions on /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/be884b3f-db5e-40e7-87dc-b98fb99fb95b/2b36748a-d4c6-4fd8-aad2-81549e665188 (fileSD:629) 2019-04-13 21:40:46,411+0200 INFO (vmrecovery) [storage.StorageDomain] Creating domain run directory u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782' (fileSD:583) 2019-04-13 21:40:46,411+0200 INFO (vmrecovery) [storage.fileUtils] Creating directory: /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782 mode: None (fileUtils:199) 2019-04-13 21:40:46,412+0200 INFO (vmrecovery) [storage.StorageDomain] Creating symlink from /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/be884b3f-db5e-40e7-87dc-b98fb99fb95b to /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/be884b3f-db5e-40e7-87dc-b98fb99fb95b (fileSD:586) 2019-04-13 21:40:46,416+0200 INFO (vmrecovery) [vdsm.api] FINISH prepareImage return={'info': {'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/be884b3f-db5e-40e7-87dc-b98fb99fb95b/2b36748a-d4c6-4fd8-aad2-81549e665188', 'type': 'file'}, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/be884b3f-db5e-40e7-87dc-b98fb99fb95b/2b36748a-d4c6-4fd8-aad2-81549e665188', 'imgVolumesInfo': [{'domainID': '9c3f06cf-7475-448e-819b-f4f52fa7d782', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/be884b3f-db5e-40e7-87dc-b98fb99fb95b/2b36748a-d4c6-4fd8-aad2-81549e665188', 'volumeID': u'2b36748a-d4c6-4fd8-aad2-81549e665188', 'leasePath': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/be884b3f-db5e-40e7-87dc-b98fb99fb95b/2b36748a-d4c6-4fd8-aad2-81549e665188.lease', 'imageID': 'be884b3f-db5e-40e7-87dc-b98fb99fb95b'}]} from=internal, task_id=6cb9f7a6-0f7a-47c1-b19e-2fbcdf0c42ba (api:54) 2019-04-13 21:40:46,417+0200 INFO (vmrecovery) [vds] prepared volume path: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/be884b3f-db5e-40e7-87dc-b98fb99fb95b/2b36748a-d4c6-4fd8-aad2-81549e665188 (clientIF:499) 2019-04-13 21:40:46,417+0200 INFO (vmrecovery) [virt.vm] (vmId='c8df4609-fe26-4b73-9137-57724aff3ca4') Enabling drive monitoring (drivemonitor:56) 2019-04-13 21:40:46,417+0200 INFO (vmrecovery) [vds] recovery [15/16]: preparing paths for domain cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6 (clientIF:722) 2019-04-13 21:40:46,418+0200 INFO (vmrecovery) [vdsm.api] START prepareImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', leafUUID='ada24be6-3165-4a99-a002-e951d8807deb', allowIllegal=False) from=internal, task_id=a46a1873-1753-4297-8a4d-bdff3bdd59b4 (api:48) 2019-04-13 21:40:46,442+0200 INFO (vmrecovery) [storage.StorageDomain] Fixing permissions on /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/0066d0ee-95b6-4554-bf20-84337780dfe8/ada24be6-3165-4a99-a002-e951d8807deb (fileSD:629) 2019-04-13 21:40:46,443+0200 INFO (vmrecovery) [storage.StorageDomain] Creating domain run directory u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782' (fileSD:583) 2019-04-13 21:40:46,443+0200 INFO (vmrecovery) [storage.fileUtils] Creating directory: /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782 mode: None (fileUtils:199) 2019-04-13 21:40:46,444+0200 INFO (vmrecovery) [storage.StorageDomain] Creating symlink from /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/0066d0ee-95b6-4554-bf20-84337780dfe8 to /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/0066d0ee-95b6-4554-bf20-84337780dfe8 (fileSD:586) 2019-04-13 21:40:46,448+0200 INFO (vmrecovery) [vdsm.api] FINISH prepareImage return={'info': {'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/0066d0ee-95b6-4554-bf20-84337780dfe8/ada24be6-3165-4a99-a002-e951d8807deb', 'type': 'file'}, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/0066d0ee-95b6-4554-bf20-84337780dfe8/ada24be6-3165-4a99-a002-e951d8807deb', 'imgVolumesInfo': [{'domainID': '9c3f06cf-7475-448e-819b-f4f52fa7d782', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/0066d0ee-95b6-4554-bf20-84337780dfe8/ada24be6-3165-4a99-a002-e951d8807deb', 'volumeID': u'ada24be6-3165-4a99-a002-e951d8807deb', 'leasePath': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/0066d0ee-95b6-4554-bf20-84337780dfe8/ada24be6-3165-4a99-a002-e951d8807deb.lease', 'imageID': '0066d0ee-95b6-4554-bf20-84337780dfe8'}]} from=internal, task_id=a46a1873-1753-4297-8a4d-bdff3bdd59b4 (api:54) 2019-04-13 21:40:46,449+0200 INFO (vmrecovery) [vds] prepared volume path: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/0066d0ee-95b6-4554-bf20-84337780dfe8/ada24be6-3165-4a99-a002-e951d8807deb (clientIF:499) 2019-04-13 21:40:46,449+0200 INFO (vmrecovery) [virt.vm] (vmId='cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6') Enabling drive monitoring (drivemonitor:56) 2019-04-13 21:40:46,449+0200 INFO (vmrecovery) [vds] recovery [16/16]: preparing paths for domain 5a326302-86c1-41b5-ba58-793232299713 (clientIF:722) 2019-04-13 21:40:46,450+0200 INFO (vmrecovery) [vds] prepared volume path: (clientIF:499) 2019-04-13 21:40:46,451+0200 INFO (vmrecovery) [vdsm.api] START prepareImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', leafUUID='670b65cd-5c48-4a22-b183-21b46d70cf59', allowIllegal=False) from=internal, task_id=83ab97f7-8177-47ef-9a58-1bddfa34d2e2 (api:48) 2019-04-13 21:40:46,480+0200 INFO (vmrecovery) [storage.StorageDomain] Fixing permissions on /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/ecb12678-f79f-4648-8085-57ffdd2ef8ac/670b65cd-5c48-4a22-b183-21b46d70cf59 (fileSD:629) 2019-04-13 21:40:46,481+0200 INFO (vmrecovery) [storage.StorageDomain] Creating domain run directory u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782' (fileSD:583) 2019-04-13 21:40:46,481+0200 INFO (vmrecovery) [storage.fileUtils] Creating directory: /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782 mode: None (fileUtils:199) 2019-04-13 21:40:46,482+0200 INFO (vmrecovery) [storage.StorageDomain] Creating symlink from /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/ecb12678-f79f-4648-8085-57ffdd2ef8ac to /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/ecb12678-f79f-4648-8085-57ffdd2ef8ac (fileSD:586) 2019-04-13 21:40:46,487+0200 INFO (vmrecovery) [vdsm.api] FINISH prepareImage return={'info': {'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/ecb12678-f79f-4648-8085-57ffdd2ef8ac/670b65cd-5c48-4a22-b183-21b46d70cf59', 'type': 'file'}, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/ecb12678-f79f-4648-8085-57ffdd2ef8ac/670b65cd-5c48-4a22-b183-21b46d70cf59', 'imgVolumesInfo': [{'domainID': '9c3f06cf-7475-448e-819b-f4f52fa7d782', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/ecb12678-f79f-4648-8085-57ffdd2ef8ac/670b65cd-5c48-4a22-b183-21b46d70cf59', 'volumeID': u'670b65cd-5c48-4a22-b183-21b46d70cf59', 'leasePath': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/ecb12678-f79f-4648-8085-57ffdd2ef8ac/670b65cd-5c48-4a22-b183-21b46d70cf59.lease', 'imageID': 'ecb12678-f79f-4648-8085-57ffdd2ef8ac'}]} from=internal, task_id=83ab97f7-8177-47ef-9a58-1bddfa34d2e2 (api:54) 2019-04-13 21:40:46,487+0200 INFO (vmrecovery) [vds] prepared volume path: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/ecb12678-f79f-4648-8085-57ffdd2ef8ac/670b65cd-5c48-4a22-b183-21b46d70cf59 (clientIF:499) 2019-04-13 21:40:46,488+0200 INFO (vmrecovery) [virt.vm] (vmId='5a326302-86c1-41b5-ba58-793232299713') Enabling drive monitoring (drivemonitor:56) 2019-04-13 21:40:46,488+0200 INFO (vmrecovery) [vds] recovery: completed in 34s (clientIF:619) 2019-04-13 21:40:46,610+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:40:46,611+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:40:46,611+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:40:46,703+0200 INFO (jsonrpc/1) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:192.168.0.19,51130, vmId=611486ee-b72f-4acc-8925-4d7b6df9af1d (api:48) 2019-04-13 21:40:46,703+0200 INFO (jsonrpc/1) [virt.vm] (vmId='611486ee-b72f-4acc-8925-4d7b6df9af1d') Release VM resources (vm:5146) 2019-04-13 21:40:46,703+0200 WARN (jsonrpc/1) [virt.vm] (vmId='611486ee-b72f-4acc-8925-4d7b6df9af1d') trying to set state to Powering down when already Down (vm:624) 2019-04-13 21:40:46,703+0200 INFO (jsonrpc/1) [virt.vm] (vmId='611486ee-b72f-4acc-8925-4d7b6df9af1d') Stopping connection (guestagent:455) 2019-04-13 21:40:46,704+0200 INFO (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e5cfca12-c454-4f4f-a102-b4f54ff04082', volUUID=None) from=::ffff:192.168.0.19,51130, task_id=20cadb72-12a2-4659-ac2c-c69f5ccf9bf1 (api:48) 2019-04-13 21:40:46,704+0200 INFO (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/e5cfca12-c454-4f4f-a102-b4f54ff04082' (fileSD:606) 2019-04-13 21:40:46,704+0200 INFO (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:192.168.0.19,51130, task_id=20cadb72-12a2-4659-ac2c-c69f5ccf9bf1 (api:54) 2019-04-13 21:40:46,704+0200 INFO (jsonrpc/1) [virt.vm] (vmId='611486ee-b72f-4acc-8925-4d7b6df9af1d') Stopping connection (guestagent:455) 2019-04-13 21:40:46,716+0200 WARN (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/611486ee-b72f-4acc-8925-4d7b6df9af1d (libvirtnetwork:198) 2019-04-13 21:40:46,716+0200 WARN (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/611486ee-b72f-4acc-8925-4d7b6df9af1d (libvirtnetwork:205) 2019-04-13 21:40:46,717+0200 INFO (jsonrpc/1) [vdsm.api] START inappropriateDevices(thiefId='611486ee-b72f-4acc-8925-4d7b6df9af1d') from=::ffff:192.168.0.19,51130, task_id=ae70128f-77b2-4a73-a019-d0afbdde6a53 (api:48) 2019-04-13 21:40:46,718+0200 INFO (jsonrpc/1) [vdsm.api] FINISH inappropriateDevices return=None from=::ffff:192.168.0.19,51130, task_id=ae70128f-77b2-4a73-a019-d0afbdde6a53 (api:54) 2019-04-13 21:40:46,964+0200 INFO (jsonrpc/1) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/50_vhostmd: rc=0 err= (hooks:114) 2019-04-13 21:40:47,154+0200 INFO (jsonrpc/1) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/delete_vhostuserclient_hook: rc=0 err= (hooks:114) 2019-04-13 21:40:47,156+0200 INFO (jsonrpc/1) [api.virt] FINISH destroy return={'status': {'message': 'Machine destroyed', 'code': 0}} from=::ffff:192.168.0.19,51130, vmId=611486ee-b72f-4acc-8925-4d7b6df9af1d (api:54) 2019-04-13 21:40:47,156+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.destroy succeeded in 0.45 seconds (__init__:312) 2019-04-13 21:40:47,268+0200 INFO (jsonrpc/0) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:192.168.0.19,51130, vmId=28d7d5ff-2ca1-42c6-ae8d-065c721d3066 (api:48) 2019-04-13 21:40:47,268+0200 INFO (jsonrpc/0) [virt.vm] (vmId='28d7d5ff-2ca1-42c6-ae8d-065c721d3066') Release VM resources (vm:5146) 2019-04-13 21:40:47,268+0200 WARN (jsonrpc/0) [virt.vm] (vmId='28d7d5ff-2ca1-42c6-ae8d-065c721d3066') trying to set state to Powering down when already Down (vm:624) 2019-04-13 21:40:47,268+0200 INFO (jsonrpc/0) [virt.vm] (vmId='28d7d5ff-2ca1-42c6-ae8d-065c721d3066') Stopping connection (guestagent:455) 2019-04-13 21:40:47,269+0200 INFO (jsonrpc/0) [vdsm.api] START teardownImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='4a9cd960-b3db-4e18-a7b7-a5bde23abea5', volUUID=None) from=::ffff:192.168.0.19,51130, task_id=64203720-27cc-461c-a6ed-d0941d888592 (api:48) 2019-04-13 21:40:47,269+0200 INFO (jsonrpc/0) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/4a9cd960-b3db-4e18-a7b7-a5bde23abea5' (fileSD:606) 2019-04-13 21:40:47,269+0200 INFO (jsonrpc/0) [vdsm.api] FINISH teardownImage return=None from=::ffff:192.168.0.19,51130, task_id=64203720-27cc-461c-a6ed-d0941d888592 (api:54) 2019-04-13 21:40:47,270+0200 INFO (jsonrpc/0) [virt.vm] (vmId='28d7d5ff-2ca1-42c6-ae8d-065c721d3066') Stopping connection (guestagent:455) 2019-04-13 21:40:47,270+0200 WARN (jsonrpc/0) [root] Attempting to remove a non existing network: ovirtmgmt/28d7d5ff-2ca1-42c6-ae8d-065c721d3066 (libvirtnetwork:198) 2019-04-13 21:40:47,270+0200 WARN (jsonrpc/0) [root] Attempting to remove a non existing net user: ovirtmgmt/28d7d5ff-2ca1-42c6-ae8d-065c721d3066 (libvirtnetwork:205) 2019-04-13 21:40:47,272+0200 INFO (jsonrpc/0) [vdsm.api] START inappropriateDevices(thiefId='28d7d5ff-2ca1-42c6-ae8d-065c721d3066') from=::ffff:192.168.0.19,51130, task_id=fef39e34-e3b7-4eb4-8ef0-2985656e26f5 (api:48) 2019-04-13 21:40:47,273+0200 INFO (jsonrpc/0) [vdsm.api] FINISH inappropriateDevices return=None from=::ffff:192.168.0.19,51130, task_id=fef39e34-e3b7-4eb4-8ef0-2985656e26f5 (api:54) 2019-04-13 21:40:47,515+0200 INFO (jsonrpc/0) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/50_vhostmd: rc=0 err= (hooks:114) 2019-04-13 21:40:47,700+0200 INFO (jsonrpc/0) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/delete_vhostuserclient_hook: rc=0 err= (hooks:114) 2019-04-13 21:40:47,701+0200 INFO (jsonrpc/0) [api.virt] FINISH destroy return={'status': {'message': 'Machine destroyed', 'code': 0}} from=::ffff:192.168.0.19,51130, vmId=28d7d5ff-2ca1-42c6-ae8d-065c721d3066 (api:54) 2019-04-13 21:40:47,702+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.destroy succeeded in 0.43 seconds (__init__:312) 2019-04-13 21:40:47,732+0200 INFO (jsonrpc/2) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:192.168.0.19,51130, vmId=40eaf7e2-659c-4790-8362-49fe12c5f61a (api:48) 2019-04-13 21:40:47,733+0200 INFO (jsonrpc/2) [virt.vm] (vmId='40eaf7e2-659c-4790-8362-49fe12c5f61a') Release VM resources (vm:5146) 2019-04-13 21:40:47,733+0200 WARN (jsonrpc/2) [virt.vm] (vmId='40eaf7e2-659c-4790-8362-49fe12c5f61a') trying to set state to Powering down when already Down (vm:624) 2019-04-13 21:40:47,733+0200 INFO (jsonrpc/2) [virt.vm] (vmId='40eaf7e2-659c-4790-8362-49fe12c5f61a') Stopping connection (guestagent:455) 2019-04-13 21:40:47,733+0200 INFO (jsonrpc/2) [vdsm.api] START teardownImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7cb20658-d49a-4a0b-b168-1680a56f8292', volUUID=None) from=::ffff:192.168.0.19,51130, task_id=70144602-efc3-4a1b-ae13-c96db0f0e8fd (api:48) 2019-04-13 21:40:47,734+0200 INFO (jsonrpc/2) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/7cb20658-d49a-4a0b-b168-1680a56f8292' (fileSD:606) 2019-04-13 21:40:47,734+0200 INFO (jsonrpc/2) [vdsm.api] FINISH teardownImage return=None from=::ffff:192.168.0.19,51130, task_id=70144602-efc3-4a1b-ae13-c96db0f0e8fd (api:54) 2019-04-13 21:40:47,734+0200 INFO (jsonrpc/2) [virt.vm] (vmId='40eaf7e2-659c-4790-8362-49fe12c5f61a') Stopping connection (guestagent:455) 2019-04-13 21:40:47,740+0200 WARN (jsonrpc/2) [root] Attempting to remove a non existing network: ovirtmgmt/40eaf7e2-659c-4790-8362-49fe12c5f61a (libvirtnetwork:198) 2019-04-13 21:40:47,740+0200 WARN (jsonrpc/2) [root] Attempting to remove a non existing net user: ovirtmgmt/40eaf7e2-659c-4790-8362-49fe12c5f61a (libvirtnetwork:205) 2019-04-13 21:40:47,741+0200 INFO (jsonrpc/2) [vdsm.api] START inappropriateDevices(thiefId='40eaf7e2-659c-4790-8362-49fe12c5f61a') from=::ffff:192.168.0.19,51130, task_id=73cc23dd-653b-4b96-af03-5f8a0d7b4c9a (api:48) 2019-04-13 21:40:47,743+0200 INFO (jsonrpc/2) [vdsm.api] FINISH inappropriateDevices return=None from=::ffff:192.168.0.19,51130, task_id=73cc23dd-653b-4b96-af03-5f8a0d7b4c9a (api:54) 2019-04-13 21:40:47,984+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:40:47,985+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:40:47,986+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:40:47,987+0200 INFO (jsonrpc/2) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/50_vhostmd: rc=0 err= (hooks:114) 2019-04-13 21:40:48,172+0200 INFO (jsonrpc/2) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/delete_vhostuserclient_hook: rc=0 err= (hooks:114) 2019-04-13 21:40:48,174+0200 INFO (jsonrpc/2) [api.virt] FINISH destroy return={'status': {'message': 'Machine destroyed', 'code': 0}} from=::ffff:192.168.0.19,51130, vmId=40eaf7e2-659c-4790-8362-49fe12c5f61a (api:54) 2019-04-13 21:40:48,174+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call VM.destroy succeeded in 0.44 seconds (__init__:312) 2019-04-13 21:40:48,204+0200 INFO (jsonrpc/7) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:192.168.0.19,51130, vmId=cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6 (api:48) 2019-04-13 21:40:48,204+0200 INFO (jsonrpc/7) [virt.vm] (vmId='cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6') Release VM resources (vm:5146) 2019-04-13 21:40:48,204+0200 WARN (jsonrpc/7) [virt.vm] (vmId='cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6') trying to set state to Powering down when already Down (vm:624) 2019-04-13 21:40:48,204+0200 INFO (jsonrpc/7) [virt.vm] (vmId='cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6') Stopping connection (guestagent:455) 2019-04-13 21:40:48,205+0200 INFO (jsonrpc/7) [vdsm.api] START teardownImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID=None) from=::ffff:192.168.0.19,51130, task_id=55deeb99-8c80-4da1-9598-e6903c5fc85b (api:48) 2019-04-13 21:40:48,206+0200 INFO (jsonrpc/7) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/0066d0ee-95b6-4554-bf20-84337780dfe8' (fileSD:606) 2019-04-13 21:40:48,206+0200 INFO (jsonrpc/7) [vdsm.api] FINISH teardownImage return=None from=::ffff:192.168.0.19,51130, task_id=55deeb99-8c80-4da1-9598-e6903c5fc85b (api:54) 2019-04-13 21:40:48,206+0200 INFO (jsonrpc/7) [virt.vm] (vmId='cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6') Stopping connection (guestagent:455) 2019-04-13 21:40:48,206+0200 WARN (jsonrpc/7) [root] Attempting to remove a non existing network: ovirtmgmt/cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6 (libvirtnetwork:198) 2019-04-13 21:40:48,206+0200 WARN (jsonrpc/7) [root] Attempting to remove a non existing net user: ovirtmgmt/cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6 (libvirtnetwork:205) 2019-04-13 21:40:48,208+0200 INFO (jsonrpc/7) [vdsm.api] START inappropriateDevices(thiefId='cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6') from=::ffff:192.168.0.19,51130, task_id=4734106a-f797-485c-8528-37340a76de48 (api:48) 2019-04-13 21:40:48,210+0200 INFO (jsonrpc/7) [vdsm.api] FINISH inappropriateDevices return=None from=::ffff:192.168.0.19,51130, task_id=4734106a-f797-485c-8528-37340a76de48 (api:54) 2019-04-13 21:40:48,452+0200 INFO (jsonrpc/7) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/50_vhostmd: rc=0 err= (hooks:114) 2019-04-13 21:40:48,636+0200 INFO (jsonrpc/7) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/delete_vhostuserclient_hook: rc=0 err= (hooks:114) 2019-04-13 21:40:48,637+0200 INFO (jsonrpc/7) [api.virt] FINISH destroy return={'status': {'message': 'Machine destroyed', 'code': 0}} from=::ffff:192.168.0.19,51130, vmId=cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6 (api:54) 2019-04-13 21:40:48,637+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.destroy succeeded in 0.44 seconds (__init__:312) 2019-04-13 21:40:48,674+0200 INFO (jsonrpc/5) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:192.168.0.19,51130, vmId=6772f848-4874-4785-9bb0-88d40bfb71c4 (api:48) 2019-04-13 21:40:48,675+0200 INFO (jsonrpc/5) [virt.vm] (vmId='6772f848-4874-4785-9bb0-88d40bfb71c4') Release VM resources (vm:5146) 2019-04-13 21:40:48,675+0200 WARN (jsonrpc/5) [virt.vm] (vmId='6772f848-4874-4785-9bb0-88d40bfb71c4') trying to set state to Powering down when already Down (vm:624) 2019-04-13 21:40:48,675+0200 INFO (jsonrpc/5) [virt.vm] (vmId='6772f848-4874-4785-9bb0-88d40bfb71c4') Stopping connection (guestagent:455) 2019-04-13 21:40:48,675+0200 INFO (jsonrpc/5) [vdsm.api] START teardownImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='467e6c8c-7c9f-4fc1-866b-3f96940305a9', volUUID=None) from=::ffff:192.168.0.19,51130, task_id=10d4ba0c-1ed7-46d2-b0d0-737c55a77a0a (api:48) 2019-04-13 21:40:48,676+0200 INFO (jsonrpc/5) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/467e6c8c-7c9f-4fc1-866b-3f96940305a9' (fileSD:606) 2019-04-13 21:40:48,676+0200 INFO (jsonrpc/5) [vdsm.api] FINISH teardownImage return=None from=::ffff:192.168.0.19,51130, task_id=10d4ba0c-1ed7-46d2-b0d0-737c55a77a0a (api:54) 2019-04-13 21:40:48,676+0200 INFO (jsonrpc/5) [virt.vm] (vmId='6772f848-4874-4785-9bb0-88d40bfb71c4') Stopping connection (guestagent:455) 2019-04-13 21:40:48,677+0200 WARN (jsonrpc/5) [root] Attempting to remove a non existing network: ovirtmgmt/6772f848-4874-4785-9bb0-88d40bfb71c4 (libvirtnetwork:198) 2019-04-13 21:40:48,677+0200 WARN (jsonrpc/5) [root] Attempting to remove a non existing net user: ovirtmgmt/6772f848-4874-4785-9bb0-88d40bfb71c4 (libvirtnetwork:205) 2019-04-13 21:40:48,679+0200 INFO (jsonrpc/5) [vdsm.api] START inappropriateDevices(thiefId='6772f848-4874-4785-9bb0-88d40bfb71c4') from=::ffff:192.168.0.19,51130, task_id=f46dbcc6-c350-4c7c-94ad-b1b7aa80b78b (api:48) 2019-04-13 21:40:48,680+0200 INFO (jsonrpc/5) [vdsm.api] FINISH inappropriateDevices return=None from=::ffff:192.168.0.19,51130, task_id=f46dbcc6-c350-4c7c-94ad-b1b7aa80b78b (api:54) 2019-04-13 21:40:48,922+0200 INFO (jsonrpc/5) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/50_vhostmd: rc=0 err= (hooks:114) 2019-04-13 21:40:49,106+0200 INFO (jsonrpc/5) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/delete_vhostuserclient_hook: rc=0 err= (hooks:114) 2019-04-13 21:40:49,108+0200 INFO (jsonrpc/5) [api.virt] FINISH destroy return={'status': {'message': 'Machine destroyed', 'code': 0}} from=::ffff:192.168.0.19,51130, vmId=6772f848-4874-4785-9bb0-88d40bfb71c4 (api:54) 2019-04-13 21:40:49,108+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.destroy succeeded in 0.44 seconds (__init__:312) 2019-04-13 21:40:49,141+0200 INFO (jsonrpc/4) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:192.168.0.19,51130, vmId=afa2c17f-39e4-4168-aad7-2548dd19cc26 (api:48) 2019-04-13 21:40:49,141+0200 INFO (jsonrpc/4) [virt.vm] (vmId='afa2c17f-39e4-4168-aad7-2548dd19cc26') Release VM resources (vm:5146) 2019-04-13 21:40:49,141+0200 WARN (jsonrpc/4) [virt.vm] (vmId='afa2c17f-39e4-4168-aad7-2548dd19cc26') trying to set state to Powering down when already Down (vm:624) 2019-04-13 21:40:49,141+0200 INFO (jsonrpc/4) [virt.vm] (vmId='afa2c17f-39e4-4168-aad7-2548dd19cc26') Stopping connection (guestagent:455) 2019-04-13 21:40:49,142+0200 INFO (jsonrpc/4) [vdsm.api] START teardownImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='e4b540ec-1e39-4349-b79b-f446f0b74237', volUUID=None) from=::ffff:192.168.0.19,51130, task_id=54f24e6d-bdec-4006-83d6-d40c0226d335 (api:48) 2019-04-13 21:40:49,142+0200 INFO (jsonrpc/4) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/e4b540ec-1e39-4349-b79b-f446f0b74237' (fileSD:606) 2019-04-13 21:40:49,142+0200 INFO (jsonrpc/4) [vdsm.api] FINISH teardownImage return=None from=::ffff:192.168.0.19,51130, task_id=54f24e6d-bdec-4006-83d6-d40c0226d335 (api:54) 2019-04-13 21:40:49,143+0200 INFO (jsonrpc/4) [virt.vm] (vmId='afa2c17f-39e4-4168-aad7-2548dd19cc26') Stopping connection (guestagent:455) 2019-04-13 21:40:49,143+0200 WARN (jsonrpc/4) [root] Attempting to remove a non existing network: ovirtmgmt/afa2c17f-39e4-4168-aad7-2548dd19cc26 (libvirtnetwork:198) 2019-04-13 21:40:49,143+0200 WARN (jsonrpc/4) [root] Attempting to remove a non existing net user: ovirtmgmt/afa2c17f-39e4-4168-aad7-2548dd19cc26 (libvirtnetwork:205) 2019-04-13 21:40:49,145+0200 INFO (jsonrpc/4) [vdsm.api] START inappropriateDevices(thiefId='afa2c17f-39e4-4168-aad7-2548dd19cc26') from=::ffff:192.168.0.19,51130, task_id=f27b90c4-a074-421a-a7aa-88a938fc4b77 (api:48) 2019-04-13 21:40:49,146+0200 INFO (jsonrpc/4) [vdsm.api] FINISH inappropriateDevices return=None from=::ffff:192.168.0.19,51130, task_id=f27b90c4-a074-421a-a7aa-88a938fc4b77 (api:54) 2019-04-13 21:40:49,389+0200 INFO (jsonrpc/4) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/50_vhostmd: rc=0 err= (hooks:114) 2019-04-13 21:40:49,574+0200 INFO (jsonrpc/4) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/delete_vhostuserclient_hook: rc=0 err= (hooks:114) 2019-04-13 21:40:49,575+0200 INFO (jsonrpc/4) [api.virt] FINISH destroy return={'status': {'message': 'Machine destroyed', 'code': 0}} from=::ffff:192.168.0.19,51130, vmId=afa2c17f-39e4-4168-aad7-2548dd19cc26 (api:54) 2019-04-13 21:40:49,576+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.destroy succeeded in 0.43 seconds (__init__:312) 2019-04-13 21:40:49,615+0200 INFO (jsonrpc/6) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:192.168.0.19,51130, vmId=be42701b-3f74-4c81-b6ef-5f7e7806554a (api:48) 2019-04-13 21:40:49,615+0200 INFO (jsonrpc/6) [virt.vm] (vmId='be42701b-3f74-4c81-b6ef-5f7e7806554a') Release VM resources (vm:5146) 2019-04-13 21:40:49,615+0200 WARN (jsonrpc/6) [virt.vm] (vmId='be42701b-3f74-4c81-b6ef-5f7e7806554a') trying to set state to Powering down when already Down (vm:624) 2019-04-13 21:40:49,615+0200 INFO (jsonrpc/6) [virt.vm] (vmId='be42701b-3f74-4c81-b6ef-5f7e7806554a') Stopping connection (guestagent:455) 2019-04-13 21:40:49,615+0200 INFO (jsonrpc/6) [vdsm.api] START teardownImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='04dc2b89-95cd-49d4-aee2-6927359e8ca1', volUUID=None) from=::ffff:192.168.0.19,51130, task_id=4e055b40-919d-4ee2-9249-4bbb3efb6349 (api:48) 2019-04-13 21:40:49,616+0200 INFO (jsonrpc/6) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/04dc2b89-95cd-49d4-aee2-6927359e8ca1' (fileSD:606) 2019-04-13 21:40:49,616+0200 INFO (jsonrpc/6) [vdsm.api] FINISH teardownImage return=None from=::ffff:192.168.0.19,51130, task_id=4e055b40-919d-4ee2-9249-4bbb3efb6349 (api:54) 2019-04-13 21:40:49,616+0200 INFO (jsonrpc/6) [virt.vm] (vmId='be42701b-3f74-4c81-b6ef-5f7e7806554a') Stopping connection (guestagent:455) 2019-04-13 21:40:49,617+0200 WARN (jsonrpc/6) [root] Attempting to remove a non existing network: ovirtmgmt/be42701b-3f74-4c81-b6ef-5f7e7806554a (libvirtnetwork:198) 2019-04-13 21:40:49,617+0200 WARN (jsonrpc/6) [root] Attempting to remove a non existing net user: ovirtmgmt/be42701b-3f74-4c81-b6ef-5f7e7806554a (libvirtnetwork:205) 2019-04-13 21:40:49,619+0200 INFO (jsonrpc/6) [vdsm.api] START inappropriateDevices(thiefId='be42701b-3f74-4c81-b6ef-5f7e7806554a') from=::ffff:192.168.0.19,51130, task_id=e7f1294d-845d-49e8-9cfb-3432cc07c5a0 (api:48) 2019-04-13 21:40:49,620+0200 INFO (jsonrpc/6) [vdsm.api] FINISH inappropriateDevices return=None from=::ffff:192.168.0.19,51130, task_id=e7f1294d-845d-49e8-9cfb-3432cc07c5a0 (api:54) 2019-04-13 21:40:49,862+0200 INFO (jsonrpc/6) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/50_vhostmd: rc=0 err= (hooks:114) 2019-04-13 21:40:50,047+0200 INFO (jsonrpc/6) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/delete_vhostuserclient_hook: rc=0 err= (hooks:114) 2019-04-13 21:40:50,049+0200 INFO (jsonrpc/6) [api.virt] FINISH destroy return={'status': {'message': 'Machine destroyed', 'code': 0}} from=::ffff:192.168.0.19,51130, vmId=be42701b-3f74-4c81-b6ef-5f7e7806554a (api:54) 2019-04-13 21:40:50,049+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.destroy succeeded in 0.44 seconds (__init__:312) 2019-04-13 21:40:50,085+0200 INFO (jsonrpc/1) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:192.168.0.19,51130, vmId=5a326302-86c1-41b5-ba58-793232299713 (api:48) 2019-04-13 21:40:50,085+0200 INFO (jsonrpc/1) [virt.vm] (vmId='5a326302-86c1-41b5-ba58-793232299713') Release VM resources (vm:5146) 2019-04-13 21:40:50,086+0200 WARN (jsonrpc/1) [virt.vm] (vmId='5a326302-86c1-41b5-ba58-793232299713') trying to set state to Powering down when already Down (vm:624) 2019-04-13 21:40:50,086+0200 INFO (jsonrpc/1) [virt.vm] (vmId='5a326302-86c1-41b5-ba58-793232299713') Stopping connection (guestagent:455) 2019-04-13 21:40:50,086+0200 INFO (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ecb12678-f79f-4648-8085-57ffdd2ef8ac', volUUID=None) from=::ffff:192.168.0.19,51130, task_id=67985d50-c37e-407f-a287-f74d2aeb4545 (api:48) 2019-04-13 21:40:50,087+0200 INFO (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/ecb12678-f79f-4648-8085-57ffdd2ef8ac' (fileSD:606) 2019-04-13 21:40:50,087+0200 INFO (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:192.168.0.19,51130, task_id=67985d50-c37e-407f-a287-f74d2aeb4545 (api:54) 2019-04-13 21:40:50,087+0200 INFO (jsonrpc/1) [virt.vm] (vmId='5a326302-86c1-41b5-ba58-793232299713') Stopping connection (guestagent:455) 2019-04-13 21:40:50,087+0200 WARN (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/5a326302-86c1-41b5-ba58-793232299713 (libvirtnetwork:198) 2019-04-13 21:40:50,088+0200 WARN (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/5a326302-86c1-41b5-ba58-793232299713 (libvirtnetwork:205) 2019-04-13 21:40:50,089+0200 INFO (jsonrpc/1) [vdsm.api] START inappropriateDevices(thiefId='5a326302-86c1-41b5-ba58-793232299713') from=::ffff:192.168.0.19,51130, task_id=e520524c-77d0-4974-bf5c-7835bbed000c (api:48) 2019-04-13 21:40:50,091+0200 INFO (jsonrpc/1) [vdsm.api] FINISH inappropriateDevices return=None from=::ffff:192.168.0.19,51130, task_id=e520524c-77d0-4974-bf5c-7835bbed000c (api:54) 2019-04-13 21:40:50,113+0200 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:40:50,135+0200 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=b4cca847-dd74-4b67-b7ce-621eed5fd5b5 (api:48) 2019-04-13 21:40:50,136+0200 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000603988', 'lastCheck': '5.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000445815', 'lastCheck': '5.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000537318', 'lastCheck': '5.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000308135', 'lastCheck': '5.5', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=b4cca847-dd74-4b67-b7ce-621eed5fd5b5 (api:54) 2019-04-13 21:40:50,137+0200 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=a60fc226-8c69-4540-b3d2-1fc923edc9b1 (api:48) 2019-04-13 21:40:50,137+0200 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=a60fc226-8c69-4540-b3d2-1fc923edc9b1 (api:54) 2019-04-13 21:40:50,145+0200 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.81', 'nodeIndex': 1, 'cpuSys': '0.46', 'cpuIdle': '98.73'}, '21': {'cpuUser': '0.52', 'nodeIndex': 1, 'cpuSys': '0.06', 'cpuIdle': '99.42'}, '22': {'cpuUser': '0.06', 'nodeIndex': 1, 'cpuSys': '0.06', 'cpuIdle': '99.88'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '7.51', 'nodeIndex': 0, 'cpuSys': '3.06', 'cpuIdle': '89.43'}, '0': {'cpuUser': '0.29', 'nodeIndex': 0, 'cpuSys': '0.23', 'cpuIdle': '99.48'}, '3': {'cpuUser': '0.46', 'nodeIndex': 0, 'cpuSys': '0.29', 'cpuIdle': '99.25'}, '2': {'cpuUser': '0.52', 'nodeIndex': 0, 'cpuSys': '0.46', 'cpuIdle': '99.02'}, '5': {'cpuUser': '0.40', 'nodeIndex': 0, 'cpuSys': '0.29', 'cpuIdle': '99.31'}, '4': {'cpuUser': '1.39', 'nodeIndex': 0, 'cpuSys': '0.29', 'cpuIdle': '98.32'}, '7': {'cpuUser': '0.17', 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.43'}, '6': {'cpuUser': '0.12', 'nodeIndex': 1, 'cpuSys': '0.29', 'cpuIdle': '99.59'}, '9': {'cpuUser': '0.64', 'nodeIndex': 1, 'cpuSys': '0.29', 'cpuIdle': '99.07'}, '8': {'cpuUser': '0.23', 'nodeIndex': 1, 'cpuSys': '0.35', 'cpuIdle': '99.42'}, '11': {'cpuUser': '0.12', 'nodeIndex': 1, 'cpuSys': '0.06', 'cpuIdle': '99.82'}, '10': {'cpuUser': '0.17', 'nodeIndex': 1, 'cpuSys': '0.23', 'cpuIdle': '99.60'}, '13': {'cpuUser': '0.29', 'nodeIndex': 0, 'cpuSys': '0.58', 'cpuIdle': '99.13'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.06', 'cpuIdle': '99.94'}, '14': {'cpuUser': '0.06', 'nodeIndex': 0, 'cpuSys': '0.06', 'cpuIdle': '99.88'}, '17': {'cpuUser': '0.06', 'nodeIndex': 0, 'cpuSys': '0.06', 'cpuIdle': '99.88'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '9.76', 'nodeIndex': 1, 'cpuSys': '1.15', 'cpuIdle': '89.09'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95669'}, '0': {'memPercent': 3, 'memFree': '95514'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 9, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000603988', 'lastCheck': '5.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000537318', 'lastCheck': '5.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000308135', 'lastCheck': '5.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000445815', 'lastCheck': '5.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184450.135083, 'rx': '1866', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '330502', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184450.135083, 'rx': '312248', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184450.135083, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '321644', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184450.135083, 'rx': '292692', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '1139714', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184450.135083, 'rx': '1139714', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184450.135083, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '4733', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184450.135083, 'rx': '310084', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '325769', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184450.135083, 'rx': '2164', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184450.135083, 'rx': '9652', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184450.135083, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184450.135083, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184450.135083, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184450.135083, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184450.135083, 'rx': '6578', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '42', 'ksmPages': 100, 'elapsedTime': '38.13', 'cpuLoad': '0.71', 'cpuSys': '0.36', 'diskStats': {'/var/log': {'free': '47810'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47810'}}, 'cpuUserVdsmd': '7.79', 'netConfigDirty': 'False', 'memCommitted': 26624, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191807, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:40:50 GMT', 'cpuUser': '0.98', 'memFree': 191551, 'cpuIdle': '98.66', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '2.14'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:40:50,148+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.04 seconds (__init__:312) 2019-04-13 21:40:50,333+0200 INFO (jsonrpc/1) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/50_vhostmd: rc=0 err= (hooks:114) 2019-04-13 21:40:50,447+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=2fbb1d63-41be-4800-b57a-869829b739e8 (api:48) 2019-04-13 21:40:50,447+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000603988', 'lastCheck': '5.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000445815', 'lastCheck': '5.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000537318', 'lastCheck': '5.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000308135', 'lastCheck': '5.8', 'valid': True}} from=internal, task_id=2fbb1d63-41be-4800-b57a-869829b739e8 (api:54) 2019-04-13 21:40:50,535+0200 INFO (jsonrpc/1) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/delete_vhostuserclient_hook: rc=0 err= (hooks:114) 2019-04-13 21:40:50,537+0200 INFO (jsonrpc/1) [api.virt] FINISH destroy return={'status': {'message': 'Machine destroyed', 'code': 0}} from=::ffff:192.168.0.19,51130, vmId=5a326302-86c1-41b5-ba58-793232299713 (api:54) 2019-04-13 21:40:50,537+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.destroy succeeded in 0.46 seconds (__init__:312) 2019-04-13 21:40:50,567+0200 INFO (jsonrpc/3) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:192.168.0.19,51130, vmId=a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3 (api:48) 2019-04-13 21:40:50,568+0200 INFO (jsonrpc/3) [virt.vm] (vmId='a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3') Release VM resources (vm:5146) 2019-04-13 21:40:50,568+0200 WARN (jsonrpc/3) [virt.vm] (vmId='a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3') trying to set state to Powering down when already Down (vm:624) 2019-04-13 21:40:50,568+0200 INFO (jsonrpc/3) [virt.vm] (vmId='a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3') Stopping connection (guestagent:455) 2019-04-13 21:40:50,568+0200 INFO (jsonrpc/3) [vdsm.api] START teardownImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='614d3cdc-7f35-4c25-9b7b-711f0a5e6644', volUUID=None) from=::ffff:192.168.0.19,51130, task_id=69539084-97ce-45b6-942c-c2b651ec3829 (api:48) 2019-04-13 21:40:50,569+0200 INFO (jsonrpc/3) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/614d3cdc-7f35-4c25-9b7b-711f0a5e6644' (fileSD:606) 2019-04-13 21:40:50,569+0200 INFO (jsonrpc/3) [vdsm.api] FINISH teardownImage return=None from=::ffff:192.168.0.19,51130, task_id=69539084-97ce-45b6-942c-c2b651ec3829 (api:54) 2019-04-13 21:40:50,569+0200 INFO (jsonrpc/3) [virt.vm] (vmId='a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3') Stopping connection (guestagent:455) 2019-04-13 21:40:50,570+0200 WARN (jsonrpc/3) [root] Attempting to remove a non existing network: ovirtmgmt/a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3 (libvirtnetwork:198) 2019-04-13 21:40:50,570+0200 WARN (jsonrpc/3) [root] Attempting to remove a non existing net user: ovirtmgmt/a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3 (libvirtnetwork:205) 2019-04-13 21:40:50,572+0200 INFO (jsonrpc/3) [vdsm.api] START inappropriateDevices(thiefId='a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3') from=::ffff:192.168.0.19,51130, task_id=9079455e-3167-4376-b398-1d5425c6d310 (api:48) 2019-04-13 21:40:50,574+0200 INFO (jsonrpc/3) [vdsm.api] FINISH inappropriateDevices return=None from=::ffff:192.168.0.19,51130, task_id=9079455e-3167-4376-b398-1d5425c6d310 (api:54) 2019-04-13 21:40:50,816+0200 INFO (jsonrpc/3) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/50_vhostmd: rc=0 err= (hooks:114) 2019-04-13 21:40:51,000+0200 INFO (jsonrpc/3) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/delete_vhostuserclient_hook: rc=0 err= (hooks:114) 2019-04-13 21:40:51,002+0200 INFO (jsonrpc/3) [api.virt] FINISH destroy return={'status': {'message': 'Machine destroyed', 'code': 0}} from=::ffff:192.168.0.19,51130, vmId=a40d738c-e8ec-43a8-8ae3-1fe3ebf2dfa3 (api:54) 2019-04-13 21:40:51,002+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call VM.destroy succeeded in 0.43 seconds (__init__:312) 2019-04-13 21:40:51,043+0200 INFO (jsonrpc/2) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:192.168.0.19,51130, vmId=b5713713-ac04-4a91-95b8-3c3f17eba32f (api:48) 2019-04-13 21:40:51,043+0200 INFO (jsonrpc/2) [virt.vm] (vmId='b5713713-ac04-4a91-95b8-3c3f17eba32f') Release VM resources (vm:5146) 2019-04-13 21:40:51,043+0200 WARN (jsonrpc/2) [virt.vm] (vmId='b5713713-ac04-4a91-95b8-3c3f17eba32f') trying to set state to Powering down when already Down (vm:624) 2019-04-13 21:40:51,043+0200 INFO (jsonrpc/2) [virt.vm] (vmId='b5713713-ac04-4a91-95b8-3c3f17eba32f') Stopping connection (guestagent:455) 2019-04-13 21:40:51,044+0200 INFO (jsonrpc/2) [vdsm.api] START teardownImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fd3b80fd-49ad-44ac-9efd-1328300582cd', volUUID=None) from=::ffff:192.168.0.19,51130, task_id=40baff40-8cf0-4ad6-8b99-7b744ee55505 (api:48) 2019-04-13 21:40:51,044+0200 INFO (jsonrpc/2) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/fd3b80fd-49ad-44ac-9efd-1328300582cd' (fileSD:606) 2019-04-13 21:40:51,044+0200 INFO (jsonrpc/2) [vdsm.api] FINISH teardownImage return=None from=::ffff:192.168.0.19,51130, task_id=40baff40-8cf0-4ad6-8b99-7b744ee55505 (api:54) 2019-04-13 21:40:51,045+0200 INFO (jsonrpc/2) [virt.vm] (vmId='b5713713-ac04-4a91-95b8-3c3f17eba32f') Stopping connection (guestagent:455) 2019-04-13 21:40:51,045+0200 WARN (jsonrpc/2) [root] Attempting to remove a non existing network: ovirtmgmt/b5713713-ac04-4a91-95b8-3c3f17eba32f (libvirtnetwork:198) 2019-04-13 21:40:51,045+0200 WARN (jsonrpc/2) [root] Attempting to remove a non existing net user: ovirtmgmt/b5713713-ac04-4a91-95b8-3c3f17eba32f (libvirtnetwork:205) 2019-04-13 21:40:51,047+0200 INFO (jsonrpc/2) [vdsm.api] START inappropriateDevices(thiefId='b5713713-ac04-4a91-95b8-3c3f17eba32f') from=::ffff:192.168.0.19,51130, task_id=2c3a9fca-36fa-4e79-a2ea-16d80e94973f (api:48) 2019-04-13 21:40:51,048+0200 INFO (jsonrpc/2) [vdsm.api] FINISH inappropriateDevices return=None from=::ffff:192.168.0.19,51130, task_id=2c3a9fca-36fa-4e79-a2ea-16d80e94973f (api:54) 2019-04-13 21:40:51,290+0200 INFO (jsonrpc/2) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/50_vhostmd: rc=0 err= (hooks:114) 2019-04-13 21:40:51,474+0200 INFO (jsonrpc/2) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/delete_vhostuserclient_hook: rc=0 err= (hooks:114) 2019-04-13 21:40:51,476+0200 INFO (jsonrpc/2) [api.virt] FINISH destroy return={'status': {'message': 'Machine destroyed', 'code': 0}} from=::ffff:192.168.0.19,51130, vmId=b5713713-ac04-4a91-95b8-3c3f17eba32f (api:54) 2019-04-13 21:40:51,476+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call VM.destroy succeeded in 0.43 seconds (__init__:312) 2019-04-13 21:40:51,508+0200 INFO (jsonrpc/7) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:192.168.0.19,51130, vmId=139534f6-1741-4ded-8ea6-ea4e3ecdad1f (api:48) 2019-04-13 21:40:51,508+0200 INFO (jsonrpc/7) [virt.vm] (vmId='139534f6-1741-4ded-8ea6-ea4e3ecdad1f') Release VM resources (vm:5146) 2019-04-13 21:40:51,508+0200 WARN (jsonrpc/7) [virt.vm] (vmId='139534f6-1741-4ded-8ea6-ea4e3ecdad1f') trying to set state to Powering down when already Down (vm:624) 2019-04-13 21:40:51,508+0200 INFO (jsonrpc/7) [virt.vm] (vmId='139534f6-1741-4ded-8ea6-ea4e3ecdad1f') Stopping connection (guestagent:455) 2019-04-13 21:40:51,509+0200 INFO (jsonrpc/7) [vdsm.api] START teardownImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='fc829fbd-ef00-4ae4-b06a-74a972251909', volUUID=None) from=::ffff:192.168.0.19,51130, task_id=8cef4ac1-0789-40d4-9969-d5e19c35aeec (api:48) 2019-04-13 21:40:51,510+0200 INFO (jsonrpc/7) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/fc829fbd-ef00-4ae4-b06a-74a972251909' (fileSD:606) 2019-04-13 21:40:51,510+0200 INFO (jsonrpc/7) [vdsm.api] FINISH teardownImage return=None from=::ffff:192.168.0.19,51130, task_id=8cef4ac1-0789-40d4-9969-d5e19c35aeec (api:54) 2019-04-13 21:40:51,510+0200 INFO (jsonrpc/7) [virt.vm] (vmId='139534f6-1741-4ded-8ea6-ea4e3ecdad1f') Stopping connection (guestagent:455) 2019-04-13 21:40:51,510+0200 WARN (jsonrpc/7) [root] Attempting to remove a non existing network: ovirtmgmt/139534f6-1741-4ded-8ea6-ea4e3ecdad1f (libvirtnetwork:198) 2019-04-13 21:40:51,510+0200 WARN (jsonrpc/7) [root] Attempting to remove a non existing net user: ovirtmgmt/139534f6-1741-4ded-8ea6-ea4e3ecdad1f (libvirtnetwork:205) 2019-04-13 21:40:51,512+0200 INFO (jsonrpc/7) [vdsm.api] START inappropriateDevices(thiefId='139534f6-1741-4ded-8ea6-ea4e3ecdad1f') from=::ffff:192.168.0.19,51130, task_id=bef26f20-eba9-4213-8e84-7539d81a5c2f (api:48) 2019-04-13 21:40:51,513+0200 INFO (jsonrpc/7) [vdsm.api] FINISH inappropriateDevices return=None from=::ffff:192.168.0.19,51130, task_id=bef26f20-eba9-4213-8e84-7539d81a5c2f (api:54) 2019-04-13 21:40:51,755+0200 INFO (jsonrpc/7) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/50_vhostmd: rc=0 err= (hooks:114) 2019-04-13 21:40:51,941+0200 INFO (jsonrpc/7) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/delete_vhostuserclient_hook: rc=0 err= (hooks:114) 2019-04-13 21:40:51,942+0200 INFO (jsonrpc/7) [api.virt] FINISH destroy return={'status': {'message': 'Machine destroyed', 'code': 0}} from=::ffff:192.168.0.19,51130, vmId=139534f6-1741-4ded-8ea6-ea4e3ecdad1f (api:54) 2019-04-13 21:40:51,942+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.destroy succeeded in 0.43 seconds (__init__:312) 2019-04-13 21:40:51,974+0200 INFO (jsonrpc/5) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:192.168.0.19,51130, vmId=5dd5db29-143f-4d70-9469-4aaa163b7f9d (api:48) 2019-04-13 21:40:51,974+0200 INFO (jsonrpc/5) [virt.vm] (vmId='5dd5db29-143f-4d70-9469-4aaa163b7f9d') Release VM resources (vm:5146) 2019-04-13 21:40:51,974+0200 WARN (jsonrpc/5) [virt.vm] (vmId='5dd5db29-143f-4d70-9469-4aaa163b7f9d') trying to set state to Powering down when already Down (vm:624) 2019-04-13 21:40:51,974+0200 INFO (jsonrpc/5) [virt.vm] (vmId='5dd5db29-143f-4d70-9469-4aaa163b7f9d') Stopping connection (guestagent:455) 2019-04-13 21:40:51,975+0200 INFO (jsonrpc/5) [vdsm.api] START teardownImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='22b8e06e-7cde-4dcb-8676-e0502f335cb7', volUUID=None) from=::ffff:192.168.0.19,51130, task_id=af00b90c-ec59-481b-b397-12709bfee0f6 (api:48) 2019-04-13 21:40:51,976+0200 INFO (jsonrpc/5) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/22b8e06e-7cde-4dcb-8676-e0502f335cb7' (fileSD:606) 2019-04-13 21:40:51,976+0200 INFO (jsonrpc/5) [vdsm.api] FINISH teardownImage return=None from=::ffff:192.168.0.19,51130, task_id=af00b90c-ec59-481b-b397-12709bfee0f6 (api:54) 2019-04-13 21:40:51,976+0200 INFO (jsonrpc/5) [virt.vm] (vmId='5dd5db29-143f-4d70-9469-4aaa163b7f9d') Stopping connection (guestagent:455) 2019-04-13 21:40:51,976+0200 WARN (jsonrpc/5) [root] Attempting to remove a non existing network: ovirtmgmt/5dd5db29-143f-4d70-9469-4aaa163b7f9d (libvirtnetwork:198) 2019-04-13 21:40:51,976+0200 WARN (jsonrpc/5) [root] Attempting to remove a non existing net user: ovirtmgmt/5dd5db29-143f-4d70-9469-4aaa163b7f9d (libvirtnetwork:205) 2019-04-13 21:40:51,980+0200 INFO (jsonrpc/5) [vdsm.api] START inappropriateDevices(thiefId='5dd5db29-143f-4d70-9469-4aaa163b7f9d') from=::ffff:192.168.0.19,51130, task_id=f6262e32-22ab-4f22-aa21-9d16e34f488a (api:48) 2019-04-13 21:40:51,981+0200 INFO (jsonrpc/5) [vdsm.api] FINISH inappropriateDevices return=None from=::ffff:192.168.0.19,51130, task_id=f6262e32-22ab-4f22-aa21-9d16e34f488a (api:54) 2019-04-13 21:40:52,223+0200 INFO (jsonrpc/5) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/50_vhostmd: rc=0 err= (hooks:114) 2019-04-13 21:40:52,409+0200 INFO (jsonrpc/5) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/delete_vhostuserclient_hook: rc=0 err= (hooks:114) 2019-04-13 21:40:52,411+0200 INFO (jsonrpc/5) [api.virt] FINISH destroy return={'status': {'message': 'Machine destroyed', 'code': 0}} from=::ffff:192.168.0.19,51130, vmId=5dd5db29-143f-4d70-9469-4aaa163b7f9d (api:54) 2019-04-13 21:40:52,411+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.destroy succeeded in 0.44 seconds (__init__:312) 2019-04-13 21:40:52,449+0200 INFO (jsonrpc/4) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:192.168.0.19,51130, vmId=0f8aff0a-7915-4214-a37c-c9fee6518869 (api:48) 2019-04-13 21:40:52,449+0200 INFO (jsonrpc/4) [virt.vm] (vmId='0f8aff0a-7915-4214-a37c-c9fee6518869') Release VM resources (vm:5146) 2019-04-13 21:40:52,449+0200 WARN (jsonrpc/4) [virt.vm] (vmId='0f8aff0a-7915-4214-a37c-c9fee6518869') trying to set state to Powering down when already Down (vm:624) 2019-04-13 21:40:52,449+0200 INFO (jsonrpc/4) [virt.vm] (vmId='0f8aff0a-7915-4214-a37c-c9fee6518869') Stopping connection (guestagent:455) 2019-04-13 21:40:52,450+0200 INFO (jsonrpc/4) [vdsm.api] START teardownImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd', volUUID=None) from=::ffff:192.168.0.19,51130, task_id=97268ed0-7fed-4eea-b026-ef3ed025ef80 (api:48) 2019-04-13 21:40:52,450+0200 INFO (jsonrpc/4) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/9ec5b9aa-58e9-44b8-b282-7f5e8134e2cd' (fileSD:606) 2019-04-13 21:40:52,450+0200 INFO (jsonrpc/4) [vdsm.api] FINISH teardownImage return=None from=::ffff:192.168.0.19,51130, task_id=97268ed0-7fed-4eea-b026-ef3ed025ef80 (api:54) 2019-04-13 21:40:52,451+0200 INFO (jsonrpc/4) [virt.vm] (vmId='0f8aff0a-7915-4214-a37c-c9fee6518869') Stopping connection (guestagent:455) 2019-04-13 21:40:52,451+0200 WARN (jsonrpc/4) [root] Attempting to remove a non existing network: ovirtmgmt/0f8aff0a-7915-4214-a37c-c9fee6518869 (libvirtnetwork:198) 2019-04-13 21:40:52,451+0200 WARN (jsonrpc/4) [root] Attempting to remove a non existing net user: ovirtmgmt/0f8aff0a-7915-4214-a37c-c9fee6518869 (libvirtnetwork:205) 2019-04-13 21:40:52,453+0200 INFO (jsonrpc/4) [vdsm.api] START inappropriateDevices(thiefId='0f8aff0a-7915-4214-a37c-c9fee6518869') from=::ffff:192.168.0.19,51130, task_id=a94e0d74-0386-499c-a3ac-8ce702c29af2 (api:48) 2019-04-13 21:40:52,454+0200 INFO (jsonrpc/4) [vdsm.api] FINISH inappropriateDevices return=None from=::ffff:192.168.0.19,51130, task_id=a94e0d74-0386-499c-a3ac-8ce702c29af2 (api:54) 2019-04-13 21:40:52,696+0200 INFO (jsonrpc/4) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/50_vhostmd: rc=0 err= (hooks:114) 2019-04-13 21:40:52,881+0200 INFO (jsonrpc/4) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/delete_vhostuserclient_hook: rc=0 err= (hooks:114) 2019-04-13 21:40:52,883+0200 INFO (jsonrpc/4) [api.virt] FINISH destroy return={'status': {'message': 'Machine destroyed', 'code': 0}} from=::ffff:192.168.0.19,51130, vmId=0f8aff0a-7915-4214-a37c-c9fee6518869 (api:54) 2019-04-13 21:40:52,883+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.destroy succeeded in 0.43 seconds (__init__:312) 2019-04-13 21:40:52,922+0200 INFO (jsonrpc/6) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:192.168.0.19,51130, vmId=dd24e11f-40aa-47d2-9f0b-11cd7e83649d (api:48) 2019-04-13 21:40:52,922+0200 INFO (jsonrpc/6) [virt.vm] (vmId='dd24e11f-40aa-47d2-9f0b-11cd7e83649d') Release VM resources (vm:5146) 2019-04-13 21:40:52,922+0200 WARN (jsonrpc/6) [virt.vm] (vmId='dd24e11f-40aa-47d2-9f0b-11cd7e83649d') trying to set state to Powering down when already Down (vm:624) 2019-04-13 21:40:52,922+0200 INFO (jsonrpc/6) [virt.vm] (vmId='dd24e11f-40aa-47d2-9f0b-11cd7e83649d') Stopping connection (guestagent:455) 2019-04-13 21:40:52,923+0200 INFO (jsonrpc/6) [vdsm.api] START teardownImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='ce44616c-3479-4636-83e3-c1c967a108d7', volUUID=None) from=::ffff:192.168.0.19,51130, task_id=ad0d2b64-b69a-43bb-b760-b0a54645eb7a (api:48) 2019-04-13 21:40:52,923+0200 INFO (jsonrpc/6) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/ce44616c-3479-4636-83e3-c1c967a108d7' (fileSD:606) 2019-04-13 21:40:52,924+0200 INFO (jsonrpc/6) [vdsm.api] FINISH teardownImage return=None from=::ffff:192.168.0.19,51130, task_id=ad0d2b64-b69a-43bb-b760-b0a54645eb7a (api:54) 2019-04-13 21:40:52,924+0200 INFO (jsonrpc/6) [virt.vm] (vmId='dd24e11f-40aa-47d2-9f0b-11cd7e83649d') Stopping connection (guestagent:455) 2019-04-13 21:40:52,924+0200 WARN (jsonrpc/6) [root] Attempting to remove a non existing network: ovirtmgmt/dd24e11f-40aa-47d2-9f0b-11cd7e83649d (libvirtnetwork:198) 2019-04-13 21:40:52,924+0200 WARN (jsonrpc/6) [root] Attempting to remove a non existing net user: ovirtmgmt/dd24e11f-40aa-47d2-9f0b-11cd7e83649d (libvirtnetwork:205) 2019-04-13 21:40:52,926+0200 INFO (jsonrpc/6) [vdsm.api] START inappropriateDevices(thiefId='dd24e11f-40aa-47d2-9f0b-11cd7e83649d') from=::ffff:192.168.0.19,51130, task_id=8ea6e75f-2546-4cab-adb0-8e2ba6a5a118 (api:48) 2019-04-13 21:40:52,927+0200 INFO (jsonrpc/6) [vdsm.api] FINISH inappropriateDevices return=None from=::ffff:192.168.0.19,51130, task_id=8ea6e75f-2546-4cab-adb0-8e2ba6a5a118 (api:54) 2019-04-13 21:40:53,170+0200 INFO (jsonrpc/6) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/50_vhostmd: rc=0 err= (hooks:114) 2019-04-13 21:40:53,355+0200 INFO (jsonrpc/6) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/delete_vhostuserclient_hook: rc=0 err= (hooks:114) 2019-04-13 21:40:53,356+0200 INFO (jsonrpc/6) [api.virt] FINISH destroy return={'status': {'message': 'Machine destroyed', 'code': 0}} from=::ffff:192.168.0.19,51130, vmId=dd24e11f-40aa-47d2-9f0b-11cd7e83649d (api:54) 2019-04-13 21:40:53,357+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.destroy succeeded in 0.43 seconds (__init__:312) 2019-04-13 21:40:53,387+0200 INFO (jsonrpc/0) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:192.168.0.19,51130, vmId=c8df4609-fe26-4b73-9137-57724aff3ca4 (api:48) 2019-04-13 21:40:53,387+0200 INFO (jsonrpc/0) [virt.vm] (vmId='c8df4609-fe26-4b73-9137-57724aff3ca4') Release VM resources (vm:5146) 2019-04-13 21:40:53,387+0200 WARN (jsonrpc/0) [virt.vm] (vmId='c8df4609-fe26-4b73-9137-57724aff3ca4') trying to set state to Powering down when already Down (vm:624) 2019-04-13 21:40:53,387+0200 INFO (jsonrpc/0) [virt.vm] (vmId='c8df4609-fe26-4b73-9137-57724aff3ca4') Stopping connection (guestagent:455) 2019-04-13 21:40:53,388+0200 INFO (jsonrpc/0) [vdsm.api] START teardownImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='be884b3f-db5e-40e7-87dc-b98fb99fb95b', volUUID=None) from=::ffff:192.168.0.19,51130, task_id=73ccdbf5-0725-4c89-b714-516cca718a9a (api:48) 2019-04-13 21:40:53,389+0200 INFO (jsonrpc/0) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/be884b3f-db5e-40e7-87dc-b98fb99fb95b' (fileSD:606) 2019-04-13 21:40:53,389+0200 INFO (jsonrpc/0) [vdsm.api] FINISH teardownImage return=None from=::ffff:192.168.0.19,51130, task_id=73ccdbf5-0725-4c89-b714-516cca718a9a (api:54) 2019-04-13 21:40:53,389+0200 INFO (jsonrpc/0) [virt.vm] (vmId='c8df4609-fe26-4b73-9137-57724aff3ca4') Stopping connection (guestagent:455) 2019-04-13 21:40:53,390+0200 WARN (jsonrpc/0) [root] Attempting to remove a non existing network: ovirtmgmt/c8df4609-fe26-4b73-9137-57724aff3ca4 (libvirtnetwork:198) 2019-04-13 21:40:53,390+0200 WARN (jsonrpc/0) [root] Attempting to remove a non existing net user: ovirtmgmt/c8df4609-fe26-4b73-9137-57724aff3ca4 (libvirtnetwork:205) 2019-04-13 21:40:53,392+0200 INFO (jsonrpc/0) [vdsm.api] START inappropriateDevices(thiefId='c8df4609-fe26-4b73-9137-57724aff3ca4') from=::ffff:192.168.0.19,51130, task_id=e77563ed-4202-4e52-ad6f-4990ea89aec3 (api:48) 2019-04-13 21:40:53,393+0200 INFO (jsonrpc/0) [vdsm.api] FINISH inappropriateDevices return=None from=::ffff:192.168.0.19,51130, task_id=e77563ed-4202-4e52-ad6f-4990ea89aec3 (api:54) 2019-04-13 21:40:53,636+0200 INFO (jsonrpc/0) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/50_vhostmd: rc=0 err= (hooks:114) 2019-04-13 21:40:53,820+0200 INFO (jsonrpc/0) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/delete_vhostuserclient_hook: rc=0 err= (hooks:114) 2019-04-13 21:40:53,821+0200 INFO (jsonrpc/0) [api.virt] FINISH destroy return={'status': {'message': 'Machine destroyed', 'code': 0}} from=::ffff:192.168.0.19,51130, vmId=c8df4609-fe26-4b73-9137-57724aff3ca4 (api:54) 2019-04-13 21:40:53,822+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.destroy succeeded in 0.44 seconds (__init__:312) 2019-04-13 21:40:53,852+0200 INFO (jsonrpc/1) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:192.168.0.19,51130, vmId=7b5e0fdd-8757-49ce-acf9-3206a372e1c9 (api:48) 2019-04-13 21:40:53,853+0200 INFO (jsonrpc/1) [virt.vm] (vmId='7b5e0fdd-8757-49ce-acf9-3206a372e1c9') Release VM resources (vm:5146) 2019-04-13 21:40:53,853+0200 WARN (jsonrpc/1) [virt.vm] (vmId='7b5e0fdd-8757-49ce-acf9-3206a372e1c9') trying to set state to Powering down when already Down (vm:624) 2019-04-13 21:40:53,853+0200 INFO (jsonrpc/1) [virt.vm] (vmId='7b5e0fdd-8757-49ce-acf9-3206a372e1c9') Stopping connection (guestagent:455) 2019-04-13 21:40:53,853+0200 INFO (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='7bb7b150-d992-40fe-bbc9-e3fb66f21c10', volUUID=None) from=::ffff:192.168.0.19,51130, task_id=c9f3527b-ca3a-40c3-b111-ee3ece29aa87 (api:48) 2019-04-13 21:40:53,854+0200 INFO (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/7bb7b150-d992-40fe-bbc9-e3fb66f21c10' (fileSD:606) 2019-04-13 21:40:53,854+0200 INFO (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:192.168.0.19,51130, task_id=c9f3527b-ca3a-40c3-b111-ee3ece29aa87 (api:54) 2019-04-13 21:40:53,854+0200 INFO (jsonrpc/1) [virt.vm] (vmId='7b5e0fdd-8757-49ce-acf9-3206a372e1c9') Stopping connection (guestagent:455) 2019-04-13 21:40:53,855+0200 WARN (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/7b5e0fdd-8757-49ce-acf9-3206a372e1c9 (libvirtnetwork:198) 2019-04-13 21:40:53,855+0200 WARN (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/7b5e0fdd-8757-49ce-acf9-3206a372e1c9 (libvirtnetwork:205) 2019-04-13 21:40:53,857+0200 INFO (jsonrpc/1) [vdsm.api] START inappropriateDevices(thiefId='7b5e0fdd-8757-49ce-acf9-3206a372e1c9') from=::ffff:192.168.0.19,51130, task_id=69b96f59-ebe1-46b8-b626-915d87665cdb (api:48) 2019-04-13 21:40:53,858+0200 INFO (jsonrpc/1) [vdsm.api] FINISH inappropriateDevices return=None from=::ffff:192.168.0.19,51130, task_id=69b96f59-ebe1-46b8-b626-915d87665cdb (api:54) 2019-04-13 21:40:54,100+0200 INFO (jsonrpc/1) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/50_vhostmd: rc=0 err= (hooks:114) 2019-04-13 21:40:54,284+0200 INFO (jsonrpc/1) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/delete_vhostuserclient_hook: rc=0 err= (hooks:114) 2019-04-13 21:40:54,286+0200 INFO (jsonrpc/1) [api.virt] FINISH destroy return={'status': {'message': 'Machine destroyed', 'code': 0}} from=::ffff:192.168.0.19,51130, vmId=7b5e0fdd-8757-49ce-acf9-3206a372e1c9 (api:54) 2019-04-13 21:40:54,286+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.destroy succeeded in 0.43 seconds (__init__:312) 2019-04-13 21:40:54,763+0200 INFO (monitor/9c3f06c) [storage.SANLock] Acquiring host id for domain 9c3f06cf-7475-448e-819b-f4f52fa7d782 (id=1, async=True) (clusterlock:294) 2019-04-13 21:40:55,133+0200 INFO (jsonrpc/3) [api.host] START getCapabilities() from=::ffff:192.168.0.19,51130, flow_id=a2dbe5a (api:48) 2019-04-13 21:40:55,461+0200 INFO (jsonrpc/2) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=992606bd-d830-45e0-bb36-cfc80b0165b7 (api:48) 2019-04-13 21:40:55,467+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=992606bd-d830-45e0-bb36-cfc80b0165b7 (api:52) 2019-04-13 21:40:55,467+0200 ERROR (jsonrpc/2) [storage.TaskManager.Task] (Task='992606bd-d830-45e0-bb36-cfc80b0165b7') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:40:55,467+0200 INFO (jsonrpc/2) [storage.TaskManager.Task] (Task='992606bd-d830-45e0-bb36-cfc80b0165b7') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:40:55,467+0200 ERROR (jsonrpc/2) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:40:55,467+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:40:55,475+0200 INFO (jsonrpc/7) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=68d57cc3-4a48-466b-9dca-9c8e8c96e363 (api:48) 2019-04-13 21:40:55,475+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getAllTasksStatuses error=Secured object is not in safe state from=::ffff:192.168.0.19,51130, task_id=68d57cc3-4a48-466b-9dca-9c8e8c96e363 (api:52) 2019-04-13 21:40:55,475+0200 ERROR (jsonrpc/7) [storage.TaskManager.Task] (Task='68d57cc3-4a48-466b-9dca-9c8e8c96e363') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getAllTasksStatuses File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2195, in getAllTasksStatuses allTasksStatus = self._pool.getAllTasksStatuses() File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state 2019-04-13 21:40:55,475+0200 INFO (jsonrpc/7) [storage.TaskManager.Task] (Task='68d57cc3-4a48-466b-9dca-9c8e8c96e363') aborting: Task is aborted: u'Secured object is not in safe state' - code 100 (task:1181) 2019-04-13 21:40:55,475+0200 ERROR (jsonrpc/7) [storage.Dispatcher] FINISH getAllTasksStatuses error=Secured object is not in safe state (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error SecureError: Secured object is not in safe state 2019-04-13 21:40:55,475+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksStatuses failed (error 654) in 0.00 seconds (__init__:312) 2019-04-13 21:40:55,525+0200 INFO (jsonrpc/5) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,51130, task_id=44ab91f9-9418-42fb-a59c-124ac93ac32c (api:48) 2019-04-13 21:40:55,526+0200 INFO (jsonrpc/5) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 21:40:55,813+0200 INFO (jsonrpc/5) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:40:55,814+0200 INFO (jsonrpc/5) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 21:40:55,814+0200 INFO (jsonrpc/5) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 21:40:55,990+0200 WARN (jsonrpc/5) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 21:40:55,995+0200 INFO (jsonrpc/5) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 21:40:55,995+0200 INFO (jsonrpc/5) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 21:40:55,995+0200 INFO (jsonrpc/5) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 21:40:55,995+0200 INFO (jsonrpc/5) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 21:40:55,996+0200 INFO (jsonrpc/5) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 21:40:55,996+0200 INFO (jsonrpc/5) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,51130, task_id=44ab91f9-9418-42fb-a59c-124ac93ac32c (api:54) 2019-04-13 21:40:55,996+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.47 seconds (__init__:312) 2019-04-13 21:40:56,004+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=f686f4d6-6e16-41af-8cc4-8bb1dd5b7edb (api:48) 2019-04-13 21:40:56,009+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=f686f4d6-6e16-41af-8cc4-8bb1dd5b7edb (api:52) 2019-04-13 21:40:56,009+0200 ERROR (jsonrpc/4) [storage.TaskManager.Task] (Task='f686f4d6-6e16-41af-8cc4-8bb1dd5b7edb') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:40:56,009+0200 INFO (jsonrpc/4) [storage.TaskManager.Task] (Task='f686f4d6-6e16-41af-8cc4-8bb1dd5b7edb') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:40:56,009+0200 ERROR (jsonrpc/4) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:40:56,009+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:40:57,260+0200 INFO (jsonrpc/3) [root] managedvolume not supported: Managed Volume Not Supported. Missing package os-brick.: ('Cannot import os_brick',) (caps:149) 2019-04-13 21:40:57,893+0200 INFO (jsonrpc/3) [root] /usr/libexec/vdsm/hooks/after_get_caps/50_openstacknet: rc=0 err= (hooks:114) 2019-04-13 21:40:58,165+0200 INFO (jsonrpc/3) [root] /usr/libexec/vdsm/hooks/after_get_caps/openstacknet_utils.py: rc=0 err= (hooks:114) 2019-04-13 21:40:58,526+0200 INFO (jsonrpc/3) [root] /usr/libexec/vdsm/hooks/after_get_caps/ovirt_provider_ovn_hook: rc=0 err= (hooks:114) 2019-04-13 21:40:58,528+0200 INFO (jsonrpc/3) [api.host] FINISH getCapabilities return={'status': {'message': 'Done', 'code': 0}, 'info': {u'HBAInventory': {u'iSCSI': [{u'InitiatorName': u'iqn.1994-05.com.redhat:5ee45db2f0f'}], u'FC': [{u'wwpn': u'21000024ff2c1d40', u'wwnn': u'20000024ff2c1d40', u'model': u'QLE2562 - QLogic 8Gb FC Dual-port HBA for System x'}, {u'wwpn': u'21000024ff2c1d41', u'wwnn': u'20000024ff2c1d41', u'model': u'QLE2562 - QLogic 8Gb FC Dual-port HBA for System x'}]}, u'packages2': {u'kernel': {u'release': u'957.10.1.el7.x86_64', u'version': u'3.10.0'}, u'glusterfs-rdma': {u'release': u'2.el7', u'version': u'5.3'}, u'glusterfs-fuse': {u'release': u'2.el7', u'version': u'5.3'}, u'spice-server': {u'release': u'6.el7_6.1', u'version': u'0.14.0'}, u'librbd1': {u'release': u'4.el7', u'version': u'10.2.5'}, u'vdsm': {u'release': u'1.el7', u'version': u'4.30.11'}, u'qemu-kvm': {u'release': u'18.el7_6.3.1', u'version': u'2.12.0'}, u'openvswitch': {u'release': u'3.el7', u'version': u'2.10.1'}, u'libvirt': {u'release': u'10.el7_6.6', u'version': u'4.5.0'}, u'ovirt-hosted-engine-ha': {u'release': u'1.el7', u'version': u'2.3.1'}, u'qemu-img': {u'release': u'18.el7_6.3.1', u'version': u'2.12.0'}, u'mom': {u'release': u'1.el7.centos', u'version': u'0.5.12'}, u'glusterfs': {u'release': u'2.el7', u'version': u'5.3'}, u'glusterfs-cli': {u'release': u'2.el7', u'version': u'5.3'}, u'glusterfs-server': {u'release': u'2.el7', u'version': u'5.3'}, u'glusterfs-geo-replication': {u'release': u'2.el7', u'version': u'5.3'}}, u'numaNodeDistance': {u'1': [21, 10], u'0': [10, 21]}, u'cpuModel': u'Intel(R) Xeon(R) CPU X5660 @ 2.80GHz', u'nestedVirtualization': False, u'liveMerge': u'true', u'hooks': {u'after_vm_start': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'ea0a5a715da8c1badbcda28e8b8fa00e'}}, u'after_network_setup': {u'30_ethtool_options': {u'md5': u'ce1fbad7aa0389e3b06231219140bf0d'}}, u'before_vm_start': {u'50_hostedengine': {u'md5': u'95c810cdcfe4195302a59574a5148289'}, u'50_vhostmd': {u'md5': u'9206bc390bcbf208b06a8e899581be2d'}}, u'after_device_migrate_destination': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'6226fbc4d1602994828a3904fc1b875d'}}, u'before_device_migrate_destination': {u'50_vmfex': {u'md5': u'49caba1a5faadd8efacef966f79bc30a'}}, u'before_device_create': {u'50_vmfex': {u'md5': u'49caba1a5faadd8efacef966f79bc30a'}, u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'ec8d5d7ba063a109f749cd63c7a2b58d'}, u'20_ovirt_provider_ovn_vhostuser_hook': {u'md5': u'a8af653b7386c138b2e6e9738bd6b62c'}, u'10_ovirt_provider_ovn_hook': {u'md5': u'73822988042847bab1ea832a6b9fa837'}}, u'after_get_caps': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'5c3a9ab6e06e039bdd220c0216e45809'}, u'ovirt_provider_ovn_hook': {u'md5': u'4c4b1d2d5460e6a65114ae36cb775df6'}}, u'after_nic_hotplug': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'6226fbc4d1602994828a3904fc1b875d'}}, u'before_vm_migrate_destination': {u'50_vhostmd': {u'md5': u'9206bc390bcbf208b06a8e899581be2d'}}, u'after_device_create': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'6226fbc4d1602994828a3904fc1b875d'}}, u'before_vm_dehibernate': {u'50_vhostmd': {u'md5': u'9206bc390bcbf208b06a8e899581be2d'}}, u'before_nic_hotplug': {u'50_vmfex': {u'md5': u'49caba1a5faadd8efacef966f79bc30a'}, u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'ec8d5d7ba063a109f749cd63c7a2b58d'}, u'20_ovirt_provider_ovn_vhostuser_hook': {u'md5': u'a8af653b7386c138b2e6e9738bd6b62c'}, u'10_ovirt_provider_ovn_hook': {u'md5': u'73822988042847bab1ea832a6b9fa837'}}, u'before_network_setup': {u'50_fcoe': {u'md5': u'28c352339c8beef1e1b05c67d106d062'}}, u'after_vm_destroy': {u'delete_vhostuserclient_hook': {u'md5': u'c2f279cc9483a3f842f6c29df13994c1'}, u'50_vhostmd': {u'md5': u'bdf4802c0521cf1bae08f2b90a9559cf'}}}, u'supportsIPv6': True, u'realtimeKernel': False, u'vmTypes': [u'kvm'], u'liveSnapshot': u'true', u'cpuThreads': u'24', u'kdumpStatus': 1, u'networks': {u'ovirtmgmt': {u'iface': u'ovirtmgmt', u'ipv6autoconf': True, u'addr': u'192.168.0.35', u'dhcpv6': False, u'ipv6addrs': [], u'switch': u'legacy', u'bridged': True, u'southbound': u'bond0', u'dhcpv4': False, u'netmask': u'255.255.255.0', u'ipv4defaultroute': True, u'stp': u'off', u'ipv4addrs': [u'192.168.0.35/24'], u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u'192.168.0.1', u'ports': [u'bond0']}, u'streamingnet': {u'iface': u'streamingnet', u'ipv6autoconf': True, u'addr': u'', u'dhcpv6': False, u'ipv6addrs': [], u'switch': u'legacy', u'bridged': True, u'southbound': u'eno4', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'stp': u'off', u'ipv4addrs': [], u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u'', u'ports': [u'eno4']}}, u'kernelArgs': u'BOOT_IMAGE=/vmlinuz-3.10.0-957.10.1.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=de_DE.UTF-8 intel_iommu=on', u'domain_versions': [0, 2, 3, 4, 5], u'bridges': {u'ovirtmgmt': {u'ipv6autoconf': True, u'addr': u'192.168.0.35', u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'255.255.255.0', u'ipv4defaultroute': True, u'stp': u'off', u'ipv4addrs': [u'192.168.0.35/24'], u'ipv6gateway': u'::', u'gateway': u'192.168.0.1', u'opts': {u'multicast_last_member_count': u'2', u'vlan_protocol': u'0x8100', u'hash_elasticity': u'4', u'multicast_query_response_interval': u'1000', u'group_fwd_mask': u'0x0', u'multicast_snooping': u'1', u'multicast_startup_query_interval': u'3125', u'hello_timer': u'0', u'multicast_querier_interval': u'25500', u'max_age': u'2000', u'hash_max': u'512', u'stp_state': u'0', u'topology_change_detected': u'0', u'priority': u'32768', u'multicast_igmp_version': u'2', u'multicast_membership_interval': u'26000', u'root_path_cost': u'0', u'root_port': u'0', u'multicast_stats_enabled': u'0', u'multicast_startup_query_count': u'2', u'nf_call_iptables': u'0', u'vlan_stats_enabled': u'0', u'hello_time': u'200', u'topology_change': u'0', u'bridge_id': u'8000.e41f13b59416', u'topology_change_timer': u'0', u'ageing_time': u'30000', u'nf_call_ip6tables': u'0', u'multicast_mld_version': u'1', u'gc_timer': u'23132', u'root_id': u'8000.e41f13b59416', u'nf_call_arptables': u'0', u'group_addr': u'1:80:c2:0:0:0', u'multicast_last_member_interval': u'100', u'default_pvid': u'1', u'multicast_query_interval': u'12500', u'multicast_query_use_ifaddr': u'0', u'tcn_timer': u'0', u'multicast_router': u'1', u'vlan_filtering': u'0', u'multicast_querier': u'0', u'forward_delay': u'0'}, u'ports': [u'bond0']}, u'streamingnet': {u'ipv6autoconf': True, u'addr': u'', u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'stp': u'off', u'ipv4addrs': [], u'ipv6gateway': u'::', u'gateway': u'', u'opts': {u'multicast_last_member_count': u'2', u'vlan_protocol': u'0x8100', u'hash_elasticity': u'4', u'multicast_query_response_interval': u'1000', u'group_fwd_mask': u'0x0', u'multicast_snooping': u'1', u'multicast_startup_query_interval': u'3125', u'hello_timer': u'0', u'multicast_querier_interval': u'25500', u'max_age': u'2000', u'hash_max': u'512', u'stp_state': u'0', u'topology_change_detected': u'0', u'priority': u'32768', u'multicast_igmp_version': u'2', u'multicast_membership_interval': u'26000', u'root_path_cost': u'0', u'root_port': u'0', u'multicast_stats_enabled': u'0', u'multicast_startup_query_count': u'2', u'nf_call_iptables': u'0', u'vlan_stats_enabled': u'0', u'hello_time': u'200', u'topology_change': u'0', u'bridge_id': u'8000.5cf3fc3918de', u'topology_change_timer': u'0', u'ageing_time': u'30000', u'nf_call_ip6tables': u'0', u'multicast_mld_version': u'1', u'gc_timer': u'23643', u'root_id': u'8000.5cf3fc3918de', u'nf_call_arptables': u'0', u'group_addr': u'1:80:c2:0:0:0', u'multicast_last_member_interval': u'100', u'default_pvid': u'1', u'multicast_query_interval': u'12500', u'multicast_query_use_ifaddr': u'0', u'tcn_timer': u'0', u'multicast_router': u'1', u'vlan_filtering': u'0', u'multicast_querier': u'0', u'forward_delay': u'0'}, u'ports': [u'eno4']}}, u'uuid': u'2650319A-1C29-11E0-8699-E41F13B59414', u'onlineCpus': u'0,1,2,3,4,5,12,13,14,15,16,17,6,7,8,9,10,11,18,19,20,21,22,23', u'nameservers': [u'192.168.0.1'], u'nics': {u'eno1': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e4:1f:13:b5:94:14', u'ipv6gateway': u'::', u'gateway': u''}, u'eno2': {u'permhwaddr': u'e4:1f:13:b5:94:16', u'ipv6autoconf': False, u'addr': u'', u'speed': 1000, u'dhcpv6': False, u'ipv6addrs': [], u'ad_aggregator_id': u'1', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e4:1f:13:b5:94:16', u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u''}, u'eno3': {u'permhwaddr': u'5c:f3:fc:39:18:dc', u'ipv6autoconf': False, u'addr': u'', u'speed': 1000, u'dhcpv6': False, u'ipv6addrs': [], u'ad_aggregator_id': u'1', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e4:1f:13:b5:94:16', u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u''}, u'eno4': {u'ipv6autoconf': False, u'addr': u'', u'speed': 1000, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'5c:f3:fc:39:18:de', u'ipv6gateway': u'::', u'gateway': u''}, u'ens3f3': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'00:1b:21:8a:a6:cb', u'ipv6gateway': u'::', u'gateway': u''}, u'ens3f2': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'00:1b:21:8a:a6:ca', u'ipv6gateway': u'::', u'gateway': u''}, u'enp0s29f0u2': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e6:1f:13:a8:94:17', u'ipv6gateway': u'::', u'gateway': u''}, u'ens3f0': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'00:1b:21:8a:a6:c8', u'ipv6gateway': u'::', u'gateway': u''}, u'ens3f1': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'00:1b:21:8a:a6:c9', u'ipv6gateway': u'::', u'gateway': u''}}, u'software_revision': u'1', u'hostdevPassthrough': u'true', u'clusterLevels': [u'4.1', u'4.2', u'4.3'], u'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,eagerfpu,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,popcnt,aes,lahf_lm,epb,ssbd,ibrs,ibpb,stibp,tpr_shadow,vnmi,flexpriority,ept,vpid,dtherm,arat,spec_ctrl,intel_stibp,flush_l1d,model_Opteron_G2,model_kvm32,model_coreduo,model_Conroe,model_Nehalem,model_Westmere-IBRS,model_Opteron_G1,model_core2duo,model_Nehalem-IBRS,model_qemu32,model_Penryn,model_pentium2,model_pentium3,model_qemu64,model_Westmere,model_kvm64,model_pentium,model_486', u'kernelFeatures': {u'RETP': 1, u'IBRS': 0, u'PTI': 1}, u'ISCSIInitiatorName': u'iqn.1994-05.com.redhat:5ee45db2f0f', u'netConfigDirty': u'False', u'selinux': {u'mode': u'1'}, u'autoNumaBalancing': 1, u'reservedMem': u'321', u'bondings': {u'bond0': {u'ipv6autoconf': False, u'ad_partner_mac': u'ec:b1:d7:11:34:40', u'speed': 2000, u'dhcpv6': False, u'ipv6addrs': [], u'netmask': u'', u'active_slave': u'', u'ad_aggregator_id': u'1', u'dhcpv4': False, u'switch': u'legacy', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e4:1f:13:b5:94:16', u'slaves': [u'eno2', u'eno3'], u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u'', u'opts': {u'mode': u'4', u'xmit_hash_policy': u'2'}, u'addr': u''}}, u'software_version': u'4.30.11', u'supportedENGINEs': [u'4.1', u'4.2', u'4.3'], u'vncEncrypted': True, u'backupEnabled': False, u'cpuSpeed': u'2793.000', u'numaNodes': {u'1': {u'totalMemory': u'98304', u'cpus': [6, 7, 8, 9, 10, 11, 18, 19, 20, 21, 22, 23]}, u'0': {u'totalMemory': u'98291', u'cpus': [0, 1, 2, 3, 4, 5, 12, 13, 14, 15, 16, 17]}}, u'cpuSockets': u'2', u'vlans': {}, u'version_name': u'Snow Man', 'lastClientIface': 'ovirtmgmt', u'cpuCores': u'12', u'hostedEngineDeployed': False, u'hugepages': [1048576, 2048], u'guestOverhead': u'65', u'additionalFeatures': [u'libgfapi_supported', u'GLUSTER_SNAPSHOT', u'GLUSTER_GEO_REPLICATION', u'GLUSTER_BRICK_MANAGEMENT'], u'kvmEnabled': u'true', u'memSize': u'193330', u'emulatedMachines': [u'pc-i440fx-rhel7.1.0', u'pc-q35-rhel7.3.0', u'rhel6.3.0', u'pc-i440fx-rhel7.5.0', u'pc-i440fx-rhel7.0.0', u'rhel6.1.0', u'pc-q35-rhel7.6.0', u'pc-i440fx-rhel7.4.0', u'rhel6.6.0', u'pc-q35-rhel7.5.0', u'rhel6.2.0', u'pc', u'pc-i440fx-rhel7.3.0', u'q35', u'pc-i440fx-rhel7.2.0', u'rhel6.4.0', u'pc-q35-rhel7.4.0', u'pc-i440fx-rhel7.6.0', u'rhel6.0.0', u'rhel6.5.0'], u'rngSources': [u'hwrng', u'random'], u'operatingSystem': {u'release': u'6.1810.2.el7.centos', u'pretty_name': u'CentOS Linux 7 (Core)', u'version': u'7', u'name': u'RHEL'}}} from=::ffff:192.168.0.19,51130, flow_id=a2dbe5a (api:54) 2019-04-13 21:40:58,539+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getCapabilities succeeded in 3.41 seconds (__init__:312) 2019-04-13 21:40:58,625+0200 INFO (jsonrpc/6) [api.host] START getHardwareInfo() from=::ffff:192.168.0.19,51130, flow_id=a2dbe5a (api:48) 2019-04-13 21:40:58,626+0200 INFO (jsonrpc/6) [api.host] FINISH getHardwareInfo return={'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'System x3650 M3 -[7945L2G]-', 'systemSerialNumber': '99L8940', 'systemFamily': 'System x', 'systemVersion': '00', 'systemUUID': '2650319a-1c29-11e0-8699-e41f13b59414', 'systemManufacturer': 'IBM'}} from=::ffff:192.168.0.19,51130, flow_id=a2dbe5a (api:54) 2019-04-13 21:40:58,626+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getHardwareInfo succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:40:58,822+0200 INFO (jsonrpc/0) [api.host] START hostdevListByCaps(caps=None) from=::ffff:192.168.0.19,51130, flow_id=a2dbe5a (api:48) 2019-04-13 21:40:58,929+0200 INFO (jsonrpc/0) [api.host] FINISH hostdevListByCaps return={'status': {'message': 'Done', 'code': 0}, 'deviceList': {'net_eno1_e4_1f_13_b5_94_14': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'eno1', 'parent': 'pci_0000_0b_00_0'}}, 'pci_0000_00_1d_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB UHCI Controller #3', 'vendor': 'Intel Corporation', 'product_id': '0x3a36', 'parent': 'computer', 'address': {'slot': '29', 'bus': '0', 'domain': '0', 'function': '2'}, 'iommu_group': '14', 'vendor_id': '0x8086', 'driver': 'uhci_hcd'}}, 'pci_0000_00_1d_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB UHCI Controller #2', 'vendor': 'Intel Corporation', 'product_id': '0x3a35', 'parent': 'computer', 'address': {'slot': '29', 'bus': '0', 'domain': '0', 'function': '1'}, 'iommu_group': '14', 'vendor_id': '0x8086', 'driver': 'uhci_hcd'}}, 'pci_0000_00_1d_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB UHCI Controller #1', 'vendor': 'Intel Corporation', 'product_id': '0x3a34', 'parent': 'computer', 'address': {'slot': '29', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '14', 'vendor_id': '0x8086', 'driver': 'uhci_hcd'}}, 'pci_0000_00_1d_7': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB2 EHCI Controller #1', 'vendor': 'Intel Corporation', 'product_id': '0x3a3a', 'parent': 'computer', 'address': {'slot': '29', 'bus': '0', 'domain': '0', 'function': '7'}, 'iommu_group': '14', 'vendor_id': '0x8086', 'driver': 'ehci-pci'}}, 'scsi_0_2_1_0': {'params': {'product': 'ServeRAID M5015', 'vendor': 'IBM', 'parent': 'scsi_target0_2_1', 'driver': 'sd', 'capability': 'scsi', 'is_assignable': 'true', 'udev_path': '/dev/sg1', 'address': {'bus': '2', 'host': '0', 'target': '1', 'lun': '0'}}}, 'pci_0000_00_15_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 Trusted Execution Technology Registers', 'vendor': 'Intel Corporation', 'product_id': '0x342f', 'parent': 'computer', 'address': {'slot': '21', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '10', 'vendor_id': '0x8086'}}, 'usb_2_5_1_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'usbhid', 'parent': 'usb_2_5_1'}}, 'block_sda_3600605b002f69270217fb60a16a05ccf': {'params': {'capability': 'storage', 'is_assignable': 'true', 'product': 'ServeRAID M5015', 'vendor': 'IBM', 'parent': 'scsi_0_2_0_0'}}, 'pci_0000_10_00_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'NetXtreme II BCM5709 Gigabit Ethernet', 'vendor': 'Broadcom Inc. and subsidiaries', 'product_id': '0x1639', 'parent': 'pci_0000_00_02_0', 'address': {'slot': '0', 'bus': '16', 'domain': '0', 'function': '1'}, 'iommu_group': '18', 'vendor_id': '0x14e4', 'driver': 'bnx2'}}, 'usb_usb2': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '2.0 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0002', 'parent': 'pci_0000_00_1d_7', 'address': {'device': '1', 'bus': '2'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'pci_0000_fe_06_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control', 'vendor': 'Intel Corporation', 'product_id': '0x2db3', 'parent': 'computer', 'address': {'slot': '6', 'bus': '254', 'domain': '0', 'function': '3'}, 'iommu_group': '29', 'vendor_id': '0x8086'}}, 'pci_0000_fe_06_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Rank', 'vendor': 'Intel Corporation', 'product_id': '0x2db2', 'parent': 'computer', 'address': {'slot': '6', 'bus': '254', 'domain': '0', 'function': '2'}, 'iommu_group': '29', 'vendor_id': '0x8086'}}, 'pci_0000_fe_06_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Address', 'vendor': 'Intel Corporation', 'product_id': '0x2db1', 'parent': 'computer', 'address': {'slot': '6', 'bus': '254', 'domain': '0', 'function': '1'}, 'iommu_group': '29', 'vendor_id': '0x8086'}}, 'pci_0000_00_11_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500 Routing & Protocol Layer Register Port 1', 'vendor': 'Intel Corporation', 'product_id': '0x3428', 'parent': 'computer', 'address': {'slot': '17', 'bus': '0', 'domain': '0', 'function': '1'}, 'iommu_group': '8', 'vendor_id': '0x8086'}}, 'usb_2_6_4': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '4-Port USB 2.0 Hub', 'vendor': 'Realtek Semiconductor Corp.', 'product_id': '0x5411', 'parent': 'usb_2_6', 'address': {'device': '6', 'bus': '2'}, 'vendor_id': '0x0bda', 'driver': 'usb'}}, 'net_ens3f1_00_1b_21_8a_a6_c9': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'ens3f1', 'parent': 'pci_0000_24_00_1'}}, 'net_enp0s29f0u2_e6_1f_13_a8_94_17': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'enp0s29f0u2', 'parent': 'usb_5_2_1_0'}}, 'pci_0000_00_1f_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JIB (ICH10) LPC Interface Controller', 'vendor': 'Intel Corporation', 'product_id': '0x3a18', 'parent': 'computer', 'address': {'slot': '31', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '16', 'vendor_id': '0x8086', 'driver': 'lpc_ich'}}, 'pci_0000_00_1f_5': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) 2 port SATA IDE Controller #2', 'vendor': 'Intel Corporation', 'product_id': '0x3a26', 'parent': 'computer', 'address': {'slot': '31', 'bus': '0', 'domain': '0', 'function': '5'}, 'iommu_group': '16', 'vendor_id': '0x8086', 'driver': 'ata_piix'}}, 'pci_0000_24_00_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82580 Gigabit Network Connection', 'vendor': 'Intel Corporation', 'product_id': '0x150e', 'parent': 'pci_0000_00_09_0', 'address': {'slot': '0', 'bus': '36', 'domain': '0', 'function': '2'}, 'iommu_group': '22', 'vendor_id': '0x8086', 'driver': 'igb'}}, 'pci_0000_24_00_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82580 Gigabit Network Connection', 'vendor': 'Intel Corporation', 'product_id': '0x150e', 'parent': 'pci_0000_00_09_0', 'address': {'slot': '0', 'bus': '36', 'domain': '0', 'function': '3'}, 'iommu_group': '23', 'vendor_id': '0x8086', 'driver': 'igb'}}, 'pci_0000_00_1f_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) SMBus Controller', 'vendor': 'Intel Corporation', 'product_id': '0x3a30', 'parent': 'computer', 'address': {'slot': '31', 'bus': '0', 'domain': '0', 'function': '3'}, 'iommu_group': '16', 'vendor_id': '0x8086', 'driver': 'i801_smbus'}}, 'pci_0000_00_1f_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) 4 port SATA IDE Controller #1', 'vendor': 'Intel Corporation', 'product_id': '0x3a20', 'parent': 'computer', 'address': {'slot': '31', 'bus': '0', 'domain': '0', 'function': '2'}, 'iommu_group': '16', 'vendor_id': '0x8086', 'driver': 'ata_piix'}}, 'pci_0000_24_00_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82580 Gigabit Network Connection', 'vendor': 'Intel Corporation', 'product_id': '0x150e', 'parent': 'pci_0000_00_09_0', 'address': {'slot': '0', 'bus': '36', 'domain': '0', 'function': '1'}, 'iommu_group': '21', 'vendor_id': '0x8086', 'driver': 'igb'}}, 'drm_card0': {'params': {'capability': 'drm', 'is_assignable': 'true', 'parent': 'pci_0000_07_00_0'}}, 'pci_0000_00_1e_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '82801 PCI Bridge', 'vendor': 'Intel Corporation', 'product_id': '0x244e', 'parent': 'computer', 'address': {'slot': '30', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '15', 'vendor_id': '0x8086'}}, 'computer': {'params': {'capability': 'system', 'is_assignable': 'true', 'product': 'System x3650 M3 -[7945L2G]-'}}, 'pci_0000_07_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'MGA G200EV', 'vendor': 'Matrox Electronics Systems Ltd.', 'product_id': '0x0530', 'parent': 'pci_0000_06_00_0', 'address': {'slot': '0', 'bus': '7', 'domain': '0', 'function': '0'}, 'iommu_group': '13', 'vendor_id': '0x102b', 'driver': 'mgag200'}}, 'pci_0000_00_1a_7': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB2 EHCI Controller #2', 'vendor': 'Intel Corporation', 'product_id': '0x3a3c', 'parent': 'computer', 'address': {'slot': '26', 'bus': '0', 'domain': '0', 'function': '7'}, 'iommu_group': '12', 'vendor_id': '0x8086', 'driver': 'ehci-pci'}}, 'pci_0000_00_1a_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB UHCI Controller #4', 'vendor': 'Intel Corporation', 'product_id': '0x3a37', 'parent': 'computer', 'address': {'slot': '26', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '12', 'vendor_id': '0x8086', 'driver': 'uhci_hcd'}}, 'pci_0000_00_1a_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB UHCI Controller #5', 'vendor': 'Intel Corporation', 'product_id': '0x3a38', 'parent': 'computer', 'address': {'slot': '26', 'bus': '0', 'domain': '0', 'function': '1'}, 'iommu_group': '12', 'vendor_id': '0x8086', 'driver': 'uhci_hcd'}}, 'pci_0000_ff_05_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Address', 'vendor': 'Intel Corporation', 'product_id': '0x2da9', 'parent': 'computer', 'address': {'slot': '5', 'bus': '255', 'domain': '0', 'function': '1'}, 'iommu_group': '34', 'vendor_id': '0x8086'}}, 'pci_0000_ff_05_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Control', 'vendor': 'Intel Corporation', 'product_id': '0x2da8', 'parent': 'computer', 'address': {'slot': '5', 'bus': '255', 'domain': '0', 'function': '0'}, 'iommu_group': '34', 'vendor_id': '0x8086'}}, 'pci_0000_ff_05_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control', 'vendor': 'Intel Corporation', 'product_id': '0x2dab', 'parent': 'computer', 'address': {'slot': '5', 'bus': '255', 'domain': '0', 'function': '3'}, 'iommu_group': '34', 'vendor_id': '0x8086'}}, 'pci_0000_ff_05_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Rank', 'vendor': 'Intel Corporation', 'product_id': '0x2daa', 'parent': 'computer', 'address': {'slot': '5', 'bus': '255', 'domain': '0', 'function': '2'}, 'iommu_group': '34', 'vendor_id': '0x8086'}}, 'pci_0000_fe_04_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Address', 'vendor': 'Intel Corporation', 'product_id': '0x2da1', 'parent': 'computer', 'address': {'slot': '4', 'bus': '254', 'domain': '0', 'function': '1'}, 'iommu_group': '27', 'vendor_id': '0x8086'}}, 'usb_3_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb3'}}, 'net_eno3_e4_1f_13_b5_94_16': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'eno3', 'parent': 'pci_0000_10_00_0'}}, 'pci_0000_00_1c_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '82801JI (ICH10 Family) PCI Express Root Port 1', 'vendor': 'Intel Corporation', 'product_id': '0x3a40', 'parent': 'computer', 'address': {'slot': '28', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '13', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'pci_0000_00_1c_4': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '82801JI (ICH10 Family) PCI Express Root Port 5', 'vendor': 'Intel Corporation', 'product_id': '0x3a48', 'parent': 'computer', 'address': {'slot': '28', 'bus': '0', 'domain': '0', 'function': '4'}, 'iommu_group': '13', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'usb_6_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb6'}}, 'pci_0000_ff_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QuickPath Architecture Generic Non-core Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2c70', 'parent': 'computer', 'address': {'slot': '0', 'bus': '255', 'domain': '0', 'function': '0'}, 'iommu_group': '30', 'vendor_id': '0x8086'}}, 'pci_0000_ff_00_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QuickPath Architecture System Address Decoder', 'vendor': 'Intel Corporation', 'product_id': '0x2d81', 'parent': 'computer', 'address': {'slot': '0', 'bus': '255', 'domain': '0', 'function': '1'}, 'iommu_group': '30', 'vendor_id': '0x8086'}}, 'net_eno4_5c_f3_fc_39_18_de': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'eno4', 'parent': 'pci_0000_10_00_1'}}, 'net_ens3f3_00_1b_21_8a_a6_cb': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'ens3f3', 'parent': 'pci_0000_24_00_3'}}, 'usb_4_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb4'}}, 'pci_0000_fe_04_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Control', 'vendor': 'Intel Corporation', 'product_id': '0x2da0', 'parent': 'computer', 'address': {'slot': '4', 'bus': '254', 'domain': '0', 'function': '0'}, 'iommu_group': '27', 'vendor_id': '0x8086'}}, 'pci_0000_fe_04_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control', 'vendor': 'Intel Corporation', 'product_id': '0x2da3', 'parent': 'computer', 'address': {'slot': '4', 'bus': '254', 'domain': '0', 'function': '3'}, 'iommu_group': '27', 'vendor_id': '0x8086'}}, 'pci_0000_fe_04_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Rank', 'vendor': 'Intel Corporation', 'product_id': '0x2da2', 'parent': 'computer', 'address': {'slot': '4', 'bus': '254', 'domain': '0', 'function': '2'}, 'iommu_group': '27', 'vendor_id': '0x8086'}}, 'usb_2_5_3_1_1_1': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'usbhid', 'parent': 'usb_2_5_3_1'}}, 'usb_1_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb1'}}, 'usb_2_5_3_1_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'usbhid', 'parent': 'usb_2_5_3_1'}}, 'pci_0000_ff_06_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Rank', 'vendor': 'Intel Corporation', 'product_id': '0x2db2', 'parent': 'computer', 'address': {'slot': '6', 'bus': '255', 'domain': '0', 'function': '2'}, 'iommu_group': '35', 'vendor_id': '0x8086'}}, 'pci_0000_ff_06_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control', 'vendor': 'Intel Corporation', 'product_id': '0x2db3', 'parent': 'computer', 'address': {'slot': '6', 'bus': '255', 'domain': '0', 'function': '3'}, 'iommu_group': '35', 'vendor_id': '0x8086'}}, 'pci_0000_ff_06_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Control', 'vendor': 'Intel Corporation', 'product_id': '0x2db0', 'parent': 'computer', 'address': {'slot': '6', 'bus': '255', 'domain': '0', 'function': '0'}, 'iommu_group': '35', 'vendor_id': '0x8086'}}, 'pci_0000_ff_06_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Address', 'vendor': 'Intel Corporation', 'product_id': '0x2db1', 'parent': 'computer', 'address': {'slot': '6', 'bus': '255', 'domain': '0', 'function': '1'}, 'iommu_group': '35', 'vendor_id': '0x8086'}}, 'pci_0000_fe_02_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Mirror Port Link 1', 'vendor': 'Intel Corporation', 'product_id': '0x2d93', 'parent': 'computer', 'address': {'slot': '2', 'bus': '254', 'domain': '0', 'function': '3'}, 'iommu_group': '25', 'vendor_id': '0x8086'}}, 'pci_0000_fe_02_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Mirror Port Link 0', 'vendor': 'Intel Corporation', 'product_id': '0x2d92', 'parent': 'computer', 'address': {'slot': '2', 'bus': '254', 'domain': '0', 'function': '2'}, 'iommu_group': '25', 'vendor_id': '0x8086'}}, 'pci_0000_fe_02_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Physical 0', 'vendor': 'Intel Corporation', 'product_id': '0x2d91', 'parent': 'computer', 'address': {'slot': '2', 'bus': '254', 'domain': '0', 'function': '1'}, 'iommu_group': '25', 'vendor_id': '0x8086'}}, 'pci_0000_fe_02_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Link 0', 'vendor': 'Intel Corporation', 'product_id': '0x2d90', 'parent': 'computer', 'address': {'slot': '2', 'bus': '254', 'domain': '0', 'function': '0'}, 'iommu_group': '25', 'vendor_id': '0x8086'}}, 'pci_0000_fe_02_5': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Physical 1', 'vendor': 'Intel Corporation', 'product_id': '0x2d95', 'parent': 'computer', 'address': {'slot': '2', 'bus': '254', 'domain': '0', 'function': '5'}, 'iommu_group': '25', 'vendor_id': '0x8086'}}, 'net_ens3f2_00_1b_21_8a_a6_ca': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'ens3f2', 'parent': 'pci_0000_24_00_2'}}, 'scsi_0_2_0_0': {'params': {'product': 'ServeRAID M5015', 'vendor': 'IBM', 'parent': 'scsi_target0_2_0', 'driver': 'sd', 'capability': 'scsi', 'is_assignable': 'true', 'udev_path': '/dev/sg0', 'address': {'bus': '2', 'host': '0', 'target': '0', 'lun': '0'}}}, 'usb_5_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb5'}}, 'pci_0000_ff_02_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Mirror Port Link 0', 'vendor': 'Intel Corporation', 'product_id': '0x2d92', 'parent': 'computer', 'address': {'slot': '2', 'bus': '255', 'domain': '0', 'function': '2'}, 'iommu_group': '31', 'vendor_id': '0x8086'}}, 'pci_0000_00_01_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '5520/5500/X58 I/O Hub PCI Express Root Port 1', 'vendor': 'Intel Corporation', 'product_id': '0x3408', 'parent': 'computer', 'address': {'slot': '1', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '1', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'pci_0000_ff_02_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Mirror Port Link 1', 'vendor': 'Intel Corporation', 'product_id': '0x2d93', 'parent': 'computer', 'address': {'slot': '2', 'bus': '255', 'domain': '0', 'function': '3'}, 'iommu_group': '31', 'vendor_id': '0x8086'}}, 'pci_0000_0b_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'NetXtreme II BCM5709 Gigabit Ethernet', 'vendor': 'Broadcom Inc. and subsidiaries', 'product_id': '0x1639', 'parent': 'pci_0000_00_01_0', 'address': {'slot': '0', 'bus': '11', 'domain': '0', 'function': '0'}, 'iommu_group': '17', 'vendor_id': '0x14e4', 'driver': 'bnx2'}}, 'usb_7_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb7'}}, 'usb_2_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb2'}}, 'pci_0000_0b_00_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'NetXtreme II BCM5709 Gigabit Ethernet', 'vendor': 'Broadcom Inc. and subsidiaries', 'product_id': '0x1639', 'parent': 'pci_0000_00_01_0', 'address': {'slot': '0', 'bus': '11', 'domain': '0', 'function': '1'}, 'iommu_group': '17', 'vendor_id': '0x14e4', 'driver': 'bnx2'}}, 'pci_0000_00_14_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 I/O Hub Throttle Registers', 'vendor': 'Intel Corporation', 'product_id': '0x3438', 'parent': 'computer', 'address': {'slot': '20', 'bus': '0', 'domain': '0', 'function': '3'}, 'iommu_group': '9', 'vendor_id': '0x8086'}}, 'pci_0000_00_14_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 I/O Hub Control Status and RAS Registers', 'vendor': 'Intel Corporation', 'product_id': '0x3423', 'parent': 'computer', 'address': {'slot': '20', 'bus': '0', 'domain': '0', 'function': '2'}, 'iommu_group': '9', 'vendor_id': '0x8086'}}, 'pci_0000_00_14_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers', 'vendor': 'Intel Corporation', 'product_id': '0x3422', 'parent': 'computer', 'address': {'slot': '20', 'bus': '0', 'domain': '0', 'function': '1'}, 'iommu_group': '9', 'vendor_id': '0x8086'}}, 'pci_0000_00_14_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 I/O Hub System Management Registers', 'vendor': 'Intel Corporation', 'product_id': '0x342e', 'parent': 'computer', 'address': {'slot': '20', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '9', 'vendor_id': '0x8086', 'driver': 'i7core_edac'}}, 'pci_0000_ff_04_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Control', 'vendor': 'Intel Corporation', 'product_id': '0x2da0', 'parent': 'computer', 'address': {'slot': '4', 'bus': '255', 'domain': '0', 'function': '0'}, 'iommu_group': '33', 'vendor_id': '0x8086'}}, 'pci_0000_ff_04_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Address', 'vendor': 'Intel Corporation', 'product_id': '0x2da1', 'parent': 'computer', 'address': {'slot': '4', 'bus': '255', 'domain': '0', 'function': '1'}, 'iommu_group': '33', 'vendor_id': '0x8086'}}, 'pci_0000_ff_04_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Rank', 'vendor': 'Intel Corporation', 'product_id': '0x2da2', 'parent': 'computer', 'address': {'slot': '4', 'bus': '255', 'domain': '0', 'function': '2'}, 'iommu_group': '33', 'vendor_id': '0x8086'}}, 'pci_0000_ff_04_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control', 'vendor': 'Intel Corporation', 'product_id': '0x2da3', 'parent': 'computer', 'address': {'slot': '4', 'bus': '255', 'domain': '0', 'function': '3'}, 'iommu_group': '33', 'vendor_id': '0x8086'}}, 'pci_0000_00_05_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '5520/X58 I/O Hub PCI Express Root Port 5', 'vendor': 'Intel Corporation', 'product_id': '0x340c', 'parent': 'computer', 'address': {'slot': '5', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '4', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'pci_0000_ff_02_4': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Link 1', 'vendor': 'Intel Corporation', 'product_id': '0x2d94', 'parent': 'computer', 'address': {'slot': '2', 'bus': '255', 'domain': '0', 'function': '4'}, 'iommu_group': '31', 'vendor_id': '0x8086'}}, 'pci_0000_ff_02_5': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Physical 1', 'vendor': 'Intel Corporation', 'product_id': '0x2d95', 'parent': 'computer', 'address': {'slot': '2', 'bus': '255', 'domain': '0', 'function': '5'}, 'iommu_group': '31', 'vendor_id': '0x8086'}}, 'pci_0000_fe_00_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QuickPath Architecture System Address Decoder', 'vendor': 'Intel Corporation', 'product_id': '0x2d81', 'parent': 'computer', 'address': {'slot': '0', 'bus': '254', 'domain': '0', 'function': '1'}, 'iommu_group': '24', 'vendor_id': '0x8086'}}, 'pci_0000_fe_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QuickPath Architecture Generic Non-core Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2c70', 'parent': 'computer', 'address': {'slot': '0', 'bus': '254', 'domain': '0', 'function': '0'}, 'iommu_group': '24', 'vendor_id': '0x8086'}}, 'pci_0000_ff_02_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Link 0', 'vendor': 'Intel Corporation', 'product_id': '0x2d90', 'parent': 'computer', 'address': {'slot': '2', 'bus': '255', 'domain': '0', 'function': '0'}, 'iommu_group': '31', 'vendor_id': '0x8086'}}, 'pci_0000_ff_02_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Physical 0', 'vendor': 'Intel Corporation', 'product_id': '0x2d91', 'parent': 'computer', 'address': {'slot': '2', 'bus': '255', 'domain': '0', 'function': '1'}, 'iommu_group': '31', 'vendor_id': '0x8086'}}, 'pci_0000_00_16_5': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x342a', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '5'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_00_16_4': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x3429', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '4'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_00_16_7': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x342c', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '7'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_00_16_6': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x342b', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '6'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_00_16_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x3431', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '1'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_00_16_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x3430', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_00_16_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x3433', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '3'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_01_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'MegaRAID SAS 2108 [Liberator]', 'vendor': 'LSI Logic / Symbios Logic', 'product_id': '0x0079', 'parent': 'pci_0000_00_1c_0', 'address': {'slot': '0', 'bus': '1', 'domain': '0', 'function': '0'}, 'iommu_group': '13', 'vendor_id': '0x1000', 'driver': 'megaraid_sas'}}, 'scsi_target0_2_1': {'params': {'capability': 'scsi_target', 'is_assignable': 'true', 'parent': 'scsi_host0'}}, 'pci_0000_00_03_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '5520/5500/X58 I/O Hub PCI Express Root Port 3', 'vendor': 'Intel Corporation', 'product_id': '0x340a', 'parent': 'computer', 'address': {'slot': '3', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '3', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'pci_0000_fe_05_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Control', 'vendor': 'Intel Corporation', 'product_id': '0x2da8', 'parent': 'computer', 'address': {'slot': '5', 'bus': '254', 'domain': '0', 'function': '0'}, 'iommu_group': '28', 'vendor_id': '0x8086'}}, 'pci_0000_fe_05_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Address', 'vendor': 'Intel Corporation', 'product_id': '0x2da9', 'parent': 'computer', 'address': {'slot': '5', 'bus': '254', 'domain': '0', 'function': '1'}, 'iommu_group': '28', 'vendor_id': '0x8086'}}, 'pci_0000_fe_05_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Rank', 'vendor': 'Intel Corporation', 'product_id': '0x2daa', 'parent': 'computer', 'address': {'slot': '5', 'bus': '254', 'domain': '0', 'function': '2'}, 'iommu_group': '28', 'vendor_id': '0x8086'}}, 'pci_0000_fe_05_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control', 'vendor': 'Intel Corporation', 'product_id': '0x2dab', 'parent': 'computer', 'address': {'slot': '5', 'bus': '254', 'domain': '0', 'function': '3'}, 'iommu_group': '28', 'vendor_id': '0x8086'}}, 'pci_0000_00_11_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500 Physical and Link Layer Registers Port 1', 'vendor': 'Intel Corporation', 'product_id': '0x3427', 'parent': 'computer', 'address': {'slot': '17', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '8', 'vendor_id': '0x8086'}}, 'scsi_generic_sg1': {'params': {'capability': 'scsi_generic', 'is_assignable': 'true', 'udev_path': '/dev/sg1', 'parent': 'scsi_0_2_1_0'}}, 'scsi_generic_sg0': {'params': {'capability': 'scsi_generic', 'is_assignable': 'true', 'udev_path': '/dev/sg0', 'parent': 'scsi_0_2_0_0'}}, 'pci_0000_00_16_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x3432', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '2'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_fe_03_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller RAS Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2d9a', 'parent': 'computer', 'address': {'slot': '3', 'bus': '254', 'domain': '0', 'function': '2'}, 'iommu_group': '26', 'vendor_id': '0x8086'}}, 'pci_0000_fe_03_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2d98', 'parent': 'computer', 'address': {'slot': '3', 'bus': '254', 'domain': '0', 'function': '0'}, 'iommu_group': '26', 'vendor_id': '0x8086'}}, 'pci_0000_fe_03_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Target Address Decoder', 'vendor': 'Intel Corporation', 'product_id': '0x2d99', 'parent': 'computer', 'address': {'slot': '3', 'bus': '254', 'domain': '0', 'function': '1'}, 'iommu_group': '26', 'vendor_id': '0x8086'}}, 'pci_0000_fe_03_4': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Test Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2d9c', 'parent': 'computer', 'address': {'slot': '3', 'bus': '254', 'domain': '0', 'function': '4'}, 'iommu_group': '26', 'vendor_id': '0x8086'}}, 'usb_usb7': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '1.1 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0001', 'parent': 'pci_0000_00_1d_2', 'address': {'device': '1', 'bus': '7'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'usb_usb6': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '1.1 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0001', 'parent': 'pci_0000_00_1d_1', 'address': {'device': '1', 'bus': '6'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'usb_usb5': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '1.1 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0001', 'parent': 'pci_0000_00_1d_0', 'address': {'device': '1', 'bus': '5'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'usb_usb4': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '1.1 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0001', 'parent': 'pci_0000_00_1a_1', 'address': {'device': '1', 'bus': '4'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'usb_usb3': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '1.1 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0001', 'parent': 'pci_0000_00_1a_0', 'address': {'device': '1', 'bus': '3'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'pci_0000_10_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'NetXtreme II BCM5709 Gigabit Ethernet', 'vendor': 'Broadcom Inc. and subsidiaries', 'product_id': '0x1639', 'parent': 'pci_0000_00_02_0', 'address': {'slot': '0', 'bus': '16', 'domain': '0', 'function': '0'}, 'iommu_group': '18', 'vendor_id': '0x14e4', 'driver': 'bnx2'}}, 'usb_usb1': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '2.0 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0002', 'parent': 'pci_0000_00_1a_7', 'address': {'device': '1', 'bus': '1'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'usb_2_5_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_2_5'}}, 'usb_2_5_3_1': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'CS682 2-Port USB 2.0 DVI KVM Switch', 'vendor': 'ATEN International Co., Ltd', 'product_id': '0x2213', 'parent': 'usb_2_5_3', 'address': {'device': '9', 'bus': '2'}, 'vendor_id': '0x0557', 'driver': 'usb'}}, 'usb_2_6': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'Hub', 'vendor': 'Terminus Technology Inc.', 'product_id': '0x0101', 'parent': 'usb_usb2', 'address': {'device': '4', 'bus': '2'}, 'vendor_id': '0x1a40', 'driver': 'usb'}}, 'usb_2_5': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'Hub', 'vendor': 'Genesys Logic, Inc.', 'product_id': '0x0608', 'parent': 'usb_usb2', 'address': {'device': '3', 'bus': '2'}, 'vendor_id': '0x05e3', 'driver': 'usb'}}, 'usb_2_5_1': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'Uninterruptible Power Supply', 'vendor': 'American Power Conversion', 'product_id': '0x0002', 'parent': 'usb_2_5', 'address': {'device': '5', 'bus': '2'}, 'vendor_id': '0x051d', 'driver': 'usb'}}, 'pci_0000_fe_06_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Control', 'vendor': 'Intel Corporation', 'product_id': '0x2db0', 'parent': 'computer', 'address': {'slot': '6', 'bus': '254', 'domain': '0', 'function': '0'}, 'iommu_group': '29', 'vendor_id': '0x8086'}}, 'usb_2_5_3': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'Hub', 'vendor': 'ATEN International Co., Ltd', 'product_id': '0x8021', 'parent': 'usb_2_5', 'address': {'device': '8', 'bus': '2'}, 'vendor_id': '0x0557', 'driver': 'usb'}}, 'usb_2_5_2': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'HASP copy protection dongle', 'vendor': 'Aladdin Knowledge Systems', 'product_id': '0x0001', 'parent': 'usb_2_5', 'address': {'device': '7', 'bus': '2'}, 'vendor_id': '0x0529', 'driver': 'usb'}}, 'pci_0000_00_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520 I/O Hub to ESI Port', 'vendor': 'Intel Corporation', 'product_id': '0x3406', 'parent': 'computer', 'address': {'slot': '0', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '0', 'vendor_id': '0x8086'}}, 'pci_0000_00_09_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '7500/5520/5500/X58 I/O Hub PCI Express Root Port 9', 'vendor': 'Intel Corporation', 'product_id': '0x3410', 'parent': 'computer', 'address': {'slot': '9', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '6', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'pci_0000_1f_00_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'ISP2532-based 8Gb Fibre Channel to PCI Express HBA', 'vendor': 'QLogic Corp.', 'product_id': '0x2532', 'parent': 'pci_0000_00_07_0', 'address': {'slot': '0', 'bus': '31', 'domain': '0', 'function': '1'}, 'iommu_group': '19', 'vendor_id': '0x1077', 'driver': 'qla2xxx'}}, 'pci_0000_1f_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'ISP2532-based 8Gb Fibre Channel to PCI Express HBA', 'vendor': 'QLogic Corp.', 'product_id': '0x2532', 'parent': 'pci_0000_00_07_0', 'address': {'slot': '0', 'bus': '31', 'domain': '0', 'function': '0'}, 'iommu_group': '19', 'vendor_id': '0x1077', 'driver': 'qla2xxx'}}, 'pci_0000_24_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82580 Gigabit Network Connection', 'vendor': 'Intel Corporation', 'product_id': '0x150e', 'parent': 'pci_0000_00_09_0', 'address': {'slot': '0', 'bus': '36', 'domain': '0', 'function': '0'}, 'iommu_group': '20', 'vendor_id': '0x8086', 'driver': 'igb'}}, 'usb_2_5_2_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'parent': 'usb_2_5_2'}}, 'pci_0000_00_07_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '5520/5500/X58 I/O Hub PCI Express Root Port 7', 'vendor': 'Intel Corporation', 'product_id': '0x340e', 'parent': 'computer', 'address': {'slot': '7', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '5', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'net_eno2_e4_1f_13_b5_94_16': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'eno2', 'parent': 'pci_0000_0b_00_1'}}, 'usb_2_5_3_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_2_5_3'}}, 'usb_5_2_1_1': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'cdc_ether', 'parent': 'usb_5_2'}}, 'usb_5_2_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'cdc_ether', 'parent': 'usb_5_2'}}, 'pci_0000_fe_02_4': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Link 1', 'vendor': 'Intel Corporation', 'product_id': '0x2d94', 'parent': 'computer', 'address': {'slot': '2', 'bus': '254', 'domain': '0', 'function': '4'}, 'iommu_group': '25', 'vendor_id': '0x8086'}}, 'usb_2_6_4_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_2_6_4'}}, 'pci_0000_00_02_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '5520/5500/X58 I/O Hub PCI Express Root Port 2', 'vendor': 'Intel Corporation', 'product_id': '0x3409', 'parent': 'computer', 'address': {'slot': '2', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '2', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'pci_0000_ff_03_4': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Test Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2d9c', 'parent': 'computer', 'address': {'slot': '3', 'bus': '255', 'domain': '0', 'function': '4'}, 'iommu_group': '32', 'vendor_id': '0x8086'}}, 'pci_0000_ff_03_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller RAS Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2d9a', 'parent': 'computer', 'address': {'slot': '3', 'bus': '255', 'domain': '0', 'function': '2'}, 'iommu_group': '32', 'vendor_id': '0x8086'}}, 'pci_0000_ff_03_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Target Address Decoder', 'vendor': 'Intel Corporation', 'product_id': '0x2d99', 'parent': 'computer', 'address': {'slot': '3', 'bus': '255', 'domain': '0', 'function': '1'}, 'iommu_group': '32', 'vendor_id': '0x8086'}}, 'pci_0000_ff_03_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2d98', 'parent': 'computer', 'address': {'slot': '3', 'bus': '255', 'domain': '0', 'function': '0'}, 'iommu_group': '32', 'vendor_id': '0x8086'}}, 'pci_0000_00_10_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 Routing and Protocol Layer Registers Port 0', 'vendor': 'Intel Corporation', 'product_id': '0x3426', 'parent': 'computer', 'address': {'slot': '16', 'bus': '0', 'domain': '0', 'function': '1'}, 'iommu_group': '7', 'vendor_id': '0x8086'}}, 'pci_0000_00_10_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 Physical and Link Layer Registers Port 0', 'vendor': 'Intel Corporation', 'product_id': '0x3425', 'parent': 'computer', 'address': {'slot': '16', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '7', 'vendor_id': '0x8086'}}, 'pci_0000_06_00_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': 'VSC452 [SuperBMC]', 'vendor': 'Vitesse Semiconductor', 'product_id': '0x0452', 'parent': 'pci_0000_00_1c_4', 'address': {'slot': '0', 'bus': '6', 'domain': '0', 'function': '0'}, 'iommu_group': '13', 'vendor_id': '0x101b'}}, 'scsi_host3': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_00_1f_5'}}, 'scsi_host2': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_00_1f_2'}}, 'scsi_host1': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_00_1f_2'}}, 'scsi_host0': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_01_00_0'}}, 'scsi_host6': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_1f_00_1'}}, 'scsi_host5': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_1f_00_0'}}, 'scsi_host4': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_00_1f_5'}}, 'usb_2_6_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_2_6'}}, 'scsi_target0_2_0': {'params': {'capability': 'scsi_target', 'is_assignable': 'true', 'parent': 'scsi_host0'}}, 'net_lo_00_00_00_00_00_00': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'lo', 'parent': 'computer'}}, 'net_ens3f0_00_1b_21_8a_a6_c8': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'ens3f0', 'parent': 'pci_0000_24_00_0'}}, 'usb_5_2': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'RNDIS/CDC ETHER', 'vendor': 'IBM Corp.', 'product_id': '0x4010', 'parent': 'usb_usb5', 'address': {'device': '2', 'bus': '5'}, 'vendor_id': '0x04b3', 'driver': 'usb'}}, 'block_sdb_3600605b002f6927021946993801eb9a6': {'params': {'capability': 'storage', 'is_assignable': 'true', 'product': 'ServeRAID M5015', 'vendor': 'IBM', 'parent': 'scsi_0_2_1_0'}}}} from=::ffff:192.168.0.19,51130, flow_id=a2dbe5a (api:54) 2019-04-13 21:40:58,939+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.hostdevListByCaps succeeded in 0.12 seconds (__init__:312) 2019-04-13 21:41:03,015+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:41:03,016+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:41:03,016+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:41:04,770+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:04,805+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:04,806+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:05,484+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=2b24aa9d-bcef-45ec-b7b3-a35c4ecd2669 (api:48) 2019-04-13 21:41:05,484+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000406063', 'lastCheck': '0.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000706039', 'lastCheck': '0.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000498919', 'lastCheck': '0.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000864768', 'lastCheck': '0.8', 'valid': True}} from=internal, task_id=2b24aa9d-bcef-45ec-b7b3-a35c4ecd2669 (api:54) 2019-04-13 21:41:06,039+0200 INFO (jsonrpc/2) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=31dd684c-9ada-4e78-a31f-a56561e8ae5a (api:48) 2019-04-13 21:41:06,045+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=31dd684c-9ada-4e78-a31f-a56561e8ae5a (api:52) 2019-04-13 21:41:06,045+0200 ERROR (jsonrpc/2) [storage.TaskManager.Task] (Task='31dd684c-9ada-4e78-a31f-a56561e8ae5a') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:06,045+0200 INFO (jsonrpc/2) [storage.TaskManager.Task] (Task='31dd684c-9ada-4e78-a31f-a56561e8ae5a') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:41:06,045+0200 ERROR (jsonrpc/2) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:06,045+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.00 seconds (__init__:312) 2019-04-13 21:41:06,050+0200 INFO (jsonrpc/7) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=4378f23c-6273-4e2f-9edc-4da6d1319d8b (api:48) 2019-04-13 21:41:06,051+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getAllTasksStatuses error=Secured object is not in safe state from=::ffff:192.168.0.19,51130, task_id=4378f23c-6273-4e2f-9edc-4da6d1319d8b (api:52) 2019-04-13 21:41:06,051+0200 ERROR (jsonrpc/7) [storage.TaskManager.Task] (Task='4378f23c-6273-4e2f-9edc-4da6d1319d8b') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getAllTasksStatuses File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2195, in getAllTasksStatuses allTasksStatus = self._pool.getAllTasksStatuses() File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state 2019-04-13 21:41:06,051+0200 INFO (jsonrpc/7) [storage.TaskManager.Task] (Task='4378f23c-6273-4e2f-9edc-4da6d1319d8b') aborting: Task is aborted: u'Secured object is not in safe state' - code 100 (task:1181) 2019-04-13 21:41:06,051+0200 ERROR (jsonrpc/7) [storage.Dispatcher] FINISH getAllTasksStatuses error=Secured object is not in safe state (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error SecureError: Secured object is not in safe state 2019-04-13 21:41:06,051+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksStatuses failed (error 654) in 0.00 seconds (__init__:312) 2019-04-13 21:41:06,103+0200 INFO (jsonrpc/5) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,51130, task_id=f493a3da-2151-4a1c-8c6e-cb223d24bfae (api:48) 2019-04-13 21:41:06,104+0200 INFO (jsonrpc/5) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 21:41:06,386+0200 INFO (jsonrpc/5) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:06,386+0200 INFO (jsonrpc/5) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 21:41:06,386+0200 INFO (jsonrpc/5) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 21:41:06,587+0200 WARN (jsonrpc/5) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 21:41:06,593+0200 INFO (jsonrpc/5) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 21:41:06,593+0200 INFO (jsonrpc/5) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 21:41:06,593+0200 INFO (jsonrpc/5) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 21:41:06,593+0200 INFO (jsonrpc/5) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 21:41:06,593+0200 INFO (jsonrpc/5) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 21:41:06,593+0200 INFO (jsonrpc/5) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,51130, task_id=f493a3da-2151-4a1c-8c6e-cb223d24bfae (api:54) 2019-04-13 21:41:06,594+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.49 seconds (__init__:312) 2019-04-13 21:41:06,600+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=7140d381-2a01-4c9d-bee5-46030603ff86 (api:48) 2019-04-13 21:41:06,605+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=7140d381-2a01-4c9d-bee5-46030603ff86 (api:52) 2019-04-13 21:41:06,605+0200 ERROR (jsonrpc/4) [storage.TaskManager.Task] (Task='7140d381-2a01-4c9d-bee5-46030603ff86') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:06,605+0200 INFO (jsonrpc/4) [storage.TaskManager.Task] (Task='7140d381-2a01-4c9d-bee5-46030603ff86') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:41:06,606+0200 ERROR (jsonrpc/4) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:06,606+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.00 seconds (__init__:312) 2019-04-13 21:41:08,175+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:41:08,196+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=2a41ad67-3cf7-463e-9c16-e076d5ca82f3 (api:48) 2019-04-13 21:41:08,197+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000406063', 'lastCheck': '3.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000706039', 'lastCheck': '3.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000498919', 'lastCheck': '3.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000864768', 'lastCheck': '3.5', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=2a41ad67-3cf7-463e-9c16-e076d5ca82f3 (api:54) 2019-04-13 21:41:08,197+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=c7e4ac0d-0627-41d0-866d-a933aa2deda4 (api:48) 2019-04-13 21:41:08,198+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=c7e4ac0d-0627-41d0-866d-a933aa2deda4 (api:54) 2019-04-13 21:41:08,203+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '21': {'cpuUser': '0.93', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '98.87'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '4.45', 'nodeIndex': 0, 'cpuSys': '2.92', 'cpuIdle': '92.63'}, '0': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.67'}, '3': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.60'}, '2': {'cpuUser': '0.40', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.40'}, '5': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.73'}, '4': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '7': {'cpuUser': '3.99', 'nodeIndex': 1, 'cpuSys': '0.66', 'cpuIdle': '95.35'}, '6': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '9': {'cpuUser': '1.73', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '97.94'}, '8': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '13': {'cpuUser': '8.44', 'nodeIndex': 0, 'cpuSys': '1.13', 'cpuIdle': '90.43'}, '12': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '17': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '19': {'cpuUser': '10.83', 'nodeIndex': 1, 'cpuSys': '1.99', 'cpuIdle': '87.18'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95662'}, '0': {'memPercent': 3, 'memFree': '95456'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000406063', 'lastCheck': '3.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000498919', 'lastCheck': '3.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000864768', 'lastCheck': '3.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000706039', 'lastCheck': '3.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184468.196043, 'rx': '2386', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '434157', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184468.196043, 'rx': '351665', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184468.196043, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '422501', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184468.196043, 'rx': '328608', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '5387801', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184468.196043, 'rx': '5387801', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184468.196043, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '4861', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184468.196043, 'rx': '349027', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '429296', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184468.196043, 'rx': '2638', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184468.196043, 'rx': '12686', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184468.196043, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184468.196043, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184468.196043, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184468.196043, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184468.196043, 'rx': '8694', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '50', 'ksmPages': 100, 'elapsedTime': '56.19', 'cpuLoad': '0.69', 'cpuSys': '0.37', 'diskStats': {'/var/log': {'free': '47809'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47809'}}, 'cpuUserVdsmd': '1.66', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191810, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:41:08 GMT', 'cpuUser': '1.34', 'memFree': 191554, 'cpuIdle': '98.29', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '1.20'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:41:08,206+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:41:09,407+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:41:09,407+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:41:09,407+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:41:14,798+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:14,821+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:14,825+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:16,638+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=c104a302-c990-4382-b24b-8f451ecbd871 (api:48) 2019-04-13 21:41:16,643+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=c104a302-c990-4382-b24b-8f451ecbd871 (api:52) 2019-04-13 21:41:16,643+0200 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='c104a302-c990-4382-b24b-8f451ecbd871') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:16,643+0200 INFO (jsonrpc/0) [storage.TaskManager.Task] (Task='c104a302-c990-4382-b24b-8f451ecbd871') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:41:16,643+0200 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:16,644+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.00 seconds (__init__:312) 2019-04-13 21:41:16,658+0200 INFO (jsonrpc/1) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=0ba87987-c4ce-4951-b8b8-da3e298e5a8b (api:48) 2019-04-13 21:41:16,659+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getAllTasksStatuses error=Secured object is not in safe state from=::ffff:192.168.0.19,51130, task_id=0ba87987-c4ce-4951-b8b8-da3e298e5a8b (api:52) 2019-04-13 21:41:16,659+0200 ERROR (jsonrpc/1) [storage.TaskManager.Task] (Task='0ba87987-c4ce-4951-b8b8-da3e298e5a8b') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getAllTasksStatuses File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2195, in getAllTasksStatuses allTasksStatus = self._pool.getAllTasksStatuses() File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state 2019-04-13 21:41:16,659+0200 INFO (jsonrpc/1) [storage.TaskManager.Task] (Task='0ba87987-c4ce-4951-b8b8-da3e298e5a8b') aborting: Task is aborted: u'Secured object is not in safe state' - code 100 (task:1181) 2019-04-13 21:41:16,659+0200 ERROR (jsonrpc/1) [storage.Dispatcher] FINISH getAllTasksStatuses error=Secured object is not in safe state (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error SecureError: Secured object is not in safe state 2019-04-13 21:41:16,659+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksStatuses failed (error 654) in 0.00 seconds (__init__:312) 2019-04-13 21:41:16,713+0200 INFO (jsonrpc/2) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,51130, task_id=a3d91345-655e-4317-86ef-c470dd623130 (api:48) 2019-04-13 21:41:16,714+0200 INFO (jsonrpc/2) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 21:41:16,984+0200 INFO (jsonrpc/2) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:16,984+0200 INFO (jsonrpc/2) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 21:41:16,984+0200 INFO (jsonrpc/2) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 21:41:17,149+0200 WARN (jsonrpc/2) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 21:41:17,154+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 21:41:17,154+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 21:41:17,154+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 21:41:17,154+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 21:41:17,154+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 21:41:17,155+0200 INFO (jsonrpc/2) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,51130, task_id=a3d91345-655e-4317-86ef-c470dd623130 (api:54) 2019-04-13 21:41:17,155+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.44 seconds (__init__:312) 2019-04-13 21:41:17,161+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=c6cdde0d-0dfc-41cb-acb8-51b187306f53 (api:48) 2019-04-13 21:41:17,167+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=c6cdde0d-0dfc-41cb-acb8-51b187306f53 (api:52) 2019-04-13 21:41:17,167+0200 ERROR (jsonrpc/7) [storage.TaskManager.Task] (Task='c6cdde0d-0dfc-41cb-acb8-51b187306f53') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:17,167+0200 INFO (jsonrpc/7) [storage.TaskManager.Task] (Task='c6cdde0d-0dfc-41cb-acb8-51b187306f53') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:41:17,167+0200 ERROR (jsonrpc/7) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:17,167+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:41:18,034+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:41:18,034+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:41:18,034+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:41:20,521+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=ffde26d0-035c-4be6-81aa-ddd014ea566f (api:48) 2019-04-13 21:41:20,521+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000578149', 'lastCheck': '5.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000661547', 'lastCheck': '5.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000545764', 'lastCheck': '5.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000451871', 'lastCheck': '5.8', 'valid': True}} from=internal, task_id=ffde26d0-035c-4be6-81aa-ddd014ea566f (api:54) 2019-04-13 21:41:23,222+0200 INFO (jsonrpc/4) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:41:23,242+0200 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=ea83c438-f874-4917-891e-417c442260cd (api:48) 2019-04-13 21:41:23,243+0200 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000578149', 'lastCheck': '8.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000661547', 'lastCheck': '8.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000545764', 'lastCheck': '8.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000451871', 'lastCheck': '8.5', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=ea83c438-f874-4917-891e-417c442260cd (api:54) 2019-04-13 21:41:23,243+0200 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=d45721e0-334e-49b5-b71f-ebbe300ab9c1 (api:48) 2019-04-13 21:41:23,244+0200 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=d45721e0-334e-49b5-b71f-ebbe300ab9c1 (api:54) 2019-04-13 21:41:23,248+0200 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '1.06', 'nodeIndex': 0, 'cpuSys': '0.86', 'cpuIdle': '98.08'}, '0': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.74'}, '3': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '2': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.74'}, '5': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '7': {'cpuUser': '0.60', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.27'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '9': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '8': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '13': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.74'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '14': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.87'}, '17': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '3.39', 'nodeIndex': 1, 'cpuSys': '0.47', 'cpuIdle': '96.14'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95663'}, '0': {'memPercent': 3, 'memFree': '95457'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000578149', 'lastCheck': '8.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000545764', 'lastCheck': '8.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000451871', 'lastCheck': '8.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000661547', 'lastCheck': '8.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184483.242477, 'rx': '2776', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '465843', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184483.242477, 'rx': '377972', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184483.242477, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '453319', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184483.242477, 'rx': '352985', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '7518764', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184483.242477, 'rx': '7518764', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184483.242477, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '5083', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184483.242477, 'rx': '375092', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '460760', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184483.242477, 'rx': '2880', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184483.242477, 'rx': '15310', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184483.242477, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184483.242477, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184483.242477, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184483.242477, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184483.242477, 'rx': '10488', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '52', 'ksmPages': 100, 'elapsedTime': '71.24', 'cpuLoad': '0.66', 'cpuSys': '0.10', 'diskStats': {'/var/log': {'free': '47809'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47809'}}, 'cpuUserVdsmd': '1.06', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191812, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:41:23 GMT', 'cpuUser': '0.25', 'memFree': 191556, 'cpuIdle': '99.64', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.40'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:41:23,251+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:41:24,414+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:41:24,414+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:41:24,414+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:41:24,813+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:24,836+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:24,842+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:27,194+0200 INFO (jsonrpc/6) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=072108ca-0670-48d1-99e6-e24172c41522 (api:48) 2019-04-13 21:41:27,199+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=072108ca-0670-48d1-99e6-e24172c41522 (api:52) 2019-04-13 21:41:27,200+0200 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='072108ca-0670-48d1-99e6-e24172c41522') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:27,200+0200 INFO (jsonrpc/6) [storage.TaskManager.Task] (Task='072108ca-0670-48d1-99e6-e24172c41522') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:41:27,200+0200 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:27,200+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:41:27,206+0200 INFO (jsonrpc/0) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=0b4eb7b0-ce7f-4a9d-871b-b26e69649739 (api:48) 2019-04-13 21:41:27,206+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getAllTasksStatuses error=Secured object is not in safe state from=::ffff:192.168.0.19,51130, task_id=0b4eb7b0-ce7f-4a9d-871b-b26e69649739 (api:52) 2019-04-13 21:41:27,206+0200 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='0b4eb7b0-ce7f-4a9d-871b-b26e69649739') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getAllTasksStatuses File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2195, in getAllTasksStatuses allTasksStatus = self._pool.getAllTasksStatuses() File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state 2019-04-13 21:41:27,206+0200 INFO (jsonrpc/0) [storage.TaskManager.Task] (Task='0b4eb7b0-ce7f-4a9d-871b-b26e69649739') aborting: Task is aborted: u'Secured object is not in safe state' - code 100 (task:1181) 2019-04-13 21:41:27,206+0200 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH getAllTasksStatuses error=Secured object is not in safe state (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error SecureError: Secured object is not in safe state 2019-04-13 21:41:27,207+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksStatuses failed (error 654) in 0.00 seconds (__init__:312) 2019-04-13 21:41:27,259+0200 INFO (jsonrpc/1) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,51130, task_id=6ae80bde-cb0c-4df7-a204-ae10e00d4dd0 (api:48) 2019-04-13 21:41:27,259+0200 INFO (jsonrpc/1) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 21:41:27,531+0200 INFO (jsonrpc/1) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:27,532+0200 INFO (jsonrpc/1) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 21:41:27,532+0200 INFO (jsonrpc/1) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 21:41:27,698+0200 WARN (jsonrpc/1) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 21:41:27,703+0200 INFO (jsonrpc/1) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 21:41:27,703+0200 INFO (jsonrpc/1) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 21:41:27,703+0200 INFO (jsonrpc/1) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 21:41:27,703+0200 INFO (jsonrpc/1) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 21:41:27,703+0200 INFO (jsonrpc/1) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 21:41:27,703+0200 INFO (jsonrpc/1) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,51130, task_id=6ae80bde-cb0c-4df7-a204-ae10e00d4dd0 (api:54) 2019-04-13 21:41:27,704+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.44 seconds (__init__:312) 2019-04-13 21:41:27,710+0200 INFO (jsonrpc/2) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=62d6e28e-099c-4318-a7a6-ba8fc29db7b4 (api:48) 2019-04-13 21:41:27,715+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=62d6e28e-099c-4318-a7a6-ba8fc29db7b4 (api:52) 2019-04-13 21:41:27,715+0200 ERROR (jsonrpc/2) [storage.TaskManager.Task] (Task='62d6e28e-099c-4318-a7a6-ba8fc29db7b4') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:27,715+0200 INFO (jsonrpc/2) [storage.TaskManager.Task] (Task='62d6e28e-099c-4318-a7a6-ba8fc29db7b4') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:41:27,715+0200 ERROR (jsonrpc/2) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:27,716+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.00 seconds (__init__:312) 2019-04-13 21:41:33,039+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:41:33,039+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:41:33,039+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:41:34,828+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:34,849+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:34,857+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:35,557+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=9f65f3ad-6800-45cd-832f-c6726a3105a7 (api:48) 2019-04-13 21:41:35,558+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000739355', 'lastCheck': '0.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000761674', 'lastCheck': '0.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000520082', 'lastCheck': '0.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000427457', 'lastCheck': '0.8', 'valid': True}} from=internal, task_id=9f65f3ad-6800-45cd-832f-c6726a3105a7 (api:54) 2019-04-13 21:41:37,740+0200 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=fe10ddfd-8ccf-4f79-9b4b-7f2001d3047c (api:48) 2019-04-13 21:41:37,745+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=fe10ddfd-8ccf-4f79-9b4b-7f2001d3047c (api:52) 2019-04-13 21:41:37,745+0200 ERROR (jsonrpc/5) [storage.TaskManager.Task] (Task='fe10ddfd-8ccf-4f79-9b4b-7f2001d3047c') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:37,745+0200 INFO (jsonrpc/5) [storage.TaskManager.Task] (Task='fe10ddfd-8ccf-4f79-9b4b-7f2001d3047c') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:41:37,745+0200 ERROR (jsonrpc/5) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:37,746+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.00 seconds (__init__:312) 2019-04-13 21:41:37,750+0200 INFO (jsonrpc/4) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=2d5a7f18-ebbb-408f-bc72-ff002e3184da (api:48) 2019-04-13 21:41:37,750+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getAllTasksStatuses error=Secured object is not in safe state from=::ffff:192.168.0.19,51130, task_id=2d5a7f18-ebbb-408f-bc72-ff002e3184da (api:52) 2019-04-13 21:41:37,750+0200 ERROR (jsonrpc/4) [storage.TaskManager.Task] (Task='2d5a7f18-ebbb-408f-bc72-ff002e3184da') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getAllTasksStatuses File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2195, in getAllTasksStatuses allTasksStatus = self._pool.getAllTasksStatuses() File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state 2019-04-13 21:41:37,750+0200 INFO (jsonrpc/4) [storage.TaskManager.Task] (Task='2d5a7f18-ebbb-408f-bc72-ff002e3184da') aborting: Task is aborted: u'Secured object is not in safe state' - code 100 (task:1181) 2019-04-13 21:41:37,750+0200 ERROR (jsonrpc/4) [storage.Dispatcher] FINISH getAllTasksStatuses error=Secured object is not in safe state (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error SecureError: Secured object is not in safe state 2019-04-13 21:41:37,750+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksStatuses failed (error 654) in 0.00 seconds (__init__:312) 2019-04-13 21:41:37,806+0200 INFO (jsonrpc/3) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,51130, task_id=5318e552-2adf-4f54-996f-d874918bf6a3 (api:48) 2019-04-13 21:41:37,807+0200 INFO (jsonrpc/3) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 21:41:38,079+0200 INFO (jsonrpc/3) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:38,079+0200 INFO (jsonrpc/3) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 21:41:38,079+0200 INFO (jsonrpc/3) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 21:41:38,245+0200 WARN (jsonrpc/3) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 21:41:38,250+0200 INFO (jsonrpc/3) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 21:41:38,250+0200 INFO (jsonrpc/3) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 21:41:38,250+0200 INFO (jsonrpc/3) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 21:41:38,250+0200 INFO (jsonrpc/3) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 21:41:38,250+0200 INFO (jsonrpc/3) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 21:41:38,251+0200 INFO (jsonrpc/3) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,51130, task_id=5318e552-2adf-4f54-996f-d874918bf6a3 (api:54) 2019-04-13 21:41:38,251+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.45 seconds (__init__:312) 2019-04-13 21:41:38,260+0200 INFO (jsonrpc/6) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=70b076dc-8417-4eae-9bbc-fb857203f02c (api:48) 2019-04-13 21:41:38,265+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=70b076dc-8417-4eae-9bbc-fb857203f02c (api:52) 2019-04-13 21:41:38,265+0200 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='70b076dc-8417-4eae-9bbc-fb857203f02c') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:38,265+0200 INFO (jsonrpc/6) [storage.TaskManager.Task] (Task='70b076dc-8417-4eae-9bbc-fb857203f02c') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:41:38,265+0200 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:38,265+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.00 seconds (__init__:312) 2019-04-13 21:41:38,270+0200 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:41:38,292+0200 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=ccb384c2-fb93-45c7-b02f-0996b0282330 (api:48) 2019-04-13 21:41:38,293+0200 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000739355', 'lastCheck': '3.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000761674', 'lastCheck': '3.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000520082', 'lastCheck': '3.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000427457', 'lastCheck': '3.6', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=ccb384c2-fb93-45c7-b02f-0996b0282330 (api:54) 2019-04-13 21:41:38,293+0200 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=1b9c46a8-0586-4014-93c1-b77fdf3b7d50 (api:48) 2019-04-13 21:41:38,294+0200 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=1b9c46a8-0586-4014-93c1-b77fdf3b7d50 (api:54) 2019-04-13 21:41:38,299+0200 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '1.00', 'nodeIndex': 0, 'cpuSys': '1.00', 'cpuIdle': '98.00'}, '0': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '3': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '2': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '5': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '4': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.87'}, '7': {'cpuUser': '0.40', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.53'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '8': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '13': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '12': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '17': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '1.86', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '97.81'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95663'}, '0': {'memPercent': 3, 'memFree': '95457'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000739355', 'lastCheck': '3.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000520082', 'lastCheck': '3.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000427457', 'lastCheck': '3.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000761674', 'lastCheck': '3.5', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184498.292106, 'rx': '3166', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '510641', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184498.292106, 'rx': '416321', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184498.292106, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '500191', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184498.292106, 'rx': '394236', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '11768427', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184498.292106, 'rx': '11768427', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184498.292106, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '5083', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184498.292106, 'rx': '413223', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '505558', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184498.292106, 'rx': '3098', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184498.292106, 'rx': '17704', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184498.292106, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184498.292106, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184498.292106, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184498.292106, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184498.292106, 'rx': '12052', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '52', 'ksmPages': 100, 'elapsedTime': '86.29', 'cpuLoad': '0.62', 'cpuSys': '0.08', 'diskStats': {'/var/log': {'free': '47809'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47809'}}, 'cpuUserVdsmd': '1.00', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191812, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:41:38 GMT', 'cpuUser': '0.16', 'memFree': 191556, 'cpuIdle': '99.76', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.53'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:41:38,302+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:41:39,421+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:41:39,421+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:41:39,422+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:41:44,722+0200 INFO (monitor/6a4cde7) [IOProcessClient] (Global) Closing client (__init__:584) 2019-04-13 21:41:44,842+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:44,866+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:44,872+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:48,058+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:41:48,059+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:41:48,059+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:41:48,297+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=3d078a84-92a2-4d79-9716-f6a05a8f6fcc (api:48) 2019-04-13 21:41:48,303+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=3d078a84-92a2-4d79-9716-f6a05a8f6fcc (api:52) 2019-04-13 21:41:48,303+0200 ERROR (jsonrpc/7) [storage.TaskManager.Task] (Task='3d078a84-92a2-4d79-9716-f6a05a8f6fcc') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:48,303+0200 INFO (jsonrpc/7) [storage.TaskManager.Task] (Task='3d078a84-92a2-4d79-9716-f6a05a8f6fcc') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:41:48,303+0200 ERROR (jsonrpc/7) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:48,303+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:41:48,321+0200 INFO (jsonrpc/5) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=56d87659-c5a0-469b-a807-fdb2c053f15d (api:48) 2019-04-13 21:41:48,321+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getAllTasksStatuses error=Secured object is not in safe state from=::ffff:192.168.0.19,51130, task_id=56d87659-c5a0-469b-a807-fdb2c053f15d (api:52) 2019-04-13 21:41:48,321+0200 ERROR (jsonrpc/5) [storage.TaskManager.Task] (Task='56d87659-c5a0-469b-a807-fdb2c053f15d') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getAllTasksStatuses File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2195, in getAllTasksStatuses allTasksStatus = self._pool.getAllTasksStatuses() File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state 2019-04-13 21:41:48,321+0200 INFO (jsonrpc/5) [storage.TaskManager.Task] (Task='56d87659-c5a0-469b-a807-fdb2c053f15d') aborting: Task is aborted: u'Secured object is not in safe state' - code 100 (task:1181) 2019-04-13 21:41:48,322+0200 ERROR (jsonrpc/5) [storage.Dispatcher] FINISH getAllTasksStatuses error=Secured object is not in safe state (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error SecureError: Secured object is not in safe state 2019-04-13 21:41:48,322+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksStatuses failed (error 654) in 0.00 seconds (__init__:312) 2019-04-13 21:41:48,430+0200 INFO (jsonrpc/3) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,51130, task_id=6c9d855e-d430-42ee-bb95-8ca63935e7e8 (api:48) 2019-04-13 21:41:48,431+0200 INFO (jsonrpc/3) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 21:41:48,710+0200 INFO (jsonrpc/3) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:48,710+0200 INFO (jsonrpc/3) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 21:41:48,711+0200 INFO (jsonrpc/3) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 21:41:48,860+0200 WARN (jsonrpc/3) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 21:41:48,865+0200 INFO (jsonrpc/3) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 21:41:48,865+0200 INFO (jsonrpc/3) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 21:41:48,865+0200 INFO (jsonrpc/3) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 21:41:48,865+0200 INFO (jsonrpc/3) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 21:41:48,865+0200 INFO (jsonrpc/3) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 21:41:48,865+0200 INFO (jsonrpc/3) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,51130, task_id=6c9d855e-d430-42ee-bb95-8ca63935e7e8 (api:54) 2019-04-13 21:41:48,866+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.43 seconds (__init__:312) 2019-04-13 21:41:48,875+0200 INFO (jsonrpc/6) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=71fd7399-e97a-49d3-ab5c-d7ab9f372b1f (api:48) 2019-04-13 21:41:48,881+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=71fd7399-e97a-49d3-ab5c-d7ab9f372b1f (api:52) 2019-04-13 21:41:48,881+0200 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='71fd7399-e97a-49d3-ab5c-d7ab9f372b1f') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:48,881+0200 INFO (jsonrpc/6) [storage.TaskManager.Task] (Task='71fd7399-e97a-49d3-ab5c-d7ab9f372b1f') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:41:48,881+0200 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:48,881+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:41:50,588+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=27633a3d-4680-484f-b4cf-c5c07a62e27b (api:48) 2019-04-13 21:41:50,588+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000675761', 'lastCheck': '5.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000635265', 'lastCheck': '5.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000686626', 'lastCheck': '5.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000305595', 'lastCheck': '5.9', 'valid': True}} from=internal, task_id=27633a3d-4680-484f-b4cf-c5c07a62e27b (api:54) 2019-04-13 21:41:53,392+0200 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:41:53,412+0200 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=e7615ef7-a9f0-4349-abb2-2a773acecbf3 (api:48) 2019-04-13 21:41:53,412+0200 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000675761', 'lastCheck': '8.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000635265', 'lastCheck': '8.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000686626', 'lastCheck': '8.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000305595', 'lastCheck': '8.7', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=e7615ef7-a9f0-4349-abb2-2a773acecbf3 (api:54) 2019-04-13 21:41:53,413+0200 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=00d472e2-dd8c-4fc5-824b-501bb312dae4 (api:48) 2019-04-13 21:41:53,413+0200 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=00d472e2-dd8c-4fc5-824b-501bb312dae4 (api:54) 2019-04-13 21:41:53,418+0200 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '1.46', 'nodeIndex': 0, 'cpuSys': '1.00', 'cpuIdle': '97.54'}, '0': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.74'}, '3': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.87'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '5': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.74'}, '4': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '7': {'cpuUser': '0.67', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.20'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '8': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '13': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.53'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '15': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '17': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '2.73', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '96.94'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95666'}, '0': {'memPercent': 3, 'memFree': '95458'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000675761', 'lastCheck': '8.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000686626', 'lastCheck': '8.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000305595', 'lastCheck': '8.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000635265', 'lastCheck': '8.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184513.411877, 'rx': '3556', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '550225', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184513.411877, 'rx': '452160', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184513.411877, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '535851', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184513.411877, 'rx': '421627', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '13899546', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184513.411877, 'rx': '13899546', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184513.411877, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '5211', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184513.411877, 'rx': '448400', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '545014', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184513.411877, 'rx': '3760', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184513.411877, 'rx': '20136', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184513.411877, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184513.411877, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184513.411877, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184513.411877, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184513.411877, 'rx': '13846', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '52', 'ksmPages': 100, 'elapsedTime': '101.41', 'cpuLoad': '0.59', 'cpuSys': '0.11', 'diskStats': {'/var/log': {'free': '47809'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47809'}}, 'cpuUserVdsmd': '0.93', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191816, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:41:53 GMT', 'cpuUser': '0.25', 'memFree': 191560, 'cpuIdle': '99.65', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.53'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:41:53,420+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:41:54,428+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:41:54,428+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:41:54,429+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:41:54,857+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:54,878+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:54,887+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:58,910+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=b2d0c945-4e16-4e65-888e-947c6732d21b (api:48) 2019-04-13 21:41:58,915+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=b2d0c945-4e16-4e65-888e-947c6732d21b (api:52) 2019-04-13 21:41:58,915+0200 ERROR (jsonrpc/4) [storage.TaskManager.Task] (Task='b2d0c945-4e16-4e65-888e-947c6732d21b') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:58,915+0200 INFO (jsonrpc/4) [storage.TaskManager.Task] (Task='b2d0c945-4e16-4e65-888e-947c6732d21b') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:41:58,915+0200 ERROR (jsonrpc/4) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:58,916+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.00 seconds (__init__:312) 2019-04-13 21:41:58,923+0200 INFO (jsonrpc/2) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=1efcc784-8b49-4978-86d6-7bc4e6b88b61 (api:48) 2019-04-13 21:41:58,923+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getAllTasksStatuses error=Secured object is not in safe state from=::ffff:192.168.0.19,51130, task_id=1efcc784-8b49-4978-86d6-7bc4e6b88b61 (api:52) 2019-04-13 21:41:58,923+0200 ERROR (jsonrpc/2) [storage.TaskManager.Task] (Task='1efcc784-8b49-4978-86d6-7bc4e6b88b61') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getAllTasksStatuses File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2195, in getAllTasksStatuses allTasksStatus = self._pool.getAllTasksStatuses() File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state 2019-04-13 21:41:58,923+0200 INFO (jsonrpc/2) [storage.TaskManager.Task] (Task='1efcc784-8b49-4978-86d6-7bc4e6b88b61') aborting: Task is aborted: u'Secured object is not in safe state' - code 100 (task:1181) 2019-04-13 21:41:58,923+0200 ERROR (jsonrpc/2) [storage.Dispatcher] FINISH getAllTasksStatuses error=Secured object is not in safe state (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error SecureError: Secured object is not in safe state 2019-04-13 21:41:58,923+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksStatuses failed (error 654) in 0.00 seconds (__init__:312) 2019-04-13 21:41:59,026+0200 INFO (jsonrpc/7) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,51130, task_id=88a39213-fa76-4212-bf59-f2fc84fff74d (api:48) 2019-04-13 21:41:59,027+0200 INFO (jsonrpc/7) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 21:41:59,290+0200 INFO (jsonrpc/7) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:41:59,291+0200 INFO (jsonrpc/7) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 21:41:59,291+0200 INFO (jsonrpc/7) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 21:41:59,440+0200 WARN (jsonrpc/7) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 21:41:59,445+0200 INFO (jsonrpc/7) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 21:41:59,445+0200 INFO (jsonrpc/7) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 21:41:59,445+0200 INFO (jsonrpc/7) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 21:41:59,445+0200 INFO (jsonrpc/7) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 21:41:59,445+0200 INFO (jsonrpc/7) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 21:41:59,446+0200 INFO (jsonrpc/7) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,51130, task_id=88a39213-fa76-4212-bf59-f2fc84fff74d (api:54) 2019-04-13 21:41:59,446+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.42 seconds (__init__:312) 2019-04-13 21:41:59,452+0200 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=8c45afc7-376e-456b-b44b-e716d44d0f78 (api:48) 2019-04-13 21:41:59,458+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=8c45afc7-376e-456b-b44b-e716d44d0f78 (api:52) 2019-04-13 21:41:59,458+0200 ERROR (jsonrpc/5) [storage.TaskManager.Task] (Task='8c45afc7-376e-456b-b44b-e716d44d0f78') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:59,458+0200 INFO (jsonrpc/5) [storage.TaskManager.Task] (Task='8c45afc7-376e-456b-b44b-e716d44d0f78') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:41:59,458+0200 ERROR (jsonrpc/5) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:41:59,458+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:42:03,078+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:42:03,078+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:42:03,078+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:42:04,871+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:04,892+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:04,902+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:05,618+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=e622e8db-9f18-43d9-b4b4-cd65c6fbc467 (api:48) 2019-04-13 21:42:05,619+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000644146', 'lastCheck': '0.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000894576', 'lastCheck': '0.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000802724', 'lastCheck': '0.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000569045', 'lastCheck': '0.9', 'valid': True}} from=internal, task_id=e622e8db-9f18-43d9-b4b4-cd65c6fbc467 (api:54) 2019-04-13 21:42:08,437+0200 INFO (jsonrpc/6) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:42:08,458+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=f59cb15f-5359-405a-a81f-f1ab391f81d2 (api:48) 2019-04-13 21:42:08,459+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000644146', 'lastCheck': '3.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000894576', 'lastCheck': '3.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000802724', 'lastCheck': '3.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000569045', 'lastCheck': '3.7', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=f59cb15f-5359-405a-a81f-f1ab391f81d2 (api:54) 2019-04-13 21:42:08,459+0200 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=6e88c1fe-69af-4f70-ba28-49cf97ada360 (api:48) 2019-04-13 21:42:08,460+0200 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=6e88c1fe-69af-4f70-ba28-49cf97ada360 (api:54) 2019-04-13 21:42:08,464+0200 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '21': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '1.00', 'nodeIndex': 0, 'cpuSys': '1.06', 'cpuIdle': '97.94'}, '0': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '3': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '2': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '5': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '4': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '7': {'cpuUser': '0.60', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.13'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '9': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '8': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '13': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '17': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '19': {'cpuUser': '1.60', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '98.13'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95666'}, '0': {'memPercent': 3, 'memFree': '95458'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000644146', 'lastCheck': '3.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000802724', 'lastCheck': '3.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000569045', 'lastCheck': '3.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000894576', 'lastCheck': '3.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184528.458115, 'rx': '3946', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '592265', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184528.458115, 'rx': '490579', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184528.458115, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '577121', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184528.458115, 'rx': '457360', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '16031177', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184528.458115, 'rx': '16031177', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184528.458115, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '5211', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184528.458115, 'rx': '486345', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '587054', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184528.458115, 'rx': '4234', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184528.458115, 'rx': '22786', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184528.458115, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184528.458115, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184528.458115, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184528.458115, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184528.458115, 'rx': '15594', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '50', 'ksmPages': 100, 'elapsedTime': '116.46', 'cpuLoad': '0.56', 'cpuSys': '0.09', 'diskStats': {'/var/log': {'free': '47809'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47809'}}, 'cpuUserVdsmd': '0.86', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191815, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:42:08 GMT', 'cpuUser': '0.15', 'memFree': 191559, 'cpuIdle': '99.76', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.53'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:42:08,467+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:42:09,436+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:42:09,436+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:42:09,437+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:42:09,485+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=dd93db2f-9f27-4271-bd76-ae1a7e037f57 (api:48) 2019-04-13 21:42:09,491+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=dd93db2f-9f27-4271-bd76-ae1a7e037f57 (api:52) 2019-04-13 21:42:09,491+0200 ERROR (jsonrpc/1) [storage.TaskManager.Task] (Task='dd93db2f-9f27-4271-bd76-ae1a7e037f57') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:09,491+0200 INFO (jsonrpc/1) [storage.TaskManager.Task] (Task='dd93db2f-9f27-4271-bd76-ae1a7e037f57') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:42:09,491+0200 ERROR (jsonrpc/1) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:09,491+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:42:09,497+0200 INFO (jsonrpc/4) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=91b5e59c-ddec-476a-9ebc-73d28c30bd1d (api:48) 2019-04-13 21:42:09,497+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getAllTasksStatuses error=Secured object is not in safe state from=::ffff:192.168.0.19,51130, task_id=91b5e59c-ddec-476a-9ebc-73d28c30bd1d (api:52) 2019-04-13 21:42:09,497+0200 ERROR (jsonrpc/4) [storage.TaskManager.Task] (Task='91b5e59c-ddec-476a-9ebc-73d28c30bd1d') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getAllTasksStatuses File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2195, in getAllTasksStatuses allTasksStatus = self._pool.getAllTasksStatuses() File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state 2019-04-13 21:42:09,497+0200 INFO (jsonrpc/4) [storage.TaskManager.Task] (Task='91b5e59c-ddec-476a-9ebc-73d28c30bd1d') aborting: Task is aborted: u'Secured object is not in safe state' - code 100 (task:1181) 2019-04-13 21:42:09,497+0200 ERROR (jsonrpc/4) [storage.Dispatcher] FINISH getAllTasksStatuses error=Secured object is not in safe state (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error SecureError: Secured object is not in safe state 2019-04-13 21:42:09,497+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksStatuses failed (error 654) in 0.01 seconds (__init__:312) 2019-04-13 21:42:09,543+0200 INFO (jsonrpc/2) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,51130, task_id=4ada0bfb-5925-4160-b170-419e82baac2e (api:48) 2019-04-13 21:42:09,544+0200 INFO (jsonrpc/2) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 21:42:09,813+0200 INFO (jsonrpc/2) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:09,814+0200 INFO (jsonrpc/2) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 21:42:09,814+0200 INFO (jsonrpc/2) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 21:42:09,979+0200 WARN (jsonrpc/2) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 21:42:09,984+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 21:42:09,984+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 21:42:09,985+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 21:42:09,985+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 21:42:09,985+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 21:42:09,985+0200 INFO (jsonrpc/2) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,51130, task_id=4ada0bfb-5925-4160-b170-419e82baac2e (api:54) 2019-04-13 21:42:09,986+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.44 seconds (__init__:312) 2019-04-13 21:42:09,993+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=ac9c612d-0eac-49df-8fad-179783e2c8ad (api:48) 2019-04-13 21:42:09,998+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=ac9c612d-0eac-49df-8fad-179783e2c8ad (api:52) 2019-04-13 21:42:09,998+0200 ERROR (jsonrpc/7) [storage.TaskManager.Task] (Task='ac9c612d-0eac-49df-8fad-179783e2c8ad') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:09,998+0200 INFO (jsonrpc/7) [storage.TaskManager.Task] (Task='ac9c612d-0eac-49df-8fad-179783e2c8ad') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:42:09,998+0200 ERROR (jsonrpc/7) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:09,999+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:42:14,887+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:14,907+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:14,918+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:18,082+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:42:18,082+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:42:18,083+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:42:20,025+0200 INFO (jsonrpc/3) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=73afa630-9ed2-49b4-ab84-d441ee704f14 (api:48) 2019-04-13 21:42:20,031+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=73afa630-9ed2-49b4-ab84-d441ee704f14 (api:52) 2019-04-13 21:42:20,031+0200 ERROR (jsonrpc/3) [storage.TaskManager.Task] (Task='73afa630-9ed2-49b4-ab84-d441ee704f14') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:20,031+0200 INFO (jsonrpc/3) [storage.TaskManager.Task] (Task='73afa630-9ed2-49b4-ab84-d441ee704f14') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:42:20,031+0200 ERROR (jsonrpc/3) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:20,031+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:42:20,046+0200 INFO (jsonrpc/6) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=0ea2fe1d-a93b-4fa2-88e7-bebd20d8dc72 (api:48) 2019-04-13 21:42:20,046+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getAllTasksStatuses error=Secured object is not in safe state from=::ffff:192.168.0.19,51130, task_id=0ea2fe1d-a93b-4fa2-88e7-bebd20d8dc72 (api:52) 2019-04-13 21:42:20,046+0200 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='0ea2fe1d-a93b-4fa2-88e7-bebd20d8dc72') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getAllTasksStatuses File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2195, in getAllTasksStatuses allTasksStatus = self._pool.getAllTasksStatuses() File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state 2019-04-13 21:42:20,046+0200 INFO (jsonrpc/6) [storage.TaskManager.Task] (Task='0ea2fe1d-a93b-4fa2-88e7-bebd20d8dc72') aborting: Task is aborted: u'Secured object is not in safe state' - code 100 (task:1181) 2019-04-13 21:42:20,046+0200 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH getAllTasksStatuses error=Secured object is not in safe state (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error SecureError: Secured object is not in safe state 2019-04-13 21:42:20,046+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksStatuses failed (error 654) in 0.00 seconds (__init__:312) 2019-04-13 21:42:20,105+0200 INFO (jsonrpc/0) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,51130, task_id=b6d0e992-5b80-41fc-94ae-87a6794ea73f (api:48) 2019-04-13 21:42:20,106+0200 INFO (jsonrpc/0) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 21:42:20,378+0200 INFO (jsonrpc/0) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:20,379+0200 INFO (jsonrpc/0) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 21:42:20,379+0200 INFO (jsonrpc/0) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 21:42:20,543+0200 WARN (jsonrpc/0) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 21:42:20,548+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 21:42:20,548+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 21:42:20,549+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 21:42:20,549+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 21:42:20,549+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 21:42:20,549+0200 INFO (jsonrpc/0) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,51130, task_id=b6d0e992-5b80-41fc-94ae-87a6794ea73f (api:54) 2019-04-13 21:42:20,550+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.45 seconds (__init__:312) 2019-04-13 21:42:20,557+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=7223beb2-35e0-43e7-bdd9-d657c108b19e (api:48) 2019-04-13 21:42:20,562+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=7223beb2-35e0-43e7-bdd9-d657c108b19e (api:52) 2019-04-13 21:42:20,562+0200 ERROR (jsonrpc/1) [storage.TaskManager.Task] (Task='7223beb2-35e0-43e7-bdd9-d657c108b19e') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:20,562+0200 INFO (jsonrpc/1) [storage.TaskManager.Task] (Task='7223beb2-35e0-43e7-bdd9-d657c108b19e') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:42:20,562+0200 ERROR (jsonrpc/1) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:20,562+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:42:20,648+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=5bd33fd3-9fdd-429c-b5e2-141a48c797dc (api:48) 2019-04-13 21:42:20,648+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000646036', 'lastCheck': '5.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000421731', 'lastCheck': '5.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000797911', 'lastCheck': '5.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000466555', 'lastCheck': '5.9', 'valid': True}} from=internal, task_id=5bd33fd3-9fdd-429c-b5e2-141a48c797dc (api:54) 2019-04-13 21:42:23,482+0200 INFO (jsonrpc/4) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:42:23,502+0200 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=45995a89-9f27-4869-a926-f70dbbeb6313 (api:48) 2019-04-13 21:42:23,502+0200 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000646036', 'lastCheck': '8.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000421731', 'lastCheck': '8.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000797911', 'lastCheck': '8.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000466555', 'lastCheck': '8.8', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=45995a89-9f27-4869-a926-f70dbbeb6313 (api:54) 2019-04-13 21:42:23,503+0200 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=242f77e1-2a55-478d-8afd-13b5c9091864 (api:48) 2019-04-13 21:42:23,503+0200 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=242f77e1-2a55-478d-8afd-13b5c9091864 (api:54) 2019-04-13 21:42:23,508+0200 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '1.13', 'nodeIndex': 0, 'cpuSys': '0.87', 'cpuIdle': '98.00'}, '0': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.74'}, '3': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.74'}, '2': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '5': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '4': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.87'}, '7': {'cpuUser': '0.80', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.07'}, '6': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '9': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '8': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '10': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '13': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '12': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '17': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '16': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '19': {'cpuUser': '3.13', 'nodeIndex': 1, 'cpuSys': '0.47', 'cpuIdle': '96.40'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95665'}, '0': {'memPercent': 3, 'memFree': '95459'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000646036', 'lastCheck': '8.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000797911', 'lastCheck': '8.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000466555', 'lastCheck': '8.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000421731', 'lastCheck': '8.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184543.502052, 'rx': '4413', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '629791', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184543.502052, 'rx': '522546', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184543.502052, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '613697', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184543.502052, 'rx': '487055', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '20280380', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184543.502052, 'rx': '20280380', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184543.502052, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '5433', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184543.502052, 'rx': '518184', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '624358', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184543.502052, 'rx': '4362', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184543.502052, 'rx': '25154', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184543.502052, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184543.502052, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184543.502052, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184543.502052, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184543.502052, 'rx': '17342', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '52', 'ksmPages': 100, 'elapsedTime': '131.50', 'cpuLoad': '0.54', 'cpuSys': '0.10', 'diskStats': {'/var/log': {'free': '47809'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47809'}}, 'cpuUserVdsmd': '1.06', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191817, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:42:23 GMT', 'cpuUser': '0.25', 'memFree': 191561, 'cpuIdle': '99.66', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.47'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:42:23,511+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:42:24,443+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:42:24,443+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:42:24,444+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:42:24,902+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:24,920+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:24,933+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:30,596+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=00d586c8-2a8b-47b5-a317-1ea1c42677e0 (api:48) 2019-04-13 21:42:30,601+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=00d586c8-2a8b-47b5-a317-1ea1c42677e0 (api:52) 2019-04-13 21:42:30,601+0200 ERROR (jsonrpc/7) [storage.TaskManager.Task] (Task='00d586c8-2a8b-47b5-a317-1ea1c42677e0') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:30,602+0200 INFO (jsonrpc/7) [storage.TaskManager.Task] (Task='00d586c8-2a8b-47b5-a317-1ea1c42677e0') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:42:30,602+0200 ERROR (jsonrpc/7) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:30,602+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:42:30,608+0200 INFO (jsonrpc/5) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=07e31dff-53a0-46d8-a2c8-64c2e60b1a33 (api:48) 2019-04-13 21:42:30,608+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getAllTasksStatuses error=Secured object is not in safe state from=::ffff:192.168.0.19,51130, task_id=07e31dff-53a0-46d8-a2c8-64c2e60b1a33 (api:52) 2019-04-13 21:42:30,608+0200 ERROR (jsonrpc/5) [storage.TaskManager.Task] (Task='07e31dff-53a0-46d8-a2c8-64c2e60b1a33') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getAllTasksStatuses File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2195, in getAllTasksStatuses allTasksStatus = self._pool.getAllTasksStatuses() File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state 2019-04-13 21:42:30,608+0200 INFO (jsonrpc/5) [storage.TaskManager.Task] (Task='07e31dff-53a0-46d8-a2c8-64c2e60b1a33') aborting: Task is aborted: u'Secured object is not in safe state' - code 100 (task:1181) 2019-04-13 21:42:30,608+0200 ERROR (jsonrpc/5) [storage.Dispatcher] FINISH getAllTasksStatuses error=Secured object is not in safe state (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error SecureError: Secured object is not in safe state 2019-04-13 21:42:30,608+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksStatuses failed (error 654) in 0.00 seconds (__init__:312) 2019-04-13 21:42:30,648+0200 INFO (jsonrpc/3) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,51130, task_id=7279f990-b362-46ae-b9d8-0f850002ffba (api:48) 2019-04-13 21:42:30,649+0200 INFO (jsonrpc/3) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 21:42:30,918+0200 INFO (jsonrpc/3) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:30,918+0200 INFO (jsonrpc/3) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 21:42:30,918+0200 INFO (jsonrpc/3) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 21:42:31,084+0200 WARN (jsonrpc/3) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 21:42:31,089+0200 INFO (jsonrpc/3) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 21:42:31,089+0200 INFO (jsonrpc/3) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 21:42:31,089+0200 INFO (jsonrpc/3) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 21:42:31,090+0200 INFO (jsonrpc/3) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 21:42:31,090+0200 INFO (jsonrpc/3) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 21:42:31,090+0200 INFO (jsonrpc/3) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,51130, task_id=7279f990-b362-46ae-b9d8-0f850002ffba (api:54) 2019-04-13 21:42:31,090+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.44 seconds (__init__:312) 2019-04-13 21:42:31,097+0200 INFO (jsonrpc/6) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=81711613-7aee-4ea8-8696-eb1583228f45 (api:48) 2019-04-13 21:42:31,102+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=81711613-7aee-4ea8-8696-eb1583228f45 (api:52) 2019-04-13 21:42:31,102+0200 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='81711613-7aee-4ea8-8696-eb1583228f45') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:31,102+0200 INFO (jsonrpc/6) [storage.TaskManager.Task] (Task='81711613-7aee-4ea8-8696-eb1583228f45') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:42:31,102+0200 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:31,102+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:42:33,101+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:42:33,101+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:42:33,101+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:42:34,917+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:34,932+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:34,946+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:35,677+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=7e73b90d-ad26-4672-9b29-a4290b633088 (api:48) 2019-04-13 21:42:35,677+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000744357', 'lastCheck': '0.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000790057', 'lastCheck': '0.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.0005309', 'lastCheck': '0.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.0003434', 'lastCheck': '0.9', 'valid': True}} from=internal, task_id=7e73b90d-ad26-4672-9b29-a4290b633088 (api:54) 2019-04-13 21:42:38,530+0200 INFO (jsonrpc/1) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:42:38,550+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=302d9386-2fae-4f65-bb87-5dd609c781aa (api:48) 2019-04-13 21:42:38,551+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000744357', 'lastCheck': '3.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000790057', 'lastCheck': '3.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.0005309', 'lastCheck': '3.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.0003434', 'lastCheck': '3.8', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=302d9386-2fae-4f65-bb87-5dd609c781aa (api:54) 2019-04-13 21:42:38,551+0200 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=2ce210e2-4e2e-40c5-9f62-957fac009522 (api:48) 2019-04-13 21:42:38,552+0200 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=2ce210e2-4e2e-40c5-9f62-957fac009522 (api:54) 2019-04-13 21:42:38,556+0200 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '1.00', 'nodeIndex': 0, 'cpuSys': '0.93', 'cpuIdle': '98.07'}, '0': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '3': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '5': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '7': {'cpuUser': '0.40', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.53'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '9': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '8': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '10': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '13': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '12': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.87'}, '17': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '1.80', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '97.87'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95665'}, '0': {'memPercent': 3, 'memFree': '95458'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000744357', 'lastCheck': '3.6', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.0005309', 'lastCheck': '3.6', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.0003434', 'lastCheck': '3.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000790057', 'lastCheck': '3.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184558.550409, 'rx': '4880', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '671959', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184558.550409, 'rx': '560661', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184558.550409, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '655051', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184558.550409, 'rx': '522538', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '22412003', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184558.550409, 'rx': '22412003', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184558.550409, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '5433', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184558.550409, 'rx': '556081', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '666526', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184558.550409, 'rx': '4580', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184558.550409, 'rx': '27868', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184558.550409, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184558.550409, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184558.550409, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184558.550409, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184558.550409, 'rx': '18998', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '50', 'ksmPages': 100, 'elapsedTime': '146.55', 'cpuLoad': '0.51', 'cpuSys': '0.08', 'diskStats': {'/var/log': {'free': '47809'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47809'}}, 'cpuUserVdsmd': '0.93', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191815, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:42:38 GMT', 'cpuUser': '0.16', 'memFree': 191559, 'cpuIdle': '99.76', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.47'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:42:38,559+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:42:39,450+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:42:39,451+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:42:39,451+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:42:41,129+0200 INFO (jsonrpc/2) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=ca85eda5-1529-4d5b-8b4c-fc5bd2703e31 (api:48) 2019-04-13 21:42:41,135+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=ca85eda5-1529-4d5b-8b4c-fc5bd2703e31 (api:52) 2019-04-13 21:42:41,135+0200 ERROR (jsonrpc/2) [storage.TaskManager.Task] (Task='ca85eda5-1529-4d5b-8b4c-fc5bd2703e31') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:41,135+0200 INFO (jsonrpc/2) [storage.TaskManager.Task] (Task='ca85eda5-1529-4d5b-8b4c-fc5bd2703e31') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:42:41,135+0200 ERROR (jsonrpc/2) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:41,135+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.00 seconds (__init__:312) 2019-04-13 21:42:41,140+0200 INFO (jsonrpc/7) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=5ccc4999-1f80-41e2-83e8-3918982799bc (api:48) 2019-04-13 21:42:41,140+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getAllTasksStatuses error=Secured object is not in safe state from=::ffff:192.168.0.19,51130, task_id=5ccc4999-1f80-41e2-83e8-3918982799bc (api:52) 2019-04-13 21:42:41,140+0200 ERROR (jsonrpc/7) [storage.TaskManager.Task] (Task='5ccc4999-1f80-41e2-83e8-3918982799bc') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getAllTasksStatuses File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2195, in getAllTasksStatuses allTasksStatus = self._pool.getAllTasksStatuses() File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state 2019-04-13 21:42:41,140+0200 INFO (jsonrpc/7) [storage.TaskManager.Task] (Task='5ccc4999-1f80-41e2-83e8-3918982799bc') aborting: Task is aborted: u'Secured object is not in safe state' - code 100 (task:1181) 2019-04-13 21:42:41,140+0200 ERROR (jsonrpc/7) [storage.Dispatcher] FINISH getAllTasksStatuses error=Secured object is not in safe state (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error SecureError: Secured object is not in safe state 2019-04-13 21:42:41,140+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksStatuses failed (error 654) in 0.00 seconds (__init__:312) 2019-04-13 21:42:41,185+0200 INFO (jsonrpc/5) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,51130, task_id=90a3ddc6-bc3f-4a69-a37d-e0e5016e4ad6 (api:48) 2019-04-13 21:42:41,186+0200 INFO (jsonrpc/5) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 21:42:41,458+0200 INFO (jsonrpc/5) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:41,458+0200 INFO (jsonrpc/5) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 21:42:41,458+0200 INFO (jsonrpc/5) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 21:42:41,624+0200 WARN (jsonrpc/5) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 21:42:41,629+0200 INFO (jsonrpc/5) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 21:42:41,629+0200 INFO (jsonrpc/5) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 21:42:41,629+0200 INFO (jsonrpc/5) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 21:42:41,629+0200 INFO (jsonrpc/5) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 21:42:41,629+0200 INFO (jsonrpc/5) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 21:42:41,630+0200 INFO (jsonrpc/5) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,51130, task_id=90a3ddc6-bc3f-4a69-a37d-e0e5016e4ad6 (api:54) 2019-04-13 21:42:41,630+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.45 seconds (__init__:312) 2019-04-13 21:42:41,637+0200 INFO (jsonrpc/3) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=288f4338-ddfc-4543-b212-032003fc04ea (api:48) 2019-04-13 21:42:41,642+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=288f4338-ddfc-4543-b212-032003fc04ea (api:52) 2019-04-13 21:42:41,642+0200 ERROR (jsonrpc/3) [storage.TaskManager.Task] (Task='288f4338-ddfc-4543-b212-032003fc04ea') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:41,642+0200 INFO (jsonrpc/3) [storage.TaskManager.Task] (Task='288f4338-ddfc-4543-b212-032003fc04ea') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:42:41,642+0200 ERROR (jsonrpc/3) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:41,643+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:42:44,933+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:44,946+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:44,960+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:48,120+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:42:48,121+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:42:48,121+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:42:50,704+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=4f77f55b-de47-4bae-b8ff-afd0cf8e9098 (api:48) 2019-04-13 21:42:50,705+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000666267', 'lastCheck': '5.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000610728', 'lastCheck': '5.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000537685', 'lastCheck': '5.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000333063', 'lastCheck': '5.9', 'valid': True}} from=internal, task_id=4f77f55b-de47-4bae-b8ff-afd0cf8e9098 (api:54) 2019-04-13 21:42:51,673+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=f8a917b4-3449-4f47-ba45-9dc612b5e19f (api:48) 2019-04-13 21:42:51,678+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=f8a917b4-3449-4f47-ba45-9dc612b5e19f (api:52) 2019-04-13 21:42:51,679+0200 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='f8a917b4-3449-4f47-ba45-9dc612b5e19f') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:51,679+0200 INFO (jsonrpc/0) [storage.TaskManager.Task] (Task='f8a917b4-3449-4f47-ba45-9dc612b5e19f') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:42:51,679+0200 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:51,679+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:42:51,710+0200 INFO (jsonrpc/1) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=3b6a745a-9d89-40ea-a2cf-cb516586b0a3 (api:48) 2019-04-13 21:42:51,710+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getAllTasksStatuses error=Secured object is not in safe state from=::ffff:192.168.0.19,51130, task_id=3b6a745a-9d89-40ea-a2cf-cb516586b0a3 (api:52) 2019-04-13 21:42:51,710+0200 ERROR (jsonrpc/1) [storage.TaskManager.Task] (Task='3b6a745a-9d89-40ea-a2cf-cb516586b0a3') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getAllTasksStatuses File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2195, in getAllTasksStatuses allTasksStatus = self._pool.getAllTasksStatuses() File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state 2019-04-13 21:42:51,710+0200 INFO (jsonrpc/1) [storage.TaskManager.Task] (Task='3b6a745a-9d89-40ea-a2cf-cb516586b0a3') aborting: Task is aborted: u'Secured object is not in safe state' - code 100 (task:1181) 2019-04-13 21:42:51,710+0200 ERROR (jsonrpc/1) [storage.Dispatcher] FINISH getAllTasksStatuses error=Secured object is not in safe state (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error SecureError: Secured object is not in safe state 2019-04-13 21:42:51,710+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksStatuses failed (error 654) in 0.00 seconds (__init__:312) 2019-04-13 21:42:51,744+0200 INFO (jsonrpc/4) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,51130, task_id=97468b61-db5a-4fcd-8e1b-d90ca37a254a (api:48) 2019-04-13 21:42:51,744+0200 INFO (jsonrpc/4) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 21:42:52,023+0200 INFO (jsonrpc/4) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:52,024+0200 INFO (jsonrpc/4) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 21:42:52,024+0200 INFO (jsonrpc/4) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 21:42:52,175+0200 WARN (jsonrpc/4) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 21:42:52,180+0200 INFO (jsonrpc/4) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 21:42:52,180+0200 INFO (jsonrpc/4) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 21:42:52,180+0200 INFO (jsonrpc/4) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 21:42:52,181+0200 INFO (jsonrpc/4) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 21:42:52,181+0200 INFO (jsonrpc/4) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 21:42:52,181+0200 INFO (jsonrpc/4) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,51130, task_id=97468b61-db5a-4fcd-8e1b-d90ca37a254a (api:54) 2019-04-13 21:42:52,181+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.44 seconds (__init__:312) 2019-04-13 21:42:52,188+0200 INFO (jsonrpc/2) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=e08babe2-a183-4c4f-b9af-2d0f2ed538f4 (api:48) 2019-04-13 21:42:52,193+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=e08babe2-a183-4c4f-b9af-2d0f2ed538f4 (api:52) 2019-04-13 21:42:52,193+0200 ERROR (jsonrpc/2) [storage.TaskManager.Task] (Task='e08babe2-a183-4c4f-b9af-2d0f2ed538f4') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:52,193+0200 INFO (jsonrpc/2) [storage.TaskManager.Task] (Task='e08babe2-a183-4c4f-b9af-2d0f2ed538f4') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:42:52,193+0200 ERROR (jsonrpc/2) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:42:52,193+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.00 seconds (__init__:312) 2019-04-13 21:42:53,598+0200 INFO (jsonrpc/7) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:42:53,619+0200 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=91eebd16-ea54-4007-af3c-5cfde0ccc06e (api:48) 2019-04-13 21:42:53,620+0200 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000666267', 'lastCheck': '8.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000610728', 'lastCheck': '8.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000537685', 'lastCheck': '8.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000333063', 'lastCheck': '8.9', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=91eebd16-ea54-4007-af3c-5cfde0ccc06e (api:54) 2019-04-13 21:42:53,620+0200 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=ec1ca1ae-a171-45e1-9019-c99dad712665 (api:48) 2019-04-13 21:42:53,620+0200 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=ec1ca1ae-a171-45e1-9019-c99dad712665 (api:54) 2019-04-13 21:42:53,626+0200 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '21': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '0.80', 'nodeIndex': 0, 'cpuSys': '0.60', 'cpuIdle': '98.60'}, '0': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '3': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '2': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '5': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '7': {'cpuUser': '0.40', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.53'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '9': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '8': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '13': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '14': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.73'}, '17': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '1.86', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '97.87'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95665'}, '0': {'memPercent': 3, 'memFree': '95459'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000666267', 'lastCheck': '8.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000537685', 'lastCheck': '8.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000333063', 'lastCheck': '8.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000610728', 'lastCheck': '8.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184573.619078, 'rx': '5347', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '708529', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184573.619078, 'rx': '592557', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184573.619078, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '690695', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184573.619078, 'rx': '552130', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '26661154', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184573.619078, 'rx': '26661154', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184573.619078, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '5561', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184573.619078, 'rx': '587062', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '702968', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184573.619078, 'rx': '5495', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184573.619078, 'rx': '30364', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184573.619078, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184573.619078, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184573.619078, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184573.619078, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184573.619078, 'rx': '20792', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '50', 'ksmPages': 100, 'elapsedTime': '161.62', 'cpuLoad': '0.49', 'cpuSys': '0.06', 'diskStats': {'/var/log': {'free': '47809'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47809'}}, 'cpuUserVdsmd': '0.80', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191816, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:42:53 GMT', 'cpuUser': '0.16', 'memFree': 191560, 'cpuIdle': '99.78', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.33'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:42:53,628+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:42:54,458+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:42:54,459+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:42:54,459+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:42:54,948+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:54,959+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:42:54,972+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:43:02,223+0200 INFO (jsonrpc/3) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=4c5a980b-c126-4cf6-a302-b44ad28c4de0 (api:48) 2019-04-13 21:43:02,228+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=4c5a980b-c126-4cf6-a302-b44ad28c4de0 (api:52) 2019-04-13 21:43:02,228+0200 ERROR (jsonrpc/3) [storage.TaskManager.Task] (Task='4c5a980b-c126-4cf6-a302-b44ad28c4de0') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:43:02,228+0200 INFO (jsonrpc/3) [storage.TaskManager.Task] (Task='4c5a980b-c126-4cf6-a302-b44ad28c4de0') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:43:02,229+0200 ERROR (jsonrpc/3) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:43:02,229+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:43:02,235+0200 INFO (jsonrpc/6) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=e347fa6d-1647-4988-858e-9df81bb649e9 (api:48) 2019-04-13 21:43:02,235+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getAllTasksStatuses error=Secured object is not in safe state from=::ffff:192.168.0.19,51130, task_id=e347fa6d-1647-4988-858e-9df81bb649e9 (api:52) 2019-04-13 21:43:02,235+0200 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='e347fa6d-1647-4988-858e-9df81bb649e9') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getAllTasksStatuses File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2195, in getAllTasksStatuses allTasksStatus = self._pool.getAllTasksStatuses() File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state 2019-04-13 21:43:02,235+0200 INFO (jsonrpc/6) [storage.TaskManager.Task] (Task='e347fa6d-1647-4988-858e-9df81bb649e9') aborting: Task is aborted: u'Secured object is not in safe state' - code 100 (task:1181) 2019-04-13 21:43:02,235+0200 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH getAllTasksStatuses error=Secured object is not in safe state (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error SecureError: Secured object is not in safe state 2019-04-13 21:43:02,235+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksStatuses failed (error 654) in 0.00 seconds (__init__:312) 2019-04-13 21:43:02,264+0200 INFO (jsonrpc/0) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,51130, task_id=4dd3499c-b7d8-4baf-9e54-dbf460d12694 (api:48) 2019-04-13 21:43:02,265+0200 INFO (jsonrpc/0) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 21:43:02,538+0200 INFO (jsonrpc/0) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:43:02,538+0200 INFO (jsonrpc/0) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 21:43:02,538+0200 INFO (jsonrpc/0) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 21:43:02,728+0200 WARN (jsonrpc/0) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 21:43:02,734+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 21:43:02,734+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 21:43:02,734+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 21:43:02,734+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 21:43:02,734+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 21:43:02,735+0200 INFO (jsonrpc/0) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,51130, task_id=4dd3499c-b7d8-4baf-9e54-dbf460d12694 (api:54) 2019-04-13 21:43:02,735+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.47 seconds (__init__:312) 2019-04-13 21:43:02,742+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=ff8cbfb2-b6e7-40e6-8247-da3592a697a4 (api:48) 2019-04-13 21:43:02,748+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=ff8cbfb2-b6e7-40e6-8247-da3592a697a4 (api:52) 2019-04-13 21:43:02,748+0200 ERROR (jsonrpc/1) [storage.TaskManager.Task] (Task='ff8cbfb2-b6e7-40e6-8247-da3592a697a4') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:43:02,748+0200 INFO (jsonrpc/1) [storage.TaskManager.Task] (Task='ff8cbfb2-b6e7-40e6-8247-da3592a697a4') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:43:02,748+0200 ERROR (jsonrpc/1) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:43:02,749+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:43:03,140+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:43:03,140+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:43:03,141+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:43:04,963+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:43:04,973+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:43:04,986+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:43:05,741+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=274cba9a-8a1d-4d7f-9712-7cdd2d4f5b86 (api:48) 2019-04-13 21:43:05,741+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.00062348', 'lastCheck': '0.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000785152', 'lastCheck': '0.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000631825', 'lastCheck': '0.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000391739', 'lastCheck': '1.0', 'valid': True}} from=internal, task_id=274cba9a-8a1d-4d7f-9712-7cdd2d4f5b86 (api:54) 2019-04-13 21:43:08,650+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:43:08,670+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=e1af2a8e-7fbf-41c1-9862-1fcca5fb685b (api:48) 2019-04-13 21:43:08,670+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.00062348', 'lastCheck': '3.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000785152', 'lastCheck': '3.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000631825', 'lastCheck': '3.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000391739', 'lastCheck': '3.9', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=e1af2a8e-7fbf-41c1-9862-1fcca5fb685b (api:54) 2019-04-13 21:43:08,671+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=534458eb-122f-4ac7-ae3d-940fdc72f052 (api:48) 2019-04-13 21:43:08,671+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=534458eb-122f-4ac7-ae3d-940fdc72f052 (api:54) 2019-04-13 21:43:08,676+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '21': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '1.40', 'nodeIndex': 0, 'cpuSys': '1.40', 'cpuIdle': '97.20'}, '0': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '3': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.67'}, '2': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '5': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '4': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '7': {'cpuUser': '1.86', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '98.07'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '9': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '8': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '13': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '1.46', 'cpuIdle': '98.54'}, '14': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.66'}, '17': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '3.39', 'nodeIndex': 1, 'cpuSys': '0.67', 'cpuIdle': '95.94'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95662'}, '0': {'memPercent': 3, 'memFree': '95459'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.00062348', 'lastCheck': '3.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000631825', 'lastCheck': '3.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000391739', 'lastCheck': '3.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000785152', 'lastCheck': '3.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184588.670041, 'rx': '5737', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '756715', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184588.670041, 'rx': '638632', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184588.670041, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '737891', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184588.670041, 'rx': '594581', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '28794691', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184588.670041, 'rx': '28794691', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184588.670041, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '12127', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184588.670041, 'rx': '632805', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '744588', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184588.670041, 'rx': '5827', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184588.670041, 'rx': '33014', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184588.670041, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184588.670041, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184588.670041, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184588.670041, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184588.670041, 'rx': '22540', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '50', 'ksmPages': 100, 'elapsedTime': '176.67', 'cpuLoad': '0.46', 'cpuSys': '0.22', 'diskStats': {'/var/log': {'free': '47809'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47809'}}, 'cpuUserVdsmd': '1.26', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191812, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:43:08 GMT', 'cpuUser': '0.32', 'memFree': 191556, 'cpuIdle': '99.47', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.67'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:43:08,679+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:43:09,468+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:43:09,468+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:43:09,468+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:43:12,780+0200 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=f156d01d-6217-4697-8ad0-6efe6e98411d (api:48) 2019-04-13 21:43:12,785+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=f156d01d-6217-4697-8ad0-6efe6e98411d (api:52) 2019-04-13 21:43:12,786+0200 ERROR (jsonrpc/5) [storage.TaskManager.Task] (Task='f156d01d-6217-4697-8ad0-6efe6e98411d') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:43:12,786+0200 INFO (jsonrpc/5) [storage.TaskManager.Task] (Task='f156d01d-6217-4697-8ad0-6efe6e98411d') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:43:12,786+0200 ERROR (jsonrpc/5) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:43:12,786+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.00 seconds (__init__:312) 2019-04-13 21:43:12,790+0200 INFO (jsonrpc/3) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=fd24f582-120b-4ffc-9432-a268acc9d48f (api:48) 2019-04-13 21:43:12,790+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getAllTasksStatuses error=Secured object is not in safe state from=::ffff:192.168.0.19,51130, task_id=fd24f582-120b-4ffc-9432-a268acc9d48f (api:52) 2019-04-13 21:43:12,790+0200 ERROR (jsonrpc/3) [storage.TaskManager.Task] (Task='fd24f582-120b-4ffc-9432-a268acc9d48f') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getAllTasksStatuses File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2195, in getAllTasksStatuses allTasksStatus = self._pool.getAllTasksStatuses() File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state 2019-04-13 21:43:12,790+0200 INFO (jsonrpc/3) [storage.TaskManager.Task] (Task='fd24f582-120b-4ffc-9432-a268acc9d48f') aborting: Task is aborted: u'Secured object is not in safe state' - code 100 (task:1181) 2019-04-13 21:43:12,791+0200 ERROR (jsonrpc/3) [storage.Dispatcher] FINISH getAllTasksStatuses error=Secured object is not in safe state (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error SecureError: Secured object is not in safe state 2019-04-13 21:43:12,791+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksStatuses failed (error 654) in 0.00 seconds (__init__:312) 2019-04-13 21:43:12,819+0200 INFO (jsonrpc/6) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,51130, task_id=6378dc3e-e23b-4f2a-a511-69edd5a1bcf6 (api:48) 2019-04-13 21:43:12,820+0200 INFO (jsonrpc/6) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 21:43:13,103+0200 INFO (jsonrpc/6) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:43:13,104+0200 INFO (jsonrpc/6) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 21:43:13,104+0200 INFO (jsonrpc/6) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 21:43:13,270+0200 WARN (jsonrpc/6) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 21:43:13,275+0200 INFO (jsonrpc/6) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 21:43:13,275+0200 INFO (jsonrpc/6) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 21:43:13,275+0200 INFO (jsonrpc/6) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 21:43:13,276+0200 INFO (jsonrpc/6) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 21:43:13,276+0200 INFO (jsonrpc/6) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 21:43:13,276+0200 INFO (jsonrpc/6) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,51130, task_id=6378dc3e-e23b-4f2a-a511-69edd5a1bcf6 (api:54) 2019-04-13 21:43:13,276+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.45 seconds (__init__:312) 2019-04-13 21:43:13,283+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=1a4ef68a-4eb4-4385-b9e9-d0444bbdca07 (api:48) 2019-04-13 21:43:13,288+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=1a4ef68a-4eb4-4385-b9e9-d0444bbdca07 (api:52) 2019-04-13 21:43:13,288+0200 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='1a4ef68a-4eb4-4385-b9e9-d0444bbdca07') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:43:13,288+0200 INFO (jsonrpc/0) [storage.TaskManager.Task] (Task='1a4ef68a-4eb4-4385-b9e9-d0444bbdca07') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:43:13,288+0200 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:43:13,289+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:43:14,979+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:43:14,990+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:43:15,002+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:43:18,154+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:43:18,155+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:43:18,155+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:43:20,780+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=f57d83e9-9ace-4fff-9f9e-2f0b40395e34 (api:48) 2019-04-13 21:43:20,781+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000749996', 'lastCheck': '5.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000576969', 'lastCheck': '5.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000687087', 'lastCheck': '5.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000293354', 'lastCheck': '6.0', 'valid': True}} from=internal, task_id=f57d83e9-9ace-4fff-9f9e-2f0b40395e34 (api:54) 2019-04-13 21:43:23,313+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=b3de44d5-50a1-4065-8e09-5c06205d23ac (api:48) 2019-04-13 21:43:23,319+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=b3de44d5-50a1-4065-8e09-5c06205d23ac (api:52) 2019-04-13 21:43:23,319+0200 ERROR (jsonrpc/4) [storage.TaskManager.Task] (Task='b3de44d5-50a1-4065-8e09-5c06205d23ac') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:43:23,319+0200 INFO (jsonrpc/4) [storage.TaskManager.Task] (Task='b3de44d5-50a1-4065-8e09-5c06205d23ac') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:43:23,320+0200 ERROR (jsonrpc/4) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:43:23,320+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:43:23,341+0200 INFO (jsonrpc/2) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=30780f7c-c3c2-4bf6-9bc9-8b3d2a71d273 (api:48) 2019-04-13 21:43:23,341+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getAllTasksStatuses error=Secured object is not in safe state from=::ffff:192.168.0.19,51130, task_id=30780f7c-c3c2-4bf6-9bc9-8b3d2a71d273 (api:52) 2019-04-13 21:43:23,341+0200 ERROR (jsonrpc/2) [storage.TaskManager.Task] (Task='30780f7c-c3c2-4bf6-9bc9-8b3d2a71d273') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getAllTasksStatuses File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2195, in getAllTasksStatuses allTasksStatus = self._pool.getAllTasksStatuses() File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state 2019-04-13 21:43:23,341+0200 INFO (jsonrpc/2) [storage.TaskManager.Task] (Task='30780f7c-c3c2-4bf6-9bc9-8b3d2a71d273') aborting: Task is aborted: u'Secured object is not in safe state' - code 100 (task:1181) 2019-04-13 21:43:23,341+0200 ERROR (jsonrpc/2) [storage.Dispatcher] FINISH getAllTasksStatuses error=Secured object is not in safe state (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error SecureError: Secured object is not in safe state 2019-04-13 21:43:23,342+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksStatuses failed (error 654) in 0.00 seconds (__init__:312) 2019-04-13 21:43:23,367+0200 INFO (jsonrpc/7) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,51130, task_id=8a8b5618-dcd6-42b4-964c-91adad35d170 (api:48) 2019-04-13 21:43:23,368+0200 INFO (jsonrpc/7) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 21:43:23,643+0200 INFO (jsonrpc/7) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:43:23,643+0200 INFO (jsonrpc/7) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 21:43:23,643+0200 INFO (jsonrpc/7) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 21:43:23,699+0200 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:43:23,721+0200 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=dfad835b-938f-492b-8a12-c68176ea87f9 (api:48) 2019-04-13 21:43:23,722+0200 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000749996', 'lastCheck': '8.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000576969', 'lastCheck': '8.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000687087', 'lastCheck': '8.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000293354', 'lastCheck': '8.9', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=dfad835b-938f-492b-8a12-c68176ea87f9 (api:54) 2019-04-13 21:43:23,723+0200 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=3e683fe4-ea06-41a1-b876-7fe91fea178e (api:48) 2019-04-13 21:43:23,723+0200 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=3e683fe4-ea06-41a1-b876-7fe91fea178e (api:54) 2019-04-13 21:43:23,729+0200 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '0.80', 'nodeIndex': 0, 'cpuSys': '0.73', 'cpuIdle': '98.47'}, '0': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '3': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '2': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '5': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '4': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '7': {'cpuUser': '0.60', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.27'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '9': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.80'}, '8': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '13': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.54'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '17': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '1.60', 'nodeIndex': 1, 'cpuSys': '0.47', 'cpuIdle': '97.93'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95662'}, '0': {'memPercent': 3, 'memFree': '95459'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000749996', 'lastCheck': '8.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000687087', 'lastCheck': '8.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000293354', 'lastCheck': '8.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000576969', 'lastCheck': '8.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184603.721055, 'rx': '6127', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '789901', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184603.721055, 'rx': '669486', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184603.721055, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '773221', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184603.721055, 'rx': '622968', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '31986806', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184603.721055, 'rx': '31986806', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184603.721055, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '12255', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184603.721055, 'rx': '659503', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '777646', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184603.721055, 'rx': '6083', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184603.721055, 'rx': '35382', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184603.721055, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184603.721055, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184603.721055, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184603.721055, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184603.721055, 'rx': '24242', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '50', 'ksmPages': 100, 'elapsedTime': '191.72', 'cpuLoad': '0.46', 'cpuSys': '0.08', 'diskStats': {'/var/log': {'free': '47808'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47808'}}, 'cpuUserVdsmd': '0.80', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191813, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:43:23 GMT', 'cpuUser': '0.16', 'memFree': 191557, 'cpuIdle': '99.77', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.40'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:43:23,731+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:43:23,810+0200 WARN (jsonrpc/7) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 21:43:23,814+0200 INFO (jsonrpc/7) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 21:43:23,815+0200 INFO (jsonrpc/7) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 21:43:23,815+0200 INFO (jsonrpc/7) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 21:43:23,815+0200 INFO (jsonrpc/7) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 21:43:23,815+0200 INFO (jsonrpc/7) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 21:43:23,815+0200 INFO (jsonrpc/7) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,51130, task_id=8a8b5618-dcd6-42b4-964c-91adad35d170 (api:54) 2019-04-13 21:43:23,815+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.45 seconds (__init__:312) 2019-04-13 21:43:23,821+0200 INFO (jsonrpc/3) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=7a6e4198-681d-4b2c-8063-fea25b1adea1 (api:48) 2019-04-13 21:43:23,827+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') from=::ffff:192.168.0.19,51130, task_id=7a6e4198-681d-4b2c-8063-fea25b1adea1 (api:52) 2019-04-13 21:43:23,827+0200 ERROR (jsonrpc/3) [storage.TaskManager.Task] (Task='7a6e4198-681d-4b2c-8063-fea25b1adea1') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getSpmStatus File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 635, in getSpmStatus status = self._getSpmStatusInfo(pool) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 629, in _getSpmStatusInfo (pool.spmRole,) + pool.getSpmStatus())) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 140, in getSpmStatus return self._backend.getSpmStatus() File "/usr/lib/python2.7/site-packages/vdsm/storage/spbackends.py", line 438, in getSpmStatus raise exception.expected(e) TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:43:23,827+0200 INFO (jsonrpc/3) [storage.TaskManager.Task] (Task='7a6e4198-681d-4b2c-8063-fea25b1adea1') aborting: Task is aborted: u"Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')" - code 100 (task:1181) 2019-04-13 21:43:23,827+0200 ERROR (jsonrpc/3) [storage.Dispatcher] FINISH getSpmStatus error=Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error TemporaryFailure: Cannot inquire Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') 2019-04-13 21:43:23,827+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus failed (error 100) in 0.01 seconds (__init__:312) 2019-04-13 21:43:24,476+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:43:24,477+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:43:24,477+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:43:24,993+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:43:25,004+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:43:25,016+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:43:33,174+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:43:33,174+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:43:33,174+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:43:33,858+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=948ad41d-aeb6-48ba-a1b7-16d5d2ca7651 (api:48) 2019-04-13 21:43:33,864+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 26L}} from=::ffff:192.168.0.19,51130, task_id=948ad41d-aeb6-48ba-a1b7-16d5d2ca7651 (api:54) 2019-04-13 21:43:33,864+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:43:33,868+0200 INFO (jsonrpc/4) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=d17f1e10-ad1e-495c-a68c-a214c771b8d1 (api:48) 2019-04-13 21:43:33,868+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getAllTasksStatuses error=Secured object is not in safe state from=::ffff:192.168.0.19,51130, task_id=d17f1e10-ad1e-495c-a68c-a214c771b8d1 (api:52) 2019-04-13 21:43:33,869+0200 ERROR (jsonrpc/4) [storage.TaskManager.Task] (Task='d17f1e10-ad1e-495c-a68c-a214c771b8d1') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "", line 2, in getAllTasksStatuses File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2195, in getAllTasksStatuses allTasksStatus = self._pool.getAllTasksStatuses() File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state 2019-04-13 21:43:33,869+0200 INFO (jsonrpc/4) [storage.TaskManager.Task] (Task='d17f1e10-ad1e-495c-a68c-a214c771b8d1') aborting: Task is aborted: u'Secured object is not in safe state' - code 100 (task:1181) 2019-04-13 21:43:33,869+0200 ERROR (jsonrpc/4) [storage.Dispatcher] FINISH getAllTasksStatuses error=Secured object is not in safe state (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error SecureError: Secured object is not in safe state 2019-04-13 21:43:33,869+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksStatuses failed (error 654) in 0.00 seconds (__init__:312) 2019-04-13 21:43:34,096+0200 INFO (jsonrpc/2) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,51130, task_id=b4aa128e-0993-46b1-92ff-f1947ff3fbd6 (api:48) 2019-04-13 21:43:34,097+0200 INFO (jsonrpc/2) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 21:43:34,359+0200 INFO (jsonrpc/2) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:43:34,360+0200 INFO (jsonrpc/2) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 21:43:34,360+0200 INFO (jsonrpc/2) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 21:43:34,536+0200 WARN (jsonrpc/2) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 21:43:34,541+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 21:43:34,541+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 21:43:34,541+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 21:43:34,541+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 21:43:34,541+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 21:43:34,541+0200 INFO (jsonrpc/2) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,51130, task_id=b4aa128e-0993-46b1-92ff-f1947ff3fbd6 (api:54) 2019-04-13 21:43:34,542+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.45 seconds (__init__:312) 2019-04-13 21:43:34,548+0200 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=4f80f3e1-bc8a-4156-88ea-7a813b6b9d23 (api:48) 2019-04-13 21:43:34,553+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 26L}} from=::ffff:192.168.0.19,51130, task_id=4f80f3e1-bc8a-4156-88ea-7a813b6b9d23 (api:54) 2019-04-13 21:43:34,553+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:43:34,574+0200 INFO (jsonrpc/7) [vdsm.api] START spmStart(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', prevID=-1, prevLVER=u'26', maxHostID=250, domVersion=u'4', options=None) from=::ffff:192.168.0.19,51130, task_id=61a07f8b-47c0-479f-9574-be8a3c47c173 (api:48) 2019-04-13 21:43:34,575+0200 INFO (jsonrpc/7) [vdsm.api] FINISH spmStart return=None from=::ffff:192.168.0.19,51130, task_id=61a07f8b-47c0-479f-9574-be8a3c47c173 (api:54) 2019-04-13 21:43:34,575+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.spmStart succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:43:34,575+0200 INFO (tasks/2) [storage.ThreadPool.WorkerThread] START task 61a07f8b-47c0-479f-9574-be8a3c47c173 (cmd=>, args=None) (threadPool:208) 2019-04-13 21:43:34,581+0200 INFO (tasks/2) [storage.SANLock] Acquiring host id for domain 6a4cde72-6a6a-499e-9b6e-d490d91ca19e (id=1, async=False) (clusterlock:294) 2019-04-13 21:43:34,581+0200 INFO (tasks/2) [storage.SANLock] Host id for domain 6a4cde72-6a6a-499e-9b6e-d490d91ca19e already acquired (id=1, async=False) (clusterlock:322) 2019-04-13 21:43:34,581+0200 INFO (tasks/2) [storage.SANLock] Acquiring Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576) for host id 1 (clusterlock:387) 2019-04-13 21:43:34,658+0200 INFO (tasks/2) [storage.SANLock] Successfully acquired Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576) for host id 1 (clusterlock:425) 2019-04-13 21:43:34,662+0200 INFO (tasks/2) [storage.StorageDomain] Creating vms dir: /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/master/vms (sd:1048) 2019-04-13 21:43:34,672+0200 INFO (tasks/2) [storage.StorageDomain] Creating task dir: /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/master/tasks (sd:1051) 2019-04-13 21:43:34,700+0200 INFO (upgrade/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:43:34,705+0200 INFO (upgrade/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:43:34,710+0200 INFO (upgrade/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:43:34,712+0200 INFO (tasks/2) [storage.StorageDomain] Creating vms dir: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d/master/vms (sd:1048) 2019-04-13 21:43:34,717+0200 INFO (tasks/2) [storage.StorageDomain] Creating task dir: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d/master/tasks (sd:1051) 2019-04-13 21:43:34,723+0200 INFO (tasks/2) [IOProcessClient] (Global) Starting client (__init__:308) 2019-04-13 21:43:34,735+0200 INFO (ioprocess/15905) [IOProcess] (Global) Starting ioprocess (__init__:434) 2019-04-13 21:43:34,792+0200 INFO (monitor/6a4cde7) [storage.Monitor] Host id for domain 6a4cde72-6a6a-499e-9b6e-d490d91ca19e successfully acquired (id: 1) (monitor:448) 2019-04-13 21:43:35,434+0200 INFO (tasks/2) [storage.ThreadPool.WorkerThread] FINISH task 61a07f8b-47c0-479f-9574-be8a3c47c173 (threadPool:210) 2019-04-13 21:43:35,434+0200 INFO (tasks/2) [storage.ThreadPool.WorkerThread] START task 4c004745-e676-4abb-ba5f-8458a81dece5 (cmd=>, args=None) (threadPool:208) 2019-04-13 21:43:35,434+0200 INFO (tasks/2) [storage.ThreadPool.WorkerThread] FINISH task 4c004745-e676-4abb-ba5f-8458a81dece5 (threadPool:210) 2019-04-13 21:43:35,434+0200 INFO (tasks/2) [storage.ThreadPool.WorkerThread] START task a254cbc1-ffb6-447b-99df-98985f28d837 (cmd=>, args=None) (threadPool:208) 2019-04-13 21:43:35,434+0200 INFO (tasks/2) [storage.ThreadPool.WorkerThread] FINISH task a254cbc1-ffb6-447b-99df-98985f28d837 (threadPool:210) 2019-04-13 21:43:35,582+0200 INFO (jsonrpc/3) [vdsm.api] START getTaskStatus(taskID=u'61a07f8b-47c0-479f-9574-be8a3c47c173', spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=6e0e171c-b9a0-4002-89fd-14755e0e3daf (api:48) 2019-04-13 21:43:35,582+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': '61a07f8b-47c0-479f-9574-be8a3c47c173'}} from=::ffff:192.168.0.19,51130, task_id=6e0e171c-b9a0-4002-89fd-14755e0e3daf (api:54) 2019-04-13 21:43:35,582+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:43:35,588+0200 INFO (jsonrpc/6) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=88dcf44b-6c5f-4398-a7ac-fc417e74ffe2 (api:48) 2019-04-13 21:43:35,593+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 27L}} from=::ffff:192.168.0.19,51130, task_id=88dcf44b-6c5f-4398-a7ac-fc417e74ffe2 (api:54) 2019-04-13 21:43:35,593+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:43:35,598+0200 INFO (jsonrpc/0) [vdsm.api] START clearTask(taskID=u'61a07f8b-47c0-479f-9574-be8a3c47c173', spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=c28c63a3-04e8-47ca-aa9a-c40daa75a974 (api:48) 2019-04-13 21:43:35,598+0200 INFO (jsonrpc/0) [vdsm.api] FINISH clearTask return=None from=::ffff:192.168.0.19,51130, task_id=c28c63a3-04e8-47ca-aa9a-c40daa75a974 (api:54) 2019-04-13 21:43:35,598+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Task.clear succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:43:35,624+0200 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::ffff:192.168.0.19:51166 (protocoldetector:61) 2019-04-13 21:43:35,705+0200 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::ffff:192.168.0.19:51166 (protocoldetector:125) 2019-04-13 21:43:35,705+0200 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompserver:95) 2019-04-13 21:43:35,707+0200 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompserver:124) 2019-04-13 21:43:35,747+0200 INFO (jsonrpc/1) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51166, task_id=8a889374-b14a-405f-a043-1bea893f08a6 (api:48) 2019-04-13 21:43:35,755+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 27L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,51166, task_id=8a889374-b14a-405f-a043-1bea893f08a6 (api:54) 2019-04-13 21:43:35,755+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:43:35,817+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=0e8bfe91-51b7-4ee8-81f4-401adb40698c (api:48) 2019-04-13 21:43:35,817+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000487337', 'lastCheck': '0.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000529912', 'lastCheck': '0.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000491061', 'lastCheck': '0.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000303732', 'lastCheck': '1.0', 'valid': True}} from=internal, task_id=0e8bfe91-51b7-4ee8-81f4-401adb40698c (api:54) 2019-04-13 21:43:35,912+0200 INFO (jsonrpc/4) [vdsm.api] START getAllTasksInfo(spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=6db15c38-5b7f-48c1-bbc8-e9e499d9c7ae (api:48) 2019-04-13 21:43:35,912+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getAllTasksInfo return={'allTasksInfo': {'4c004745-e676-4abb-ba5f-8458a81dece5': {'verb': u'deleteImage', 'id': '4c004745-e676-4abb-ba5f-8458a81dece5'}, 'a254cbc1-ffb6-447b-99df-98985f28d837': {'verb': u'copyImage', 'id': 'a254cbc1-ffb6-447b-99df-98985f28d837'}}} from=::ffff:192.168.0.19,51130, task_id=6db15c38-5b7f-48c1-bbc8-e9e499d9c7ae (api:54) 2019-04-13 21:43:35,912+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksInfo succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:43:38,758+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:43:38,779+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=eebd9d9e-e2bd-4515-8fdc-d2776cc00619 (api:48) 2019-04-13 21:43:38,780+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000487337', 'lastCheck': '3.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000529912', 'lastCheck': '3.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000491061', 'lastCheck': '3.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000303732', 'lastCheck': '4.0', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=eebd9d9e-e2bd-4515-8fdc-d2776cc00619 (api:54) 2019-04-13 21:43:38,780+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=259f773b-ff90-4d60-828b-c7989ed752b2 (api:48) 2019-04-13 21:43:38,781+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=259f773b-ff90-4d60-828b-c7989ed752b2 (api:54) 2019-04-13 21:43:38,786+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '21': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '22': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '1': {'cpuUser': '1.80', 'nodeIndex': 0, 'cpuSys': '1.73', 'cpuIdle': '96.47'}, '0': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '3': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '2': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.40'}, '5': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '7': {'cpuUser': '0.53', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.47'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '9': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '8': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '13': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '17': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '2.86', 'nodeIndex': 1, 'cpuSys': '0.53', 'cpuIdle': '96.61'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95659'}, '0': {'memPercent': 3, 'memFree': '95452'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000487337', 'lastCheck': '3.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000491061', 'lastCheck': '3.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000303732', 'lastCheck': '4.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000529912', 'lastCheck': '3.8', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184618.779243, 'rx': '6517', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '853484', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184618.779243, 'rx': '1817324', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184618.779243, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '832374', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184618.779243, 'rx': '1721247', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '45727785', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184618.779243, 'rx': '45727785', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184618.779243, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '12349', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184618.779243, 'rx': '1811023', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '841135', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184618.779243, 'rx': '6301', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184618.779243, 'rx': '38096', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184618.779243, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184618.779243, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184618.779243, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184618.779243, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184618.779243, 'rx': '26036', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '50', 'ksmPages': 100, 'elapsedTime': '206.78', 'cpuLoad': '0.43', 'cpuSys': '0.15', 'diskStats': {'/var/log': {'free': '47808'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47808'}}, 'cpuUserVdsmd': '1.73', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191806, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:43:38 GMT', 'cpuUser': '0.27', 'memFree': 191550, 'cpuIdle': '99.58', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.93'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:43:38,789+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:43:39,488+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:43:39,489+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:43:39,489+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:43:45,014+0200 INFO (monitor/9c3f06c) [storage.Monitor] Host id for domain 9c3f06cf-7475-448e-819b-f4f52fa7d782 successfully acquired (id: 1) (monitor:448) 2019-04-13 21:43:45,935+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=19a97d2e-3a78-4b82-a75b-fe83a3c7dabf (api:48) 2019-04-13 21:43:45,940+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 27L}} from=::ffff:192.168.0.19,51130, task_id=19a97d2e-3a78-4b82-a75b-fe83a3c7dabf (api:54) 2019-04-13 21:43:45,940+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:43:45,953+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51166, task_id=e86e19c1-832e-49d6-85a3-a10575c6fb26 (api:48) 2019-04-13 21:43:45,959+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 27L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,51166, task_id=e86e19c1-832e-49d6-85a3-a10575c6fb26 (api:54) 2019-04-13 21:43:45,960+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:43:48,198+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:43:48,198+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:43:48,199+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:43:50,847+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=4a4ffd76-7b2d-4237-955f-9b2195465cec (api:48) 2019-04-13 21:43:50,847+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00065048', 'lastCheck': '5.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000805174', 'lastCheck': '5.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000773989', 'lastCheck': '5.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000313869', 'lastCheck': '6.0', 'valid': True}} from=internal, task_id=4a4ffd76-7b2d-4237-955f-9b2195465cec (api:54) 2019-04-13 21:43:53,812+0200 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:43:53,833+0200 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=8117794d-d94f-4600-832d-2f5dd349d39f (api:48) 2019-04-13 21:43:53,834+0200 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00065048', 'lastCheck': '8.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000805174', 'lastCheck': '8.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000773989', 'lastCheck': '8.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000313869', 'lastCheck': '9.0', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=8117794d-d94f-4600-832d-2f5dd349d39f (api:54) 2019-04-13 21:43:53,834+0200 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=bc10015b-9352-4dff-8f5d-2fdd5e58771b (api:48) 2019-04-13 21:43:53,835+0200 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=bc10015b-9352-4dff-8f5d-2fdd5e58771b (api:54) 2019-04-13 21:43:53,840+0200 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.80'}, '1': {'cpuUser': '1.00', 'nodeIndex': 0, 'cpuSys': '1.06', 'cpuIdle': '97.94'}, '0': {'cpuUser': '2.40', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '97.20'}, '3': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '5': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '4': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '7': {'cpuUser': '0.27', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.66'}, '6': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '8': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '11': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.80'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '13': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '17': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '0.27', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.46'}, '18': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95653'}, '0': {'memPercent': 3, 'memFree': '95422'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00065048', 'lastCheck': '8.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000773989', 'lastCheck': '8.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000313869', 'lastCheck': '9.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000805174', 'lastCheck': '8.8', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184633.83313, 'rx': '6907', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '897933', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184633.83313, 'rx': '1994094', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184633.83313, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '875440', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184633.83313, 'rx': '1891081', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '57127090', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184633.83313, 'rx': '57127090', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184633.83313, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '33110', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184633.83313, 'rx': '1987537', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '864823', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184633.83313, 'rx': '6557', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184633.83313, 'rx': '40272', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184633.83313, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184633.83313, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184633.83313, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184633.83313, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184633.83313, 'rx': '27646', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '48', 'ksmPages': 100, 'elapsedTime': '221.83', 'cpuLoad': '0.43', 'cpuSys': '0.11', 'diskStats': {'/var/log': {'free': '47807'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47807'}}, 'cpuUserVdsmd': '0.80', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191772, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:43:53 GMT', 'cpuUser': '0.19', 'memFree': 191516, 'cpuIdle': '99.70', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.60'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:43:53,843+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:43:54,497+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:43:54,498+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:43:54,498+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:43:56,008+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=02fefc56-7932-48d6-a98e-20c0054fba19 (api:48) 2019-04-13 21:43:56,013+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 27L}} from=::ffff:192.168.0.19,51130, task_id=02fefc56-7932-48d6-a98e-20c0054fba19 (api:54) 2019-04-13 21:43:56,013+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:43:56,025+0200 INFO (jsonrpc/2) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51166, task_id=a77545cf-befe-4f60-ac63-ef68010a3e0f (api:48) 2019-04-13 21:43:56,031+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 27L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,51166, task_id=a77545cf-befe-4f60-ac63-ef68010a3e0f (api:54) 2019-04-13 21:43:56,032+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:44:03,221+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:44:03,221+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:44:03,221+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:44:05,882+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=d02f1626-ced7-4728-95b9-c4c311097185 (api:48) 2019-04-13 21:44:05,883+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000673564', 'lastCheck': '0.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000746494', 'lastCheck': '0.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00055664', 'lastCheck': '0.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000485718', 'lastCheck': '1.1', 'valid': True}} from=internal, task_id=d02f1626-ced7-4728-95b9-c4c311097185 (api:54) 2019-04-13 21:44:06,077+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=8fdb02b4-307b-46e4-9423-44c291cf973b (api:48) 2019-04-13 21:44:06,083+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 27L}} from=::ffff:192.168.0.19,51130, task_id=8fdb02b4-307b-46e4-9423-44c291cf973b (api:54) 2019-04-13 21:44:06,083+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:44:06,095+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51166, task_id=9ff74dfb-9568-4bbc-809e-08ae0cc13c32 (api:48) 2019-04-13 21:44:06,102+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 27L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,51166, task_id=9ff74dfb-9568-4bbc-809e-08ae0cc13c32 (api:54) 2019-04-13 21:44:06,102+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:44:08,859+0200 INFO (jsonrpc/6) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:44:08,881+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=490c7a22-5c4c-42af-89eb-f894522384b7 (api:48) 2019-04-13 21:44:08,882+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000673564', 'lastCheck': '3.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000746494', 'lastCheck': '3.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00055664', 'lastCheck': '3.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000485718', 'lastCheck': '4.1', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=490c7a22-5c4c-42af-89eb-f894522384b7 (api:54) 2019-04-13 21:44:08,883+0200 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=659fa487-f79a-4f52-ab3c-0b9b53c67def (api:48) 2019-04-13 21:44:08,883+0200 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=659fa487-f79a-4f52-ab3c-0b9b53c67def (api:54) 2019-04-13 21:44:08,889+0200 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '1': {'cpuUser': '2.26', 'nodeIndex': 0, 'cpuSys': '1.40', 'cpuIdle': '96.34'}, '0': {'cpuUser': '31.41', 'nodeIndex': 0, 'cpuSys': '2.26', 'cpuIdle': '66.33'}, '3': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.67'}, '2': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.66'}, '5': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '4': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '7': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.80'}, '6': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '8': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '13': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '12': {'cpuUser': '1.06', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '98.87'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '17': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.87'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95654'}, '0': {'memPercent': 4, 'memFree': '95180'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000673564', 'lastCheck': '3.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00055664', 'lastCheck': '3.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000485718', 'lastCheck': '4.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000746494', 'lastCheck': '3.9', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184648.880879, 'rx': '7297', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '939856', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184648.880879, 'rx': '3106366', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184648.880879, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '915730', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184648.880879, 'rx': '2955029', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '69611275', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184648.880879, 'rx': '69611275', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184648.880879, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '49606', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184648.880879, 'rx': '3099591', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '890250', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184648.880879, 'rx': '6775', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184648.880879, 'rx': '42986', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184648.880879, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184648.880879, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184648.880879, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184648.880879, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184648.880879, 'rx': '29440', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '70', 'ksmPages': 100, 'elapsedTime': '236.88', 'cpuLoad': '0.42', 'cpuSys': '0.18', 'diskStats': {'/var/log': {'free': '47806'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47806'}}, 'cpuUserVdsmd': '1.40', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191660, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:44:08 GMT', 'cpuUser': '1.52', 'memFree': 191404, 'cpuIdle': '98.31', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.53'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:44:08,891+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:44:09,506+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:44:09,506+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:44:09,506+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:44:16,154+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=22a7c189-4132-4e18-b20f-a03aee8be35f (api:48) 2019-04-13 21:44:16,159+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 27L}} from=::ffff:192.168.0.19,51130, task_id=22a7c189-4132-4e18-b20f-a03aee8be35f (api:54) 2019-04-13 21:44:16,159+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:44:16,170+0200 INFO (jsonrpc/4) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51166, task_id=9e77957c-f261-476b-8847-b0284b81caf7 (api:48) 2019-04-13 21:44:16,178+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 27L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,51166, task_id=9e77957c-f261-476b-8847-b0284b81caf7 (api:54) 2019-04-13 21:44:16,178+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:44:18,244+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:44:18,244+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:44:18,245+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:44:20,916+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=a189f30c-7740-4a55-b62a-cfa7fc684d34 (api:48) 2019-04-13 21:44:20,917+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000714117', 'lastCheck': '5.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000722064', 'lastCheck': '5.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000561787', 'lastCheck': '5.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000337138', 'lastCheck': '6.1', 'valid': True}} from=internal, task_id=a189f30c-7740-4a55-b62a-cfa7fc684d34 (api:54) 2019-04-13 21:44:23,903+0200 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:44:23,924+0200 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=303811bf-e235-4012-8438-562c91244709 (api:48) 2019-04-13 21:44:23,925+0200 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000714117', 'lastCheck': '8.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000722064', 'lastCheck': '8.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000561787', 'lastCheck': '8.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000337138', 'lastCheck': '9.1', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=303811bf-e235-4012-8438-562c91244709 (api:54) 2019-04-13 21:44:23,925+0200 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=1d17afc4-d998-4073-b31e-0db228054c30 (api:48) 2019-04-13 21:44:23,926+0200 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=1d17afc4-d998-4073-b31e-0db228054c30 (api:54) 2019-04-13 21:44:23,931+0200 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '1.99', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '97.68'}, '21': {'cpuUser': '2.19', 'nodeIndex': 1, 'cpuSys': '0.73', 'cpuIdle': '97.08'}, '22': {'cpuUser': '0.80', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '98.87'}, '23': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '1': {'cpuUser': '1.33', 'nodeIndex': 0, 'cpuSys': '1.20', 'cpuIdle': '97.47'}, '0': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '3': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.60'}, '2': {'cpuUser': '0.73', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.14'}, '5': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '7': {'cpuUser': '0.40', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.53'}, '6': {'cpuUser': '0.53', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.27'}, '9': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.40'}, '8': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.80'}, '11': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '98.47'}, '10': {'cpuUser': '1.33', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '98.34'}, '13': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '12': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '15': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '14': {'cpuUser': '0.86', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '98.87'}, '17': {'cpuUser': '0.86', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '98.81'}, '16': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.54'}, '19': {'cpuUser': '0.40', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.40'}, '18': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95595'}, '0': {'memPercent': 4, 'memFree': '95165'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '2', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000714117', 'lastCheck': '8.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000561787', 'lastCheck': '8.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000337138', 'lastCheck': '9.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000722064', 'lastCheck': '8.9', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184663.924103, 'rx': '7687', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '2128464', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184663.924103, 'rx': '47127103', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184663.924103, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '2040230', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184663.924103, 'rx': '46348838', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '78930294', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184663.924103, 'rx': '78930294', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184663.924103, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '918352', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184663.924103, 'rx': '47119880', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1210112', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184663.924103, 'rx': '7223', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184663.924103, 'rx': '45610', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184663.924103, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184663.924103, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184663.924103, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184663.924103, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184663.924103, 'rx': '31326', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '70', 'ksmPages': 100, 'elapsedTime': '251.92', 'cpuLoad': '0.40', 'cpuSys': '0.22', 'diskStats': {'/var/log': {'free': '47773'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47773'}}, 'cpuUserVdsmd': '1.26', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191634, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:44:23 GMT', 'cpuUser': '0.60', 'memFree': 191378, 'cpuIdle': '99.18', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.66'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:44:23,934+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:44:24,516+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:44:24,517+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:44:24,517+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:44:26,228+0200 INFO (jsonrpc/3) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=f9119d43-bf3f-4da5-9ef5-c8c932876f4d (api:48) 2019-04-13 21:44:26,234+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 27L}} from=::ffff:192.168.0.19,51130, task_id=f9119d43-bf3f-4da5-9ef5-c8c932876f4d (api:54) 2019-04-13 21:44:26,234+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:44:26,247+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51166, task_id=be441290-15c7-41e1-bb51-2226970e856b (api:48) 2019-04-13 21:44:26,254+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 27L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,51166, task_id=be441290-15c7-41e1-bb51-2226970e856b (api:54) 2019-04-13 21:44:26,255+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:44:33,267+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:44:33,268+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:44:33,268+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:44:34,820+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_isos) Closing client (__init__:584) 2019-04-13 21:44:34,821+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/glusterSD/onode-2.discovery.intern:vol-node-2) Closing client (__init__:584) 2019-04-13 21:44:34,821+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_vms) Closing client (__init__:584) 2019-04-13 21:44:34,821+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/onode-1.discovery.intern:_mnt_store) Closing client (__init__:584) 2019-04-13 21:44:34,821+0200 INFO (monitor/6a4cde7) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Closing client (__init__:584) 2019-04-13 21:44:35,956+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=2f510b7d-91c5-4df3-a698-e5ec867c5a1c (api:48) 2019-04-13 21:44:35,956+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000848726', 'lastCheck': '0.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000767452', 'lastCheck': '0.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000627208', 'lastCheck': '0.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000457399', 'lastCheck': '1.1', 'valid': True}} from=internal, task_id=2f510b7d-91c5-4df3-a698-e5ec867c5a1c (api:54) 2019-04-13 21:44:36,296+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=c8f9e46f-bff0-4ead-a860-b7a2f8a724f7 (api:48) 2019-04-13 21:44:36,301+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 27L}} from=::ffff:192.168.0.19,51130, task_id=c8f9e46f-bff0-4ead-a860-b7a2f8a724f7 (api:54) 2019-04-13 21:44:36,302+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:44:36,314+0200 INFO (jsonrpc/4) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51166, task_id=5bf59bc3-029a-420d-b2c3-97b69df27ec8 (api:48) 2019-04-13 21:44:36,322+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 27L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,51166, task_id=5bf59bc3-029a-420d-b2c3-97b69df27ec8 (api:54) 2019-04-13 21:44:36,322+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:44:38,951+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:44:38,975+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=6de61825-8dd1-4186-a8d4-32392fb762a9 (api:48) 2019-04-13 21:44:38,976+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000848726', 'lastCheck': '3.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000767452', 'lastCheck': '4.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000627208', 'lastCheck': '3.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000457399', 'lastCheck': '4.1', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=6de61825-8dd1-4186-a8d4-32392fb762a9 (api:54) 2019-04-13 21:44:38,977+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=7e27ba7f-9de6-4688-bccf-6564ddcf71b4 (api:48) 2019-04-13 21:44:38,977+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=7e27ba7f-9de6-4688-bccf-6564ddcf71b4 (api:54) 2019-04-13 21:44:38,984+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.40', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.47'}, '21': {'cpuUser': '13.57', 'nodeIndex': 1, 'cpuSys': '2.26', 'cpuIdle': '84.17'}, '22': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.87'}, '23': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.80'}, '1': {'cpuUser': '4.06', 'nodeIndex': 0, 'cpuSys': '1.86', 'cpuIdle': '94.08'}, '0': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.54'}, '3': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '2': {'cpuUser': '22.29', 'nodeIndex': 0, 'cpuSys': '1.06', 'cpuIdle': '76.65'}, '5': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.80'}, '4': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.80'}, '7': {'cpuUser': '2.73', 'nodeIndex': 1, 'cpuSys': '0.73', 'cpuIdle': '96.54'}, '6': {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.40'}, '9': {'cpuUser': '1.86', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '97.81'}, '8': {'cpuUser': '0.60', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.13'}, '11': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '98.54'}, '10': {'cpuUser': '2.06', 'nodeIndex': 1, 'cpuSys': '0.47', 'cpuIdle': '97.47'}, '13': {'cpuUser': '7.72', 'nodeIndex': 0, 'cpuSys': '0.73', 'cpuIdle': '91.55'}, '12': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.60'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.67'}, '17': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '16': {'cpuUser': '1.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '98.47'}, '19': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.67'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95571'}, '0': {'memPercent': 4, 'memFree': '94944'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '2', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000848726', 'lastCheck': '3.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000627208', 'lastCheck': '3.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000457399', 'lastCheck': '4.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000767452', 'lastCheck': '4.0', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184678.974327, 'rx': '8077', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '2223658', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184678.974327, 'rx': '48246827', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184678.974327, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '2142094', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184678.974327, 'rx': '47418610', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '92443945', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184678.974327, 'rx': '92443945', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184678.974327, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '986210', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184678.974327, 'rx': '48239386', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1237448', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184678.974327, 'rx': '7441', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184678.974327, 'rx': '48004', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184678.974327, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184678.974327, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184678.974327, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184678.974327, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184678.974327, 'rx': '32890', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '78', 'ksmPages': 100, 'elapsedTime': '266.97', 'cpuLoad': '0.42', 'cpuSys': '0.39', 'diskStats': {'/var/log': {'free': '47444'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47444'}}, 'cpuUserVdsmd': '1.33', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191629, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:44:38 GMT', 'cpuUser': '2.50', 'memFree': 191373, 'cpuIdle': '97.11', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.67'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:44:38,987+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.04 seconds (__init__:312) 2019-04-13 21:44:39,526+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:44:39,527+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:44:39,527+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:44:43,852+0200 INFO (jsonrpc/7) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=e1eddf67-27f4-40a3-b00d-29a74ce2507a (api:48) 2019-04-13 21:44:43,852+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getAllTasksStatuses return={'allTasksStatus': {'4c004745-e676-4abb-ba5f-8458a81dece5': {'code': 0, 'message': u'1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': '4c004745-e676-4abb-ba5f-8458a81dece5'}, 'a254cbc1-ffb6-447b-99df-98985f28d837': {'code': 0, 'message': u'1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'a254cbc1-ffb6-447b-99df-98985f28d837'}}} from=::ffff:192.168.0.19,51130, task_id=e1eddf67-27f4-40a3-b00d-29a74ce2507a (api:54) 2019-04-13 21:44:43,853+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksStatuses succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:44:43,859+0200 INFO (jsonrpc/3) [vdsm.api] START clearTask(taskID=u'a254cbc1-ffb6-447b-99df-98985f28d837', spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=1dd729b1-ee2c-4a22-8bd5-64b35c99148d (api:48) 2019-04-13 21:44:43,918+0200 INFO (jsonrpc/3) [vdsm.api] FINISH clearTask return=None from=::ffff:192.168.0.19,51130, task_id=1dd729b1-ee2c-4a22-8bd5-64b35c99148d (api:54) 2019-04-13 21:44:43,918+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Task.clear succeeded in 0.06 seconds (__init__:312) 2019-04-13 21:44:43,932+0200 INFO (jsonrpc/6) [vdsm.api] START clearTask(taskID=u'4c004745-e676-4abb-ba5f-8458a81dece5', spUUID=None, options=None) from=::ffff:192.168.0.19,51130, task_id=9bf4f529-eb7e-491e-ba45-7c446b90987c (api:48) 2019-04-13 21:44:43,968+0200 INFO (jsonrpc/6) [vdsm.api] FINISH clearTask return=None from=::ffff:192.168.0.19,51130, task_id=9bf4f529-eb7e-491e-ba45-7c446b90987c (api:54) 2019-04-13 21:44:43,968+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Task.clear succeeded in 0.04 seconds (__init__:312) 2019-04-13 21:44:46,380+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=8920762e-4491-4636-a4b8-70a2a0789b41 (api:48) 2019-04-13 21:44:46,386+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 27L}} from=::ffff:192.168.0.19,51130, task_id=8920762e-4491-4636-a4b8-70a2a0789b41 (api:54) 2019-04-13 21:44:46,386+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:44:46,399+0200 INFO (jsonrpc/1) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51166, task_id=e4e260b9-43ff-4b8a-982a-f8450a8e7183 (api:48) 2019-04-13 21:44:46,407+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 27L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,51166, task_id=e4e260b9-43ff-4b8a-982a-f8450a8e7183 (api:54) 2019-04-13 21:44:46,408+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:44:48,290+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:44:48,290+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:44:48,291+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:44:48,543+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.fenceNode succeeded in 0.47 seconds (__init__:312) 2019-04-13 21:44:50,988+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=50bc591f-5a40-4587-ae4d-c8364e558004 (api:48) 2019-04-13 21:44:50,988+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000813478', 'lastCheck': '6.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000804432', 'lastCheck': '6.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000647855', 'lastCheck': '5.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000326159', 'lastCheck': '6.2', 'valid': True}} from=internal, task_id=50bc591f-5a40-4587-ae4d-c8364e558004 (api:54) 2019-04-13 21:44:53,040+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.fenceNode succeeded in 4.39 seconds (__init__:312) 2019-04-13 21:44:54,011+0200 INFO (jsonrpc/7) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:44:54,033+0200 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=149811a8-cf87-428d-86f5-d41b95ddd6fe (api:48) 2019-04-13 21:44:54,033+0200 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000813478', 'lastCheck': '9.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000804432', 'lastCheck': '9.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000647855', 'lastCheck': '9.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000326159', 'lastCheck': '9.2', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=149811a8-cf87-428d-86f5-d41b95ddd6fe (api:54) 2019-04-13 21:44:54,034+0200 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=75737946-33fb-4589-a876-221a0d7330ad (api:48) 2019-04-13 21:44:54,034+0200 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=75737946-33fb-4589-a876-221a0d7330ad (api:54) 2019-04-13 21:44:54,040+0200 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.93', 'nodeIndex': 1, 'cpuSys': '2.79', 'cpuIdle': '96.28'}, '21': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '23': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.73'}, '1': {'cpuUser': '7.85', 'nodeIndex': 0, 'cpuSys': '2.26', 'cpuIdle': '89.89'}, '0': {'cpuUser': '2.79', 'nodeIndex': 0, 'cpuSys': '1.60', 'cpuIdle': '95.61'}, '3': {'cpuUser': '5.59', 'nodeIndex': 0, 'cpuSys': '1.06', 'cpuIdle': '93.35'}, '2': {'cpuUser': '6.05', 'nodeIndex': 0, 'cpuSys': '1.06', 'cpuIdle': '92.89'}, '5': {'cpuUser': '2.39', 'nodeIndex': 0, 'cpuSys': '0.53', 'cpuIdle': '97.08'}, '4': {'cpuUser': '5.65', 'nodeIndex': 0, 'cpuSys': '1.26', 'cpuIdle': '93.09'}, '7': {'cpuUser': '0.80', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.00'}, '6': {'cpuUser': '0.80', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.07'}, '9': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.33'}, '8': {'cpuUser': '0.27', 'nodeIndex': 1, 'cpuSys': '2.93', 'cpuIdle': '96.80'}, '11': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.33'}, '10': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.80'}, '13': {'cpuUser': '5.52', 'nodeIndex': 0, 'cpuSys': '2.46', 'cpuIdle': '92.02'}, '12': {'cpuUser': '1.80', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '97.80'}, '15': {'cpuUser': '0.80', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.07'}, '14': {'cpuUser': '1.26', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '98.47'}, '17': {'cpuUser': '1.86', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '97.87'}, '16': {'cpuUser': '4.79', 'nodeIndex': 0, 'cpuSys': '0.80', 'cpuIdle': '94.41'}, '19': {'cpuUser': '2.26', 'nodeIndex': 1, 'cpuSys': '1.46', 'cpuIdle': '96.28'}, '18': {'cpuUser': '0.60', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.33'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95572'}, '0': {'memPercent': 4, 'memFree': '94961'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '2', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000813478', 'lastCheck': '9.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000647855', 'lastCheck': '9.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000326159', 'lastCheck': '9.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000804432', 'lastCheck': '9.0', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184694.032473, 'rx': '8467', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '2350770', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184694.032473, 'rx': '48270174', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184694.032473, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '2258230', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184694.032473, 'rx': '47437135', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '102827900', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184694.032473, 'rx': '102827900', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184694.032473, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '1092726', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184694.032473, 'rx': '48258788', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1258044', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184694.032473, 'rx': '11386', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184694.032473, 'rx': '50308', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184694.032473, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184694.032473, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184694.032473, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184694.032473, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184694.032473, 'rx': '34500', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '74', 'ksmPages': 100, 'elapsedTime': '282.03', 'cpuLoad': '0.46', 'cpuSys': '0.85', 'diskStats': {'/var/log': {'free': '47529'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47529'}}, 'cpuUserVdsmd': '1.46', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191603, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:44:54 GMT', 'cpuUser': '2.23', 'memFree': 191347, 'cpuIdle': '96.92', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.80'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:44:54,042+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:44:54,541+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:44:54,541+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:44:54,541+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:44:56,456+0200 INFO (jsonrpc/6) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=72bebcbd-d61d-4cc8-b2af-ac3abe540472 (api:48) 2019-04-13 21:44:56,462+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 27L}} from=::ffff:192.168.0.19,51130, task_id=72bebcbd-d61d-4cc8-b2af-ac3abe540472 (api:54) 2019-04-13 21:44:56,462+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:44:56,474+0200 INFO (jsonrpc/0) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51166, task_id=2f158ede-6619-41c1-b099-7a3ab38938ec (api:48) 2019-04-13 21:44:56,482+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 27L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,51166, task_id=2f158ede-6619-41c1-b099-7a3ab38938ec (api:54) 2019-04-13 21:44:56,482+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:44:58,656+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.fenceNode succeeded in 0.48 seconds (__init__:312) 2019-04-13 21:45:03,301+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:45:03,301+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:45:03,301+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:45:06,016+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=8aedc210-a287-4f10-b3d1-698c7812a274 (api:48) 2019-04-13 21:45:06,017+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000838547', 'lastCheck': '1.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00105209', 'lastCheck': '1.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000818302', 'lastCheck': '1.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000847891', 'lastCheck': '1.2', 'valid': True}} from=internal, task_id=8aedc210-a287-4f10-b3d1-698c7812a274 (api:54) 2019-04-13 21:45:06,750+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=2b3a67fe-a0a0-4240-8270-3ed8c94308fc (api:48) 2019-04-13 21:45:06,756+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 27L}} from=::ffff:192.168.0.19,51130, task_id=2b3a67fe-a0a0-4240-8270-3ed8c94308fc (api:54) 2019-04-13 21:45:06,756+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:45:06,772+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51166, task_id=a17fade7-a170-44fe-a3d6-5aadbec9798a (api:48) 2019-04-13 21:45:06,780+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 27L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,51166, task_id=a17fade7-a170-44fe-a3d6-5aadbec9798a (api:54) 2019-04-13 21:45:06,780+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:45:09,060+0200 INFO (jsonrpc/7) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:45:09,082+0200 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=2bf0620b-720f-4c2f-aadc-77b86b987fc6 (api:48) 2019-04-13 21:45:09,082+0200 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000838547', 'lastCheck': '4.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00105209', 'lastCheck': '4.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000818302', 'lastCheck': '4.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000847891', 'lastCheck': '4.2', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=2bf0620b-720f-4c2f-aadc-77b86b987fc6 (api:54) 2019-04-13 21:45:09,083+0200 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=1116fbf6-f9c6-403f-85fd-0ac85803508e (api:48) 2019-04-13 21:45:09,083+0200 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=1116fbf6-f9c6-403f-85fd-0ac85803508e (api:54) 2019-04-13 21:45:09,090+0200 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.60'}, '21': {'cpuUser': '1.06', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '98.67'}, '22': {'cpuUser': '0.73', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.14'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '4.72', 'nodeIndex': 0, 'cpuSys': '2.06', 'cpuIdle': '93.22'}, '0': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.80'}, '3': {'cpuUser': '6.92', 'nodeIndex': 0, 'cpuSys': '1.53', 'cpuIdle': '91.55'}, '2': {'cpuUser': '2.99', 'nodeIndex': 0, 'cpuSys': '0.60', 'cpuIdle': '96.41'}, '5': {'cpuUser': '0.40', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '99.13'}, '4': {'cpuUser': '1.46', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '98.27'}, '7': {'cpuUser': '4.39', 'nodeIndex': 1, 'cpuSys': '0.87', 'cpuIdle': '94.74'}, '6': {'cpuUser': '2.73', 'nodeIndex': 1, 'cpuSys': '0.60', 'cpuIdle': '96.67'}, '9': {'cpuUser': '3.73', 'nodeIndex': 1, 'cpuSys': '0.87', 'cpuIdle': '95.40'}, '8': {'cpuUser': '4.39', 'nodeIndex': 1, 'cpuSys': '1.13', 'cpuIdle': '94.48'}, '11': {'cpuUser': '4.39', 'nodeIndex': 1, 'cpuSys': '0.73', 'cpuIdle': '94.88'}, '10': {'cpuUser': '5.12', 'nodeIndex': 1, 'cpuSys': '0.87', 'cpuIdle': '94.01'}, '13': {'cpuUser': '0.40', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.40'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '14': {'cpuUser': '2.26', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '97.27'}, '17': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '16': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.80'}, '19': {'cpuUser': '2.93', 'nodeIndex': 1, 'cpuSys': '0.87', 'cpuIdle': '96.20'}, '18': {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.60'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95530'}, '0': {'memPercent': 4, 'memFree': '94919'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '2', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000838547', 'lastCheck': '4.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000818302', 'lastCheck': '4.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000847891', 'lastCheck': '4.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00105209', 'lastCheck': '4.1', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184709.08139, 'rx': '8857', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '2393739', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184709.08139, 'rx': '49385061', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184709.08139, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '2299189', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184709.08139, 'rx': '48502647', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '116341005', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184709.08139, 'rx': '116341005', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184709.08139, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '1109622', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184709.08139, 'rx': '49372340', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1284117', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184709.08139, 'rx': '12721', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184709.08139, 'rx': '52830', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184709.08139, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184709.08139, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184709.08139, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184709.08139, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184709.08139, 'rx': '36156', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '74', 'ksmPages': 100, 'elapsedTime': '297.08', 'cpuLoad': '0.52', 'cpuSys': '0.53', 'diskStats': {'/var/log': {'free': '47466'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47466'}}, 'cpuUserVdsmd': '1.33', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191584, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:45:09 GMT', 'cpuUser': '2.07', 'memFree': 191328, 'cpuIdle': '97.41', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.80'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:45:09,092+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:45:09,550+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:45:09,551+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:45:09,551+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:45:16,834+0200 INFO (jsonrpc/6) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=424fc3bd-b808-4c6c-bf8c-603bcafa1711 (api:48) 2019-04-13 21:45:16,840+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 27L}} from=::ffff:192.168.0.19,51130, task_id=424fc3bd-b808-4c6c-bf8c-603bcafa1711 (api:54) 2019-04-13 21:45:16,840+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:45:16,854+0200 INFO (jsonrpc/0) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51166, task_id=09446a00-305f-4482-9235-ecc5e2a15960 (api:48) 2019-04-13 21:45:16,862+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 27L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,51166, task_id=09446a00-305f-4482-9235-ecc5e2a15960 (api:54) 2019-04-13 21:45:16,862+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:45:18,324+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:45:18,324+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:45:18,325+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:45:21,048+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=219d272c-1968-460d-abbd-28e2361acb12 (api:48) 2019-04-13 21:45:21,049+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000679555', 'lastCheck': '6.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000822975', 'lastCheck': '6.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000745667', 'lastCheck': '6.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000425052', 'lastCheck': '6.2', 'valid': True}} from=internal, task_id=219d272c-1968-460d-abbd-28e2361acb12 (api:54) 2019-04-13 21:45:24,108+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:45:24,130+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=21428c0f-016f-400f-ba7e-07289df08be9 (api:48) 2019-04-13 21:45:24,131+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000679555', 'lastCheck': '9.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000822975', 'lastCheck': '9.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000745667', 'lastCheck': '9.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000425052', 'lastCheck': '9.3', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=21428c0f-016f-400f-ba7e-07289df08be9 (api:54) 2019-04-13 21:45:24,132+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=af5a8b16-d718-421d-a09f-7545b61de067 (api:48) 2019-04-13 21:45:24,132+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=af5a8b16-d718-421d-a09f-7545b61de067 (api:54) 2019-04-13 21:45:24,138+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '1.00', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '98.80'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '22': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '1': {'cpuUser': '1.40', 'nodeIndex': 0, 'cpuSys': '1.33', 'cpuIdle': '97.27'}, '0': {'cpuUser': '14.77', 'nodeIndex': 0, 'cpuSys': '2.46', 'cpuIdle': '82.77'}, '3': {'cpuUser': '0.40', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.40'}, '2': {'cpuUser': '0.93', 'nodeIndex': 0, 'cpuSys': '1.00', 'cpuIdle': '98.07'}, '5': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.60'}, '4': {'cpuUser': '0.86', 'nodeIndex': 0, 'cpuSys': '0.73', 'cpuIdle': '98.41'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '6': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '9': {'cpuUser': '0.53', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.27'}, '8': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '13': {'cpuUser': '1.20', 'nodeIndex': 0, 'cpuSys': '1.53', 'cpuIdle': '97.27'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '17': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '16': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.47'}, '19': {'cpuUser': '0.53', 'nodeIndex': 1, 'cpuSys': '1.26', 'cpuIdle': '98.21'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95602'}, '0': {'memPercent': 4, 'memFree': '94858'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '2', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000679555', 'lastCheck': '9.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000745667', 'lastCheck': '9.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000425052', 'lastCheck': '9.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000822975', 'lastCheck': '9.1', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184724.130058, 'rx': '9247', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '2425279', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184724.130058, 'rx': '50494741', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184724.130058, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '2329551', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184724.130058, 'rx': '49563589', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '125660024', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184724.130058, 'rx': '125660024', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184724.130058, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '1120880', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184724.130058, 'rx': '50481204', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1304399', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184724.130058, 'rx': '13537', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184724.130058, 'rx': '55390', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184724.130058, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184724.130058, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184724.130058, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184724.130058, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184724.130058, 'rx': '37858', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '82', 'ksmPages': 100, 'elapsedTime': '312.13', 'cpuLoad': '0.56', 'cpuSys': '0.42', 'diskStats': {'/var/log': {'free': '47572'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47572'}}, 'cpuUserVdsmd': '1.26', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191540, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:45:24 GMT', 'cpuUser': '0.93', 'memFree': 191284, 'cpuIdle': '98.65', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.67'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:45:24,141+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:45:24,558+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:45:24,559+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:45:24,559+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:45:26,908+0200 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=e2229cc3-4c2d-49b6-a840-341db14b3682 (api:48) 2019-04-13 21:45:26,914+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 27L}} from=::ffff:192.168.0.19,51130, task_id=e2229cc3-4c2d-49b6-a840-341db14b3682 (api:54) 2019-04-13 21:45:26,914+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:45:26,934+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51166, task_id=66cbb8fa-aa0d-40b4-b348-712e568d164f (api:48) 2019-04-13 21:45:26,942+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 27L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,51166, task_id=66cbb8fa-aa0d-40b4-b348-712e568d164f (api:54) 2019-04-13 21:45:26,943+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:45:33,347+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:45:33,347+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:45:33,347+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:45:36,091+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=5439166b-f2c6-4daf-b369-d0437b1df94c (api:48) 2019-04-13 21:45:36,091+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000518057', 'lastCheck': '1.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000850973', 'lastCheck': '1.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000702469', 'lastCheck': '1.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000349882', 'lastCheck': '1.2', 'valid': True}} from=internal, task_id=5439166b-f2c6-4daf-b369-d0437b1df94c (api:54) 2019-04-13 21:45:36,984+0200 INFO (jsonrpc/6) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=4a399958-40be-4d24-9eb2-79ae7305ddef (api:48) 2019-04-13 21:45:36,990+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 27L}} from=::ffff:192.168.0.19,51130, task_id=4a399958-40be-4d24-9eb2-79ae7305ddef (api:54) 2019-04-13 21:45:36,990+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:45:36,999+0200 INFO (jsonrpc/0) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51166, task_id=035553b6-7dea-4449-8930-0e26a6d89635 (api:48) 2019-04-13 21:45:37,007+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 27L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,51166, task_id=035553b6-7dea-4449-8930-0e26a6d89635 (api:54) 2019-04-13 21:45:37,007+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:45:39,151+0200 INFO (jsonrpc/1) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:45:39,173+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=73059231-e6b4-4c1f-8bb3-3200cbc774f2 (api:48) 2019-04-13 21:45:39,173+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000518057', 'lastCheck': '4.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000850973', 'lastCheck': '4.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000702469', 'lastCheck': '4.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000349882', 'lastCheck': '4.3', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=73059231-e6b4-4c1f-8bb3-3200cbc774f2 (api:54) 2019-04-13 21:45:39,174+0200 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=64b4bb51-619b-4d02-9cc9-3603c5f1547a (api:48) 2019-04-13 21:45:39,174+0200 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=64b4bb51-619b-4d02-9cc9-3603c5f1547a (api:54) 2019-04-13 21:45:39,180+0200 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.87'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '2.59', 'nodeIndex': 0, 'cpuSys': '1.73', 'cpuIdle': '95.68'}, '0': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.60', 'cpuIdle': '99.40'}, '3': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '99.26'}, '2': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '1.00', 'cpuIdle': '98.67'}, '5': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '9': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '8': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '13': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.73', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.14'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '17': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '0.27', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.53'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95623'}, '0': {'memPercent': 4, 'memFree': '95030'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000518057', 'lastCheck': '4.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000702469', 'lastCheck': '4.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000349882', 'lastCheck': '4.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000850973', 'lastCheck': '4.1', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184739.172298, 'rx': '9637', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '2463051', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184739.172298, 'rx': '51609581', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184739.172298, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '2366349', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184739.172298, 'rx': '50628919', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '138145191', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184739.172298, 'rx': '138145191', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184739.172298, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '1131856', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184739.172298, 'rx': '51594638', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1331195', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184739.172298, 'rx': '14943', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184739.172298, 'rx': '57720', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184739.172298, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184739.172298, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184739.172298, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184739.172298, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184739.172298, 'rx': '39468', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '44', 'ksmPages': 100, 'elapsedTime': '327.17', 'cpuLoad': '0.58', 'cpuSys': '0.20', 'diskStats': {'/var/log': {'free': '47693'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47693'}}, 'cpuUserVdsmd': '1.40', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191737, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:45:39 GMT', 'cpuUser': '0.19', 'memFree': 191481, 'cpuIdle': '99.62', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.60'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:45:39,183+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:45:39,566+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:45:39,566+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:45:39,566+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:45:44,882+0200 INFO (itmap/0) [IOProcessClient] (Global) Closing client (__init__:584) 2019-04-13 21:45:44,883+0200 INFO (itmap/0) [IOProcessClient] (/onode-1.discovery.intern:_mnt_store) Starting client (__init__:308) 2019-04-13 21:45:44,895+0200 INFO (ioprocess/16512) [IOProcess] (/onode-1.discovery.intern:_mnt_store) Starting ioprocess (__init__:434) 2019-04-13 21:45:44,896+0200 INFO (itmap/1) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_vms) Starting client (__init__:308) 2019-04-13 21:45:44,909+0200 INFO (itmap/2) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_isos) Starting client (__init__:308) 2019-04-13 21:45:44,912+0200 INFO (ioprocess/16521) [IOProcess] (/nas.discovery.intern:_mnt_d2600_vms) Starting ioprocess (__init__:434) 2019-04-13 21:45:44,921+0200 INFO (itmap/3) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting client (__init__:308) 2019-04-13 21:45:44,924+0200 INFO (ioprocess/16528) [IOProcess] (/nas.discovery.intern:_mnt_d2600_isos) Starting ioprocess (__init__:434) 2019-04-13 21:45:44,934+0200 INFO (itmap/4) [IOProcessClient] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting client (__init__:308) 2019-04-13 21:45:44,938+0200 INFO (ioprocess/16535) [IOProcess] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting ioprocess (__init__:434) 2019-04-13 21:45:44,948+0200 INFO (ioprocess/16542) [IOProcess] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting ioprocess (__init__:434) 2019-04-13 21:45:44,964+0200 INFO (monitor/6a4cde7) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:45:45,046+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:45:45,059+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:45:45,070+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 21:45:47,047+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=cecaaae4-6a72-49e0-891a-7b7f768a31cd (api:48) 2019-04-13 21:45:47,053+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 27L}} from=::ffff:192.168.0.19,51130, task_id=cecaaae4-6a72-49e0-891a-7b7f768a31cd (api:54) 2019-04-13 21:45:47,053+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:45:47,149+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51166, task_id=88ae1ffe-302e-4d40-a453-ce20dd6122b8 (api:48) 2019-04-13 21:45:47,156+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 27L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,51166, task_id=88ae1ffe-302e-4d40-a453-ce20dd6122b8 (api:54) 2019-04-13 21:45:47,157+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:45:48,370+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:45:48,370+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:45:48,371+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:45:51,133+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=498d65bc-7d55-4c16-ac32-45ba6718b71f (api:48) 2019-04-13 21:45:51,133+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00069129', 'lastCheck': '6.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000852079', 'lastCheck': '6.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000648306', 'lastCheck': '6.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00034102', 'lastCheck': '6.2', 'valid': True}} from=internal, task_id=498d65bc-7d55-4c16-ac32-45ba6718b71f (api:54) 2019-04-13 21:45:54,259+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:45:54,281+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=079d73bd-a471-4ac7-87e8-b8bc0289d59a (api:48) 2019-04-13 21:45:54,281+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00069129', 'lastCheck': '9.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000852079', 'lastCheck': '9.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000648306', 'lastCheck': '9.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00034102', 'lastCheck': '9.3', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=079d73bd-a471-4ac7-87e8-b8bc0289d59a (api:54) 2019-04-13 21:45:54,282+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=49d8f93d-0eaf-4989-90f5-f7a4e8648a36 (api:48) 2019-04-13 21:45:54,282+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=49d8f93d-0eaf-4989-90f5-f7a4e8648a36 (api:54) 2019-04-13 21:45:54,288+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.80'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '1.40', 'nodeIndex': 0, 'cpuSys': '1.40', 'cpuIdle': '97.20'}, '0': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '3': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '2': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '5': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '4': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '7': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '9': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '8': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '13': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '17': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95619'}, '0': {'memPercent': 4, 'memFree': '95028'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00069129', 'lastCheck': '9.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000648306', 'lastCheck': '9.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00034102', 'lastCheck': '9.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000852079', 'lastCheck': '9.2', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184754.280308, 'rx': '10027', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '2492479', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184754.280308, 'rx': '51634187', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184754.280308, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '2395011', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184754.280308, 'rx': '50651199', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '149552506', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184754.280308, 'rx': '149552506', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184754.280308, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '1134368', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184754.280308, 'rx': '51617836', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1358111', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184754.280308, 'rx': '16351', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184754.280308, 'rx': '60216', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184754.280308, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184754.280308, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184754.280308, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184754.280308, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184754.280308, 'rx': '41216', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '48', 'ksmPages': 100, 'elapsedTime': '342.28', 'cpuLoad': '0.55', 'cpuSys': '0.09', 'diskStats': {'/var/log': {'free': '47693'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47693'}}, 'cpuUserVdsmd': '1.40', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191732, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:45:54 GMT', 'cpuUser': '0.09', 'memFree': 191476, 'cpuIdle': '99.82', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.80'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:45:54,290+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:45:54,575+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:45:54,575+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:45:54,575+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:45:57,204+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=00b16cf2-b257-45f3-815e-70896253641a (api:48) 2019-04-13 21:45:57,210+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 27L}} from=::ffff:192.168.0.19,51130, task_id=00b16cf2-b257-45f3-815e-70896253641a (api:54) 2019-04-13 21:45:57,210+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:45:57,638+0200 INFO (jsonrpc/1) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51166, task_id=57985bcf-4f14-4f0d-bc20-81d6df00c88d (api:48) 2019-04-13 21:45:57,645+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 27L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,51166, task_id=57985bcf-4f14-4f0d-bc20-81d6df00c88d (api:54) 2019-04-13 21:45:57,646+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:46:03,391+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:46:03,392+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:46:03,392+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:46:06,169+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=134990de-e8fd-49b9-b413-ac78989167bd (api:48) 2019-04-13 21:46:06,169+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000744771', 'lastCheck': '1.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000927812', 'lastCheck': '1.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000761432', 'lastCheck': '1.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000490971', 'lastCheck': '1.2', 'valid': True}} from=internal, task_id=134990de-e8fd-49b9-b413-ac78989167bd (api:54) 2019-04-13 21:46:07,695+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=8e7ceda3-b29d-4227-a936-46b37219bcb0 (api:48) 2019-04-13 21:46:07,701+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 27L}} from=::ffff:192.168.0.19,51130, task_id=8e7ceda3-b29d-4227-a936-46b37219bcb0 (api:54) 2019-04-13 21:46:07,701+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:46:07,712+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51166, task_id=1eeeaaa6-955f-4422-a390-8dbe06c07bd5 (api:48) 2019-04-13 21:46:07,718+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 27L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,51166, task_id=1eeeaaa6-955f-4422-a390-8dbe06c07bd5 (api:54) 2019-04-13 21:46:07,719+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:46:09,378+0200 INFO (jsonrpc/7) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:46:09,400+0200 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=2b8be9e9-4859-483e-9591-116aae11ee29 (api:48) 2019-04-13 21:46:09,401+0200 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000744771', 'lastCheck': '4.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000927812', 'lastCheck': '4.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000761432', 'lastCheck': '4.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000490971', 'lastCheck': '4.4', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=2b8be9e9-4859-483e-9591-116aae11ee29 (api:54) 2019-04-13 21:46:09,401+0200 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=76dea1c0-8053-422d-9945-92b95e01b495 (api:48) 2019-04-13 21:46:09,401+0200 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=76dea1c0-8053-422d-9945-92b95e01b495 (api:54) 2019-04-13 21:46:09,407+0200 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.80'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '1.40', 'nodeIndex': 0, 'cpuSys': '1.20', 'cpuIdle': '97.40'}, '0': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '3': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '5': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '4': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '8': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '13': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '17': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '0.27', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.46'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95622'}, '0': {'memPercent': 4, 'memFree': '95029'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000744771', 'lastCheck': '4.3', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000761432', 'lastCheck': '4.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000490971', 'lastCheck': '4.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000927812', 'lastCheck': '4.3', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184769.399642, 'rx': '10417', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '2518597', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184769.399642, 'rx': '52745776', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184769.399642, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '2419905', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184769.399642, 'rx': '51714746', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '160984025', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184769.399642, 'rx': '160984025', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184769.399642, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '1135892', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184769.399642, 'rx': '52728657', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1382705', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184769.399642, 'rx': '17119', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184769.399642, 'rx': '62648', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184769.399642, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184769.399642, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184769.399642, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184769.399642, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184769.399642, 'rx': '43010', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '46', 'ksmPages': 100, 'elapsedTime': '357.40', 'cpuLoad': '0.53', 'cpuSys': '0.08', 'diskStats': {'/var/log': {'free': '47696'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47696'}}, 'cpuUserVdsmd': '1.40', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191735, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:46:09 GMT', 'cpuUser': '0.09', 'memFree': 191479, 'cpuIdle': '99.84', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.53'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:46:09,410+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:46:09,585+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:46:09,585+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:46:09,585+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:46:17,755+0200 INFO (jsonrpc/6) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=bfd768aa-699f-478c-ac73-4836516ca68f (api:48) 2019-04-13 21:46:17,760+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 27L}} from=::ffff:192.168.0.19,51130, task_id=bfd768aa-699f-478c-ac73-4836516ca68f (api:54) 2019-04-13 21:46:17,760+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:46:17,767+0200 INFO (jsonrpc/0) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51166, task_id=98aa71e3-26de-44f2-badf-4ab1d68e42ae (api:48) 2019-04-13 21:46:17,774+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 27L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,51166, task_id=98aa71e3-26de-44f2-badf-4ab1d68e42ae (api:54) 2019-04-13 21:46:17,775+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:46:18,415+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:46:18,415+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:46:18,415+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:46:21,197+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=b7eb96fe-3e04-4290-a03e-c9a0a5fd07e9 (api:48) 2019-04-13 21:46:21,198+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000693287', 'lastCheck': '6.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000805331', 'lastCheck': '6.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00067757', 'lastCheck': '6.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000512644', 'lastCheck': '6.2', 'valid': True}} from=internal, task_id=b7eb96fe-3e04-4290-a03e-c9a0a5fd07e9 (api:54) 2019-04-13 21:46:24,409+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:46:24,431+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=74009f00-e589-4527-a632-b68afc61cf76 (api:48) 2019-04-13 21:46:24,432+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000693287', 'lastCheck': '9.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000805331', 'lastCheck': '9.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00067757', 'lastCheck': '9.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000512644', 'lastCheck': '9.4', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=74009f00-e589-4527-a632-b68afc61cf76 (api:54) 2019-04-13 21:46:24,432+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=38e8bacc-a523-4481-a5d2-751dd0d73fa7 (api:48) 2019-04-13 21:46:24,432+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=38e8bacc-a523-4481-a5d2-751dd0d73fa7 (api:54) 2019-04-13 21:46:24,438+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.80'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '1.26', 'nodeIndex': 0, 'cpuSys': '1.06', 'cpuIdle': '97.68'}, '0': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '3': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '5': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '4': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '7': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '9': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '8': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '13': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.60'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '17': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95621'}, '0': {'memPercent': 4, 'memFree': '95029'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000693287', 'lastCheck': '9.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00067757', 'lastCheck': '9.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000512644', 'lastCheck': '9.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000805331', 'lastCheck': '9.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184784.430641, 'rx': '10807', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '2538899', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184784.430641, 'rx': '53853366', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184784.430641, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '2439225', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184784.430641, 'rx': '52773824', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '172384306', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184784.430641, 'rx': '172384306', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184784.430641, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '1136020', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184784.430641, 'rx': '53834749', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1402879', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184784.430641, 'rx': '18617', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184784.430641, 'rx': '65106', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184784.430641, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184784.430641, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184784.430641, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184784.430641, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184784.430641, 'rx': '44666', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '46', 'ksmPages': 100, 'elapsedTime': '372.43', 'cpuLoad': '0.50', 'cpuSys': '0.06', 'diskStats': {'/var/log': {'free': '47696'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47696'}}, 'cpuUserVdsmd': '1.20', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191735, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:46:24 GMT', 'cpuUser': '0.08', 'memFree': 191479, 'cpuIdle': '99.86', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.60'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:46:24,441+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:46:24,591+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:46:24,591+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:46:24,592+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:46:27,811+0200 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=9ded0237-5db7-45f2-aad8-15bf87401ba9 (api:48) 2019-04-13 21:46:27,816+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 27L}} from=::ffff:192.168.0.19,51130, task_id=9ded0237-5db7-45f2-aad8-15bf87401ba9 (api:54) 2019-04-13 21:46:27,817+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:46:27,825+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51166, task_id=139c4b55-66ae-4d6d-9bd1-3ee775b5f178 (api:48) 2019-04-13 21:46:27,832+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 27L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,51166, task_id=139c4b55-66ae-4d6d-9bd1-3ee775b5f178 (api:54) 2019-04-13 21:46:27,833+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:46:33,439+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:46:33,440+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:46:33,440+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:46:36,225+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=20e5f882-aeae-490f-8553-ba829d30078e (api:48) 2019-04-13 21:46:36,226+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000718855', 'lastCheck': '1.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000728191', 'lastCheck': '1.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000732498', 'lastCheck': '1.1', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000544318', 'lastCheck': '1.2', 'valid': True}} from=internal, task_id=20e5f882-aeae-490f-8553-ba829d30078e (api:54) 2019-04-13 21:46:37,869+0200 INFO (jsonrpc/6) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, task_id=6153d676-8111-467b-ae22-6bb89447e960 (api:48) 2019-04-13 21:46:37,874+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 27L}} from=::ffff:192.168.0.19,51130, task_id=6153d676-8111-467b-ae22-6bb89447e960 (api:54) 2019-04-13 21:46:37,875+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:46:37,885+0200 INFO (jsonrpc/0) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51166, task_id=56415239-882e-4f2a-901b-35271c1cb679 (api:48) 2019-04-13 21:46:37,892+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 27L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,51166, task_id=56415239-882e-4f2a-901b-35271c1cb679 (api:54) 2019-04-13 21:46:37,892+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:46:39,449+0200 INFO (jsonrpc/1) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:46:39,470+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=b7a048d8-f7a4-464c-b352-f2693473cc16 (api:48) 2019-04-13 21:46:39,470+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000718855', 'lastCheck': '4.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000728191', 'lastCheck': '4.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000732498', 'lastCheck': '4.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000544318', 'lastCheck': '4.5', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=b7a048d8-f7a4-464c-b352-f2693473cc16 (api:54) 2019-04-13 21:46:39,471+0200 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=26124a07-60bd-4386-a84f-9bd9cb9bf526 (api:48) 2019-04-13 21:46:39,471+0200 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=26124a07-60bd-4386-a84f-9bd9cb9bf526 (api:54) 2019-04-13 21:46:39,477+0200 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.80'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '1.40', 'nodeIndex': 0, 'cpuSys': '1.20', 'cpuIdle': '97.40'}, '0': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '3': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '2': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '5': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '4': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '9': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '8': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '13': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.54'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '17': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95622'}, '0': {'memPercent': 4, 'memFree': '95028'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000718855', 'lastCheck': '4.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000732498', 'lastCheck': '4.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000544318', 'lastCheck': '4.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000728191', 'lastCheck': '4.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184799.46973, 'rx': '11197', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '2558571', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184799.46973, 'rx': '53865055', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184799.46973, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '2458313', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184799.46973, 'rx': '52784635', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '184869473', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184799.46973, 'rx': '184869473', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184799.46973, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '1136114', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184799.46973, 'rx': '53845542', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1422457', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184799.46973, 'rx': '19513', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184799.46973, 'rx': '67602', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184799.46973, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184799.46973, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184799.46973, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184799.46973, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184799.46973, 'rx': '46414', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '46', 'ksmPages': 100, 'elapsedTime': '387.47', 'cpuLoad': '0.49', 'cpuSys': '0.07', 'diskStats': {'/var/log': {'free': '47696'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47696'}}, 'cpuUserVdsmd': '1.26', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191736, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:46:39 GMT', 'cpuUser': '0.08', 'memFree': 191480, 'cpuIdle': '99.84', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.60'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:46:39,480+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:46:39,596+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:46:39,597+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:46:39,597+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:46:45,078+0200 INFO (monitor/9c3f06c) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_isos) Closing client (__init__:584) 2019-04-13 21:46:45,078+0200 INFO (monitor/9c3f06c) [IOProcessClient] (/glusterSD/onode-2.discovery.intern:vol-node-2) Closing client (__init__:584) 2019-04-13 21:46:45,078+0200 INFO (monitor/9c3f06c) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_vms) Closing client (__init__:584) 2019-04-13 21:46:45,078+0200 INFO (monitor/9c3f06c) [IOProcessClient] (/onode-1.discovery.intern:_mnt_store) Closing client (__init__:584) 2019-04-13 21:46:45,078+0200 INFO (monitor/9c3f06c) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Closing client (__init__:584) 2019-04-13 21:46:45,746+0200 INFO (jsonrpc/4) [vdsm.api] START getAllTasksStatuses(spUUID=None, options=None) from=::ffff:192.168.0.19,51130, flow_id=e5f65a30-68fb-444e-bd06-dee750fdef60, task_id=0ca0612a-3ec4-41fc-a078-1821d5ff6f80 (api:48) 2019-04-13 21:46:45,746+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getAllTasksStatuses return={'allTasksStatus': {}} from=::ffff:192.168.0.19,51130, flow_id=e5f65a30-68fb-444e-bd06-dee750fdef60, task_id=0ca0612a-3ec4-41fc-a078-1821d5ff6f80 (api:54) 2019-04-13 21:46:45,746+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksStatuses succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:46:45,749+0200 INFO (jsonrpc/5) [vdsm.api] START spmStop(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,51130, flow_id=e5f65a30-68fb-444e-bd06-dee750fdef60, task_id=52015177-4e8e-4fa2-b04d-bff60766fb2b (api:48) 2019-04-13 21:46:45,751+0200 INFO (jsonrpc/5) [storage.SANLock] Releasing Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576) (clusterlock:488) 2019-04-13 21:46:45,783+0200 INFO (jsonrpc/5) [storage.SANLock] Successfully released Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576) (clusterlock:497) 2019-04-13 21:46:45,783+0200 INFO (jsonrpc/5) [vdsm.api] FINISH spmStop return=None from=::ffff:192.168.0.19,51130, flow_id=e5f65a30-68fb-444e-bd06-dee750fdef60, task_id=52015177-4e8e-4fa2-b04d-bff60766fb2b (api:54) 2019-04-13 21:46:45,784+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.spmStop succeeded in 0.03 seconds (__init__:312) 2019-04-13 21:46:47,416+0200 INFO (mailbox-spm) [storage.MailBox.SpmMailMonitor] SPM_MailMonitor - Incoming mail monitoring thread stopped (mailbox:767) 2019-04-13 21:46:48,461+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:46:48,461+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:46:48,462+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:46:48,525+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,51130 (api:48) 2019-04-13 21:46:48,556+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,51130, task_id=aa89fa0a-94b2-4c80-a32b-739377604504 (api:48) 2019-04-13 21:46:48,557+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000559934', 'lastCheck': '3.5', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000831309', 'lastCheck': '3.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000517452', 'lastCheck': '3.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000496842', 'lastCheck': '3.6', 'valid': True}} from=::ffff:192.168.0.19,51130, task_id=aa89fa0a-94b2-4c80-a32b-739377604504 (api:54) 2019-04-13 21:46:48,558+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,51130, task_id=6c0b3d9a-9695-452c-a7a6-1f8959afcab1 (api:48) 2019-04-13 21:46:48,558+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,51130, task_id=6c0b3d9a-9695-452c-a7a6-1f8959afcab1 (api:54) 2019-04-13 21:46:48,564+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.80'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '1.40', 'nodeIndex': 0, 'cpuSys': '1.20', 'cpuIdle': '97.40'}, '0': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '3': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '2': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '5': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '4': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '9': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '8': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '13': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.54'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '17': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95622'}, '0': {'memPercent': 4, 'memFree': '95028'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000559934', 'lastCheck': '3.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000517452', 'lastCheck': '3.5', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000496842', 'lastCheck': '3.6', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000831309', 'lastCheck': '3.5', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184808.55588, 'rx': '11457', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '2577733', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184808.55588, 'rx': '54971907', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184808.55588, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '2476763', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184808.55588, 'rx': '53843687', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '187963372', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184808.55588, 'rx': '187963372', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184808.55588, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '1136114', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184808.55588, 'rx': '54951792', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '1441619', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184808.55588, 'rx': '20115', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1420', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555184808.55588, 'rx': '69228', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184808.55588, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184808.55588, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184808.55588, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184808.55588, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555184808.55588, 'rx': '47334', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '46', 'ksmPages': 100, 'elapsedTime': '396.54', 'cpuLoad': '0.49', 'cpuSys': '0.07', 'diskStats': {'/var/log': {'free': '47696'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47696'}}, 'cpuUserVdsmd': '1.26', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191740, 'bootTime': '1555184357', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T19:46:48 GMT', 'cpuUser': '0.08', 'memFree': 191484, 'cpuIdle': '99.84', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.60'}} from=::ffff:192.168.0.19,51130 (api:54) 2019-04-13 21:46:48,567+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.04 seconds (__init__:312) 2019-04-13 21:46:48,633+0200 INFO (jsonrpc/6) [vdsm.api] START disconnectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, remove=False, options=None) from=::ffff:192.168.0.19,51130, task_id=b77227af-11da-4cf8-b53c-2b4af1efba55 (api:48) 2019-04-13 21:46:48,634+0200 INFO (jsonrpc/6) [storage.StoragePool] Disconnect from the storage pool 59ef3a18-002f-02d1-0220-000000000124 (sp:736) 2019-04-13 21:46:48,955+0200 INFO (mailbox-hsm) [storage.MailBox.HsmMailMonitor] HSM_MailboxMonitor - Incoming mail monitoring thread stopped, clearing outgoing mail (mailbox:501) 2019-04-13 21:46:48,955+0200 INFO (mailbox-hsm) [storage.MailBox.HsmMailMonitor] HSM_MailMonitor sending mail to SPM - ['/usr/bin/dd', 'of=/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd/dom_md/inbox', 'iflag=fullblock', 'oflag=direct', 'conv=notrunc', 'bs=4096', 'count=1', 'seek=1'] (mailbox:381) 2019-04-13 21:46:50,901+0200 INFO (jsonrpc/6) [storage.StoragePool] Removing pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:746) 2019-04-13 21:46:50,901+0200 INFO (jsonrpc/6) [storage.fileUtils] Removing directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 (fileUtils:180) 2019-04-13 21:46:50,901+0200 INFO (jsonrpc/6) [storage.Monitor] Stop monitoring 9c3f06cf-7475-448e-819b-f4f52fa7d782 (shutdown=False) (monitor:238) 2019-04-13 21:46:50,901+0200 INFO (jsonrpc/6) [storage.Monitor] Stop monitoring 42bb6772-12f9-4bf2-9245-43caa41c5a2a (shutdown=False) (monitor:238) 2019-04-13 21:46:50,901+0200 INFO (jsonrpc/6) [storage.Monitor] Stop monitoring 93f02ede-ec5c-4641-8e97-2949f98e856d (shutdown=False) (monitor:238) 2019-04-13 21:46:50,901+0200 INFO (jsonrpc/6) [storage.Monitor] Stop monitoring 6a4cde72-6a6a-499e-9b6e-d490d91ca19e (shutdown=False) (monitor:238) 2019-04-13 21:46:50,903+0200 INFO (monitor/9c3f06c) [storage.check] Stop checking u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/dom_md/metadata' (check:135) 2019-04-13 21:46:50,903+0200 INFO (monitor/42bb677) [storage.check] Stop checking u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/dom_md/metadata' (check:135) 2019-04-13 21:46:50,903+0200 INFO (monitor/93f02ed) [storage.check] Stop checking u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d/dom_md/metadata' (check:135) 2019-04-13 21:46:50,903+0200 INFO (monitor/6a4cde7) [storage.check] Stop checking u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/metadata' (check:135) 2019-04-13 21:46:50,903+0200 INFO (monitor/6a4cde7) [storage.SANLock] Releasing host id for domain 6a4cde72-6a6a-499e-9b6e-d490d91ca19e (id: 1) (clusterlock:335) 2019-04-13 21:46:50,904+0200 INFO (monitor/9c3f06c) [storage.SANLock] Releasing host id for domain 9c3f06cf-7475-448e-819b-f4f52fa7d782 (id: 1) (clusterlock:335) 2019-04-13 21:46:51,253+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=4cd8955c-31c8-43bd-9828-e6632f1c7ef2 (api:48) 2019-04-13 21:46:51,254+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000559934', 'lastCheck': '6.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000831309', 'lastCheck': '6.2', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000517452', 'lastCheck': '6.2', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000496842', 'lastCheck': '6.3', 'valid': True}} from=internal, task_id=4cd8955c-31c8-43bd-9828-e6632f1c7ef2 (api:54) 2019-04-13 21:46:52,904+0200 INFO (monitor/6a4cde7) [storage.SANLock] Host id for domain 6a4cde72-6a6a-499e-9b6e-d490d91ca19e released successfully (id: 1) (clusterlock:349) 2019-04-13 21:46:52,905+0200 INFO (monitor/9c3f06c) [storage.SANLock] Host id for domain 9c3f06cf-7475-448e-819b-f4f52fa7d782 released successfully (id: 1) (clusterlock:349) 2019-04-13 21:46:52,905+0200 INFO (jsonrpc/6) [vdsm.api] FINISH disconnectStoragePool return=True from=::ffff:192.168.0.19,51130, task_id=b77227af-11da-4cf8-b53c-2b4af1efba55 (api:54) 2019-04-13 21:46:52,905+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.disconnect succeeded in 4.27 seconds (__init__:312) 2019-04-13 21:46:53,034+0200 INFO (jsonrpc/0) [vdsm.api] START disconnectStorageServer(domType=1, spUUID=u'59ef3a18-002f-02d1-0220-000000000124', conList=[{u'tpgt': u'1', u'id': u'2286311a-c1c9-4dda-bea5-9c78651c6e36', u'connection': u'onode-1.discovery.intern:/mnt/store', u'iqn': u'', u'user': u'', u'ipv6_enabled': u'false', u'protocol_version': u'auto', u'password': '********', u'port': u''}, {u'tpgt': u'1', u'id': u'858731bf-2fb1-45a7-a693-235f675a667b', u'connection': u'nas.discovery.intern:/mnt/d2600/sicherung/vmBackups', u'iqn': u'', u'user': u'', u'ipv6_enabled': u'false', u'protocol_version': u'auto', u'password': '********', u'port': u''}, {u'tpgt': u'1', u'id': u'07dd9ff2-afa1-4a33-b401-935ea5b8c9db', u'connection': u'nas.discovery.intern:/mnt/d2600/isos', u'iqn': u'', u'user': u'', u'ipv6_enabled': u'false', u'protocol_version': u'auto', u'password': '********', u'port': u''}, {u'tpgt': u'1', u'id': u'b4f4bcf2-e104-468e-b330-ed731ec8ef6c', u'connection': u'nas.discovery.intern:/mnt/d2600/vms', u'iqn': u'', u'user': u'', u'ipv6_enabled': u'false', u'protocol_version': u'auto', u'password': '********', u'port': u''}], options=None) from=::ffff:192.168.0.19,51130, flow_id=6bdff425, task_id=78b65c31-7cbb-4b36-99a3-388137195182 (api:48) 2019-04-13 21:46:53,034+0200 INFO (jsonrpc/0) [storage.Mount] unmounting /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store (mount:212) 2019-04-13 21:46:53,131+0200 INFO (jsonrpc/0) [storage.Mount] unmounting /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups (mount:212) 2019-04-13 21:46:53,169+0200 INFO (jsonrpc/0) [storage.Mount] unmounting /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos (mount:212) 2019-04-13 21:46:53,207+0200 INFO (jsonrpc/0) [storage.Mount] unmounting /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms (mount:212) 2019-04-13 21:46:53,511+0200 INFO (jsonrpc/0) [vdsm.api] FINISH disconnectStorageServer return={'statuslist': [{'status': 0, 'id': u'2286311a-c1c9-4dda-bea5-9c78651c6e36'}, {'status': 0, 'id': u'858731bf-2fb1-45a7-a693-235f675a667b'}, {'status': 0, 'id': u'07dd9ff2-afa1-4a33-b401-935ea5b8c9db'}, {'status': 0, 'id': u'b4f4bcf2-e104-468e-b330-ed731ec8ef6c'}]} from=::ffff:192.168.0.19,51130, flow_id=6bdff425, task_id=78b65c31-7cbb-4b36-99a3-388137195182 (api:54) 2019-04-13 21:46:53,512+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.disconnectStorageServer succeeded in 0.48 seconds (__init__:312) 2019-04-13 21:47:03,481+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:47:03,481+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:47:03,481+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:47:06,290+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=e71b1988-9e91-4843-9ae0-2d5eeab587ee (api:48) 2019-04-13 21:47:06,291+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=e71b1988-9e91-4843-9ae0-2d5eeab587ee (api:54) 2019-04-13 21:47:18,504+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:47:18,505+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:47:18,505+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:47:21,332+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=58ddd27a-8ae0-4ad5-b3f2-76a6752c5b44 (api:48) 2019-04-13 21:47:21,332+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={} from=internal, task_id=58ddd27a-8ae0-4ad5-b3f2-76a6752c5b44 (api:54) 2019-04-13 21:47:33,527+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:47:33,527+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:47:33,527+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:47:36,365+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=a55b7220-5ca9-485e-9a2f-40623e81b7b1 (api:48) 2019-04-13 21:47:36,366+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=a55b7220-5ca9-485e-9a2f-40623e81b7b1 (api:54) 2019-04-13 21:47:48,546+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:47:48,546+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:47:48,546+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:47:51,401+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=ab24758a-bed2-4a48-98e7-f9e44e75dd0a (api:48) 2019-04-13 21:47:51,401+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={} from=internal, task_id=ab24758a-bed2-4a48-98e7-f9e44e75dd0a (api:54) 2019-04-13 21:48:03,565+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:48:03,565+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:48:03,565+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:48:06,428+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=9be6bebe-df96-4655-a89a-06eec26186fe (api:48) 2019-04-13 21:48:06,428+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=9be6bebe-df96-4655-a89a-06eec26186fe (api:54) 2019-04-13 21:48:18,588+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:48:18,589+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:48:18,589+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:48:21,453+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=258abc6a-e91a-4277-85b3-e7b0eabf98a7 (api:48) 2019-04-13 21:48:21,453+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={} from=internal, task_id=258abc6a-e91a-4277-85b3-e7b0eabf98a7 (api:54) 2019-04-13 21:48:33,607+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:48:33,608+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:48:33,608+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:48:36,490+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=f29b4095-3ac0-4a24-b2d2-d9dd24ef38bc (api:48) 2019-04-13 21:48:36,490+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={} from=internal, task_id=f29b4095-3ac0-4a24-b2d2-d9dd24ef38bc (api:54) 2019-04-13 21:48:48,626+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:48:48,626+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:48:48,627+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:48:51,516+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=349a9f06-14ce-46f6-8ed6-8e9663fa7e9c (api:48) 2019-04-13 21:48:51,516+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=349a9f06-14ce-46f6-8ed6-8e9663fa7e9c (api:54) 2019-04-13 21:49:03,647+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:49:03,647+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:49:03,647+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:49:06,542+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=401f1a08-f33a-4636-bde0-a3a389d5c873 (api:48) 2019-04-13 21:49:06,542+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=401f1a08-f33a-4636-bde0-a3a389d5c873 (api:54) 2019-04-13 21:49:18,665+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:49:18,666+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:49:18,666+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:49:21,579+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=abbec4a4-0d9b-4e97-84ef-cf19d3bae12e (api:48) 2019-04-13 21:49:21,579+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=abbec4a4-0d9b-4e97-84ef-cf19d3bae12e (api:54) 2019-04-13 21:49:33,684+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:49:33,685+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:49:33,685+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:49:36,605+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=916b3881-fdce-48d2-ae94-babd23c179ba (api:48) 2019-04-13 21:49:36,605+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={} from=internal, task_id=916b3881-fdce-48d2-ae94-babd23c179ba (api:54) 2019-04-13 21:49:48,703+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:49:48,703+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:49:48,704+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:49:51,640+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=20a623a0-729c-4d02-8569-e1bd6c2be593 (api:48) 2019-04-13 21:49:51,640+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={} from=internal, task_id=20a623a0-729c-4d02-8569-e1bd6c2be593 (api:54) 2019-04-13 21:50:03,722+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:50:03,723+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:50:03,723+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:50:06,667+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=8405d7d5-731b-4c99-b3b9-6159a0c79d02 (api:48) 2019-04-13 21:50:06,667+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={} from=internal, task_id=8405d7d5-731b-4c99-b3b9-6159a0c79d02 (api:54) 2019-04-13 21:50:18,741+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:50:18,741+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:50:18,741+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:50:21,692+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=ac3ae44f-f257-4eec-965f-f80b68338aac (api:48) 2019-04-13 21:50:21,692+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={} from=internal, task_id=ac3ae44f-f257-4eec-965f-f80b68338aac (api:54) 2019-04-13 21:50:33,760+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:50:33,760+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:50:33,760+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:50:36,728+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=a60c2a89-7ddc-422e-8d8d-7200066ca106 (api:48) 2019-04-13 21:50:36,729+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=a60c2a89-7ddc-422e-8d8d-7200066ca106 (api:54) 2019-04-13 21:50:48,779+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:50:48,779+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:50:48,779+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:50:51,755+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=6ec6eca3-6ecf-4127-a472-e69f092efcb7 (api:48) 2019-04-13 21:50:51,755+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=6ec6eca3-6ecf-4127-a472-e69f092efcb7 (api:54) 2019-04-13 21:51:03,794+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:51:03,794+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:51:03,794+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:51:06,791+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=a004a125-d503-46dc-acbc-8c17ceb5a4aa (api:48) 2019-04-13 21:51:06,791+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=a004a125-d503-46dc-acbc-8c17ceb5a4aa (api:54) 2019-04-13 21:51:18,812+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:51:18,813+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:51:18,813+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:51:21,816+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=4f7a9f5c-a888-4224-ab10-827b15657b4e (api:48) 2019-04-13 21:51:21,817+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=4f7a9f5c-a888-4224-ab10-827b15657b4e (api:54) 2019-04-13 21:51:33,831+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:51:33,831+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:51:33,832+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:51:36,842+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=8f108f71-3582-4936-bc7f-b7cf33b4ae56 (api:48) 2019-04-13 21:51:36,842+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=8f108f71-3582-4936-bc7f-b7cf33b4ae56 (api:54) 2019-04-13 21:51:48,850+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:51:48,850+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:51:48,851+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:51:51,880+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=e72dc81b-c797-4643-a95b-f32dad4ecf9a (api:48) 2019-04-13 21:51:51,881+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={} from=internal, task_id=e72dc81b-c797-4643-a95b-f32dad4ecf9a (api:54) 2019-04-13 21:52:03,869+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:52:03,869+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:52:03,869+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:52:06,905+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=87002ae6-968a-4bc0-9d83-ca113cd107f5 (api:48) 2019-04-13 21:52:06,905+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=87002ae6-968a-4bc0-9d83-ca113cd107f5 (api:54) 2019-04-13 21:52:18,888+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:52:18,888+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:52:18,888+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:52:21,932+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=4db96cda-d8e6-4885-80b4-92fad38540cb (api:48) 2019-04-13 21:52:21,933+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={} from=internal, task_id=4db96cda-d8e6-4885-80b4-92fad38540cb (api:54) 2019-04-13 21:52:33,907+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:52:33,907+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:52:33,907+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:312) 2019-04-13 21:52:36,968+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=165e3594-6ca7-416d-8fce-66dedd26ff0c (api:48) 2019-04-13 21:52:36,968+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=165e3594-6ca7-416d-8fce-66dedd26ff0c (api:54) 2019-04-13 21:52:48,926+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:52:48,926+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:52:48,926+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:52:51,994+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=3fefcc17-deb2-4e08-92ff-7f43851b7cf0 (api:48) 2019-04-13 21:52:51,994+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=3fefcc17-deb2-4e08-92ff-7f43851b7cf0 (api:54) 2019-04-13 21:53:03,945+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:53:03,945+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:53:03,945+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:53:07,031+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=78c31fd6-4ac5-45d8-9b59-1cecae0ab9c0 (api:48) 2019-04-13 21:53:07,031+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=78c31fd6-4ac5-45d8-9b59-1cecae0ab9c0 (api:54) 2019-04-13 21:53:18,964+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:53:18,964+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:53:18,964+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:53:22,056+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=9286b89b-b1c2-4999-b4fd-168053e61bbe (api:48) 2019-04-13 21:53:22,056+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=9286b89b-b1c2-4999-b4fd-168053e61bbe (api:54) 2019-04-13 21:53:33,982+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:53:33,983+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:53:33,983+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:53:37,082+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=c3e4dc2c-8e5a-44e1-a8ba-37678692827d (api:48) 2019-04-13 21:53:37,082+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=c3e4dc2c-8e5a-44e1-a8ba-37678692827d (api:54) 2019-04-13 21:53:49,001+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:53:49,002+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:53:49,002+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:53:52,119+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=66d338b3-f4af-42f5-b5b0-77c60987f222 (api:48) 2019-04-13 21:53:52,119+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={} from=internal, task_id=66d338b3-f4af-42f5-b5b0-77c60987f222 (api:54) 2019-04-13 21:54:04,022+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:54:04,023+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:54:04,023+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:54:07,148+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=64df9ad7-60d4-4a72-91a1-69631ce3e094 (api:48) 2019-04-13 21:54:07,148+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=64df9ad7-60d4-4a72-91a1-69631ce3e094 (api:54) 2019-04-13 21:54:19,045+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:54:19,045+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:54:19,045+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:54:22,182+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=68cc21a8-7081-41d8-b959-8625584d9c02 (api:48) 2019-04-13 21:54:22,182+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={} from=internal, task_id=68cc21a8-7081-41d8-b959-8625584d9c02 (api:54) 2019-04-13 21:54:34,050+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:54:34,051+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:54:34,051+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:54:37,217+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=6b8580aa-b027-42c9-a2f6-b2f9e5210c4e (api:48) 2019-04-13 21:54:37,218+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=6b8580aa-b027-42c9-a2f6-b2f9e5210c4e (api:54) 2019-04-13 21:54:49,073+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:54:49,073+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:54:49,073+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:54:52,242+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=5381bb33-ec61-4790-bfc7-b65304a9958f (api:48) 2019-04-13 21:54:52,242+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=5381bb33-ec61-4790-bfc7-b65304a9958f (api:54) 2019-04-13 21:55:04,092+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,58472 (api:48) 2019-04-13 21:55:04,092+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,58472 (api:54) 2019-04-13 21:55:04,092+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 21:55:07,278+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=bae4d8fa-4b48-452c-88cf-d7b0aa500217 (api:48) 2019-04-13 21:55:07,278+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=bae4d8fa-4b48-452c-88cf-d7b0aa500217 (api:54) 2019-04-13 21:55:08,309+0200 INFO (mpathlistener) [storage.mpathhealth] Multipath device u'3600605b002f6927021946993801eb9a6' was removed (mpathhealth:151) 2019-04-13 21:55:18,245+0200 INFO (MainThread) [vds] Received signal 15, shutting down (vdsmd:70) 2019-04-13 21:55:18,262+0200 INFO (MainThread) [vds] host in shutdown waiting (clientIF:356) 2019-04-13 21:55:20,265+0200 INFO (MainThread) [jsonrpc.JsonRpcServer] Stopping JsonRPC Server (__init__:442) 2019-04-13 21:55:20,268+0200 INFO (MainThread) [vds] Stopping http server (http:80) 2019-04-13 21:55:20,268+0200 INFO (http) [vds] Server stopped (http:70) 2019-04-13 21:55:20,269+0200 INFO (MainThread) [root] Unregistering all secrets (secret:93) 2019-04-13 21:55:20,270+0200 INFO (MainThread) [vds] Stopping QEMU-GA poller (qemuguestagent:158) 2019-04-13 21:55:20,271+0200 INFO (MainThread) [vdsm.api] START prepareForShutdown(options=None) from=internal, task_id=cdf1b4f0-da56-42fc-8bc7-3ce1cc5afb26 (api:48) 2019-04-13 21:55:20,343+0200 INFO (MainThread) [storage.Monitor] Shutting down domain monitors (monitor:223) 2019-04-13 21:55:20,344+0200 INFO (MainThread) [storage.check] Stopping check service (check:104) 2019-04-13 21:55:20,344+0200 INFO (check/loop) [storage.asyncevent] Stopping (asyncevent:220) 2019-04-13 21:55:20,345+0200 INFO (MainThread) [IOProcessClient] (42bb6772-12f9-4bf2-9245-43caa41c5a2a) Closing client (__init__:584) 2019-04-13 21:55:20,346+0200 INFO (ioprocess/14912) [IOProcessClient] (42bb6772-12f9-4bf2-9245-43caa41c5a2a) Shutdown requested (__init__:107) 2019-04-13 21:55:20,348+0200 INFO (MainThread) [IOProcessClient] (6a4cde72-6a6a-499e-9b6e-d490d91ca19e) Closing client (__init__:584) 2019-04-13 21:55:20,348+0200 INFO (ioprocess/14873) [IOProcessClient] (6a4cde72-6a6a-499e-9b6e-d490d91ca19e) Shutdown requested (__init__:107) 2019-04-13 21:55:20,349+0200 INFO (MainThread) [IOProcessClient] (93f02ede-ec5c-4641-8e97-2949f98e856d) Closing client (__init__:584) 2019-04-13 21:55:20,350+0200 INFO (ioprocess/14919) [IOProcessClient] (93f02ede-ec5c-4641-8e97-2949f98e856d) Shutdown requested (__init__:107) 2019-04-13 21:55:20,351+0200 INFO (MainThread) [IOProcessClient] (9c3f06cf-7475-448e-819b-f4f52fa7d782) Closing client (__init__:584) 2019-04-13 21:55:20,351+0200 INFO (ioprocess/14900) [IOProcessClient] (9c3f06cf-7475-448e-819b-f4f52fa7d782) Shutdown requested (__init__:107) 2019-04-13 21:55:20,352+0200 INFO (MainThread) [storage.udev] Stopping multipath event listener (udev:149) 2019-04-13 21:55:20,353+0200 INFO (MainThread) [vdsm.api] FINISH prepareForShutdown return=None from=internal, task_id=cdf1b4f0-da56-42fc-8bc7-3ce1cc5afb26 (api:54) 2019-04-13 21:55:20,354+0200 INFO (MainThread) [vds] Stopping threads (vdsmd:161) 2019-04-13 21:55:20,354+0200 INFO (MainThread) [vds] Exiting (vdsmd:172) 2019-04-13 22:00:08,055+0200 INFO (MainThread) [vds] (PID: 14300) I am the actual vdsm 4.30.11.1 onode-1.discovery.intern (3.10.0-957.10.1.el7.x86_64) (vdsmd:150) 2019-04-13 22:00:08,055+0200 INFO (MainThread) [vds] VDSM will run with cpu affinity: frozenset([1]) (vdsmd:262) 2019-04-13 22:00:08,060+0200 INFO (MainThread) [storage.HSM] START HSM init (hsm:367) 2019-04-13 22:00:08,063+0200 INFO (MainThread) [storage.HSM] Creating data-center mount directory '/rhev/data-center/mnt' (hsm:372) 2019-04-13 22:00:08,063+0200 INFO (MainThread) [storage.fileUtils] Creating directory: /rhev/data-center/mnt mode: None (fileUtils:199) 2019-04-13 22:00:08,292+0200 INFO (MainThread) [storage.udev] Registering multipath event monitor (udev:182) 2019-04-13 22:00:08,292+0200 INFO (MainThread) [storage.udev] Starting multipath event listener (udev:116) 2019-04-13 22:00:08,316+0200 INFO (MainThread) [storage.check] Starting check service (check:91) 2019-04-13 22:00:08,325+0200 INFO (MainThread) [storage.Dispatcher] Starting StorageDispatcher... (dispatcher:47) 2019-04-13 22:00:08,326+0200 INFO (check/loop) [storage.asyncevent] Starting (asyncevent:125) 2019-04-13 22:00:08,363+0200 INFO (MainThread) [vdsm.api] START registerDomainStateChangeCallback(callbackFunc=) from=internal, task_id=94e41b51-3902-48d1-8d14-88c086ce8150 (api:48) 2019-04-13 22:00:08,364+0200 INFO (MainThread) [vdsm.api] FINISH registerDomainStateChangeCallback return=None from=internal, task_id=94e41b51-3902-48d1-8d14-88c086ce8150 (api:54) 2019-04-13 22:00:08,365+0200 INFO (MainThread) [MOM] Preparing MOM interface (momIF:45) 2019-04-13 22:00:08,365+0200 INFO (MainThread) [MOM] MOM: Using named unix socket: /var/run/vdsm/mom-vdsm.sock (momIF:55) 2019-04-13 22:00:08,370+0200 INFO (MainThread) [root] Unregistering all secrets (secret:93) 2019-04-13 22:00:08,395+0200 INFO (MainThread) [vds] Setting channels' timeout to 30 seconds. (vmchannels:224) 2019-04-13 22:00:08,396+0200 INFO (MainThread) [vds] Starting QEMU-GA poller (qemuguestagent:151) 2019-04-13 22:00:08,400+0200 INFO (MainThread) [vds.MultiProtocolAcceptor] Listening at :::54321 (protocoldetector:183) 2019-04-13 22:00:08,438+0200 INFO (vmrecovery) [vds] recovery: completed in 0s (clientIF:619) 2019-04-13 22:00:08,476+0200 INFO (http) [vds] Server running (http:59) 2019-04-13 22:00:08,528+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=01f605bb-b359-45e1-a69a-6dca2dd58d01 (api:48) 2019-04-13 22:00:08,528+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=01f605bb-b359-45e1-a69a-6dca2dd58d01 (api:54) 2019-04-13 22:00:08,530+0200 WARN (periodic/1) [throttled] MOM not available. (throttledlog:104) 2019-04-13 22:00:08,530+0200 WARN (periodic/1) [throttled] MOM not available, KSM stats will be missing. (throttledlog:104) 2019-04-13 22:00:09,106+0200 INFO (hsm/init) [storage.HSM] FINISH HSM init succeeded in 1.05 seconds (hsm:410) 2019-04-13 22:00:12,891+0200 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:59530 (protocoldetector:61) 2019-04-13 22:00:12,904+0200 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::1:59530 (protocoldetector:125) 2019-04-13 22:00:12,904+0200 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompserver:95) 2019-04-13 22:00:12,905+0200 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompserver:124) 2019-04-13 22:00:12,946+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:00:12,952+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:00:12,953+0200 INFO (jsonrpc/1) [throttled] Current getAllVmStats: {} (throttledlog:104) 2019-04-13 22:00:12,953+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:00:12,953+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:00:23,837+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=fc7aca91-3ec6-49e6-b17c-bd15bfce1459 (api:48) 2019-04-13 22:00:23,838+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={} from=internal, task_id=fc7aca91-3ec6-49e6-b17c-bd15bfce1459 (api:54) 2019-04-13 22:00:27,971+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:00:27,971+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:00:27,972+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:00:28,193+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.setKsmTune succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:00:38,879+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=7d750626-44c8-4c73-8a27-2da0c0e439ff (api:48) 2019-04-13 22:00:38,880+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={} from=internal, task_id=7d750626-44c8-4c73-8a27-2da0c0e439ff (api:54) 2019-04-13 22:00:42,990+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:00:42,990+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:00:42,990+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:00:53,912+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=bb9224ce-8572-46cd-8e49-be2e02fd9943 (api:48) 2019-04-13 22:00:53,913+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=bb9224ce-8572-46cd-8e49-be2e02fd9943 (api:54) 2019-04-13 22:00:58,009+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:00:58,009+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:00:58,009+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:01:08,939+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=a4aca210-f23a-471c-9c7a-324add68ff49 (api:48) 2019-04-13 22:01:08,940+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=a4aca210-f23a-471c-9c7a-324add68ff49 (api:54) 2019-04-13 22:01:13,028+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:01:13,028+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:01:13,029+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:01:23,974+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=2b84b411-492a-4be8-9858-165abbee6601 (api:48) 2019-04-13 22:01:23,974+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={} from=internal, task_id=2b84b411-492a-4be8-9858-165abbee6601 (api:54) 2019-04-13 22:01:28,048+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:01:28,048+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:01:28,048+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:01:39,003+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=8ccfa48b-a3ad-4b81-ad5a-b9707119afa4 (api:48) 2019-04-13 22:01:39,003+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={} from=internal, task_id=8ccfa48b-a3ad-4b81-ad5a-b9707119afa4 (api:54) 2019-04-13 22:01:43,070+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:01:43,070+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:01:43,070+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:01:54,035+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=523b5725-217c-4176-8c32-4febfff8ce3e (api:48) 2019-04-13 22:01:54,036+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=523b5725-217c-4176-8c32-4febfff8ce3e (api:54) 2019-04-13 22:01:58,089+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:01:58,089+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:01:58,089+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:02:09,061+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=bd5bdc51-a8e9-452b-a761-bc336868dab5 (api:48) 2019-04-13 22:02:09,061+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=bd5bdc51-a8e9-452b-a761-bc336868dab5 (api:54) 2019-04-13 22:02:13,108+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:02:13,108+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:02:13,108+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:02:24,098+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=169adad9-5217-4a6a-8d6e-f87769b1cb32 (api:48) 2019-04-13 22:02:24,098+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={} from=internal, task_id=169adad9-5217-4a6a-8d6e-f87769b1cb32 (api:54) 2019-04-13 22:02:28,127+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:02:28,127+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:02:28,127+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:02:39,134+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=9e0a0259-14c6-4e9d-b600-b3ae43817191 (api:48) 2019-04-13 22:02:39,134+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={} from=internal, task_id=9e0a0259-14c6-4e9d-b600-b3ae43817191 (api:54) 2019-04-13 22:02:43,144+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:02:43,144+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:02:43,144+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:02:54,159+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=4a971992-6cc7-4f2d-a6d5-716eedea8afc (api:48) 2019-04-13 22:02:54,159+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=4a971992-6cc7-4f2d-a6d5-716eedea8afc (api:54) 2019-04-13 22:02:58,163+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:02:58,163+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:02:58,163+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:03:09,197+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=e230fb47-f42e-459e-b5f3-68e441df55c8 (api:48) 2019-04-13 22:03:09,197+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=e230fb47-f42e-459e-b5f3-68e441df55c8 (api:54) 2019-04-13 22:03:13,182+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:03:13,182+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:03:13,182+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:03:24,222+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=8e4d9ee3-459b-4a62-87b3-c2e7393184f7 (api:48) 2019-04-13 22:03:24,222+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={} from=internal, task_id=8e4d9ee3-459b-4a62-87b3-c2e7393184f7 (api:54) 2019-04-13 22:03:28,201+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:03:28,201+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:03:28,201+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:03:39,248+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=6597f5be-fcd4-4584-a8da-93bd6432d7eb (api:48) 2019-04-13 22:03:39,248+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={} from=internal, task_id=6597f5be-fcd4-4584-a8da-93bd6432d7eb (api:54) 2019-04-13 22:03:43,220+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:03:43,220+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:03:43,220+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:03:54,284+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=862b19b6-b587-4592-ad9d-f22ae22c578c (api:48) 2019-04-13 22:03:54,285+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=862b19b6-b587-4592-ad9d-f22ae22c578c (api:54) 2019-04-13 22:03:58,238+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:03:58,239+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:03:58,239+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:04:09,311+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=cd727530-e83f-4915-9269-53836f0cd01f (api:48) 2019-04-13 22:04:09,311+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=cd727530-e83f-4915-9269-53836f0cd01f (api:54) 2019-04-13 22:04:13,257+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:04:13,257+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:04:13,258+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:04:24,337+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=aa38e07c-9960-4ce4-9cf8-4c56e7045150 (api:48) 2019-04-13 22:04:24,337+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={} from=internal, task_id=aa38e07c-9960-4ce4-9cf8-4c56e7045150 (api:54) 2019-04-13 22:04:28,276+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:04:28,276+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:04:28,277+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:04:39,364+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=7ef058fd-3c7f-429b-adfc-66a9cbdeeefc (api:48) 2019-04-13 22:04:39,364+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=7ef058fd-3c7f-429b-adfc-66a9cbdeeefc (api:54) 2019-04-13 22:04:43,295+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:04:43,295+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:04:43,295+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:04:54,389+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=3f20f964-a870-4ec8-aec6-8c8cedf32ffd (api:48) 2019-04-13 22:04:54,390+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=3f20f964-a870-4ec8-aec6-8c8cedf32ffd (api:54) 2019-04-13 22:04:58,314+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:04:58,314+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:04:58,314+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:05:09,427+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=6f37fe3e-fd6d-4665-ad2b-dcfe12b599ce (api:48) 2019-04-13 22:05:09,427+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={} from=internal, task_id=6f37fe3e-fd6d-4665-ad2b-dcfe12b599ce (api:54) 2019-04-13 22:05:13,333+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:05:13,333+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:05:13,333+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:05:24,451+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=a0237b45-a238-4554-8615-8e3aadf3a7b0 (api:48) 2019-04-13 22:05:24,452+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={} from=internal, task_id=a0237b45-a238-4554-8615-8e3aadf3a7b0 (api:54) 2019-04-13 22:05:28,352+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:05:28,352+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:05:28,352+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:05:39,478+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=b1da653c-b019-461b-9f51-2e2eb6df3987 (api:48) 2019-04-13 22:05:39,478+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={} from=internal, task_id=b1da653c-b019-461b-9f51-2e2eb6df3987 (api:54) 2019-04-13 22:05:43,366+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:05:43,366+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:05:43,366+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:05:54,504+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=54eb7891-d283-4830-a5cd-26fd1049e9bf (api:48) 2019-04-13 22:05:54,505+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={} from=internal, task_id=54eb7891-d283-4830-a5cd-26fd1049e9bf (api:54) 2019-04-13 22:05:58,385+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:05:58,385+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:05:58,385+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:06:09,531+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=e74d8279-ea3f-4989-b7f6-b33ea46d0787 (api:48) 2019-04-13 22:06:09,531+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=e74d8279-ea3f-4989-b7f6-b33ea46d0787 (api:54) 2019-04-13 22:06:13,404+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:06:13,404+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:06:13,404+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:06:24,566+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=736b1052-bce7-431a-9455-0c9eb5036c6d (api:48) 2019-04-13 22:06:24,567+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={} from=internal, task_id=736b1052-bce7-431a-9455-0c9eb5036c6d (api:54) 2019-04-13 22:06:28,423+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:06:28,423+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:06:28,423+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:06:39,593+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=36b3bc80-6f03-4417-b388-1d75f0f585e2 (api:48) 2019-04-13 22:06:39,594+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={} from=internal, task_id=36b3bc80-6f03-4417-b388-1d75f0f585e2 (api:54) 2019-04-13 22:06:43,442+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:06:43,442+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:06:43,442+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:06:54,619+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=38da6ecc-2a54-45d0-a8f1-7016f6e0e9fc (api:48) 2019-04-13 22:06:54,619+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=38da6ecc-2a54-45d0-a8f1-7016f6e0e9fc (api:54) 2019-04-13 22:06:58,460+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:06:58,461+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:06:58,461+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:07:09,647+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=57b3507a-9902-4c7a-a8a3-f7d1aaa53e7f (api:48) 2019-04-13 22:07:09,648+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=57b3507a-9902-4c7a-a8a3-f7d1aaa53e7f (api:54) 2019-04-13 22:07:13,466+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:07:13,466+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:07:13,466+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:07:23,562+0200 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::ffff:192.168.0.19:42536 (protocoldetector:61) 2019-04-13 22:07:23,707+0200 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::ffff:192.168.0.19:42536 (protocoldetector:125) 2019-04-13 22:07:23,708+0200 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompserver:95) 2019-04-13 22:07:23,709+0200 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompserver:124) 2019-04-13 22:07:23,709+0200 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompserver:124) 2019-04-13 22:07:23,750+0200 INFO (jsonrpc/7) [api.host] START getCapabilities() from=::ffff:192.168.0.19,42536 (api:48) 2019-04-13 22:07:24,675+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=0bad5a9a-736b-4947-9b2f-d540c4854dfa (api:48) 2019-04-13 22:07:24,675+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={} from=internal, task_id=0bad5a9a-736b-4947-9b2f-d540c4854dfa (api:54) 2019-04-13 22:07:28,552+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:07:28,553+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:07:28,553+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:07:43,907+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:07:43,907+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:07:43,908+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:07:44,323+0200 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=9e533a37-b34f-4f4c-8034-df27cf4241e8 (api:48) 2019-04-13 22:07:44,323+0200 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=9e533a37-b34f-4f4c-8034-df27cf4241e8 (api:54) 2019-04-13 22:07:47,832+0200 INFO (jsonrpc/7) [root] managedvolume not supported: Managed Volume Not Supported. Missing package os-brick.: ('Cannot import os_brick',) (caps:149) 2019-04-13 22:07:48,536+0200 INFO (jsonrpc/7) [root] /usr/libexec/vdsm/hooks/after_get_caps/50_openstacknet: rc=0 err= (hooks:114) 2019-04-13 22:07:48,806+0200 INFO (jsonrpc/7) [root] /usr/libexec/vdsm/hooks/after_get_caps/openstacknet_utils.py: rc=0 err= (hooks:114) 2019-04-13 22:07:49,166+0200 INFO (jsonrpc/7) [root] /usr/libexec/vdsm/hooks/after_get_caps/ovirt_provider_ovn_hook: rc=0 err= (hooks:114) 2019-04-13 22:07:49,168+0200 INFO (jsonrpc/7) [api.host] FINISH getCapabilities return={'status': {'message': 'Done', 'code': 0}, 'info': {u'HBAInventory': {u'iSCSI': [{u'InitiatorName': u'iqn.1994-05.com.redhat:5ee45db2f0f'}], u'FC': [{u'wwpn': u'21000024ff2c1d40', u'wwnn': u'20000024ff2c1d40', u'model': u'QLE2562 - QLogic 8Gb FC Dual-port HBA for System x'}, {u'wwpn': u'21000024ff2c1d41', u'wwnn': u'20000024ff2c1d41', u'model': u'QLE2562 - QLogic 8Gb FC Dual-port HBA for System x'}]}, u'packages2': {u'kernel': {u'release': u'957.10.1.el7.x86_64', u'version': u'3.10.0'}, u'glusterfs-rdma': {u'release': u'1.el7', u'version': u'5.5'}, u'glusterfs-fuse': {u'release': u'1.el7', u'version': u'5.5'}, u'spice-server': {u'release': u'6.el7_6.1', u'version': u'0.14.0'}, u'librbd1': {u'release': u'4.el7', u'version': u'10.2.5'}, u'vdsm': {u'release': u'1.el7', u'version': u'4.30.11'}, u'qemu-kvm': {u'release': u'18.el7_6.3.1', u'version': u'2.12.0'}, u'openvswitch': {u'release': u'3.el7', u'version': u'2.10.1'}, u'libvirt': {u'release': u'10.el7_6.6', u'version': u'4.5.0'}, u'ovirt-hosted-engine-ha': {u'release': u'1.el7', u'version': u'2.3.1'}, u'qemu-img': {u'release': u'18.el7_6.3.1', u'version': u'2.12.0'}, u'mom': {u'release': u'1.el7.centos', u'version': u'0.5.12'}, u'glusterfs': {u'release': u'1.el7', u'version': u'5.5'}, u'glusterfs-cli': {u'release': u'1.el7', u'version': u'5.5'}, u'glusterfs-server': {u'release': u'1.el7', u'version': u'5.5'}, u'glusterfs-geo-replication': {u'release': u'1.el7', u'version': u'5.5'}}, u'numaNodeDistance': {u'1': [21, 10], u'0': [10, 21]}, u'cpuModel': u'Intel(R) Xeon(R) CPU X5660 @ 2.80GHz', u'nestedVirtualization': False, u'liveMerge': u'true', u'hooks': {u'after_vm_start': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'ea0a5a715da8c1badbcda28e8b8fa00e'}}, u'after_network_setup': {u'30_ethtool_options': {u'md5': u'ce1fbad7aa0389e3b06231219140bf0d'}}, u'before_vm_start': {u'50_hostedengine': {u'md5': u'95c810cdcfe4195302a59574a5148289'}, u'50_vhostmd': {u'md5': u'9206bc390bcbf208b06a8e899581be2d'}}, u'after_device_migrate_destination': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'6226fbc4d1602994828a3904fc1b875d'}}, u'before_device_migrate_destination': {u'50_vmfex': {u'md5': u'49caba1a5faadd8efacef966f79bc30a'}}, u'before_device_create': {u'50_vmfex': {u'md5': u'49caba1a5faadd8efacef966f79bc30a'}, u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'ec8d5d7ba063a109f749cd63c7a2b58d'}, u'20_ovirt_provider_ovn_vhostuser_hook': {u'md5': u'a8af653b7386c138b2e6e9738bd6b62c'}, u'10_ovirt_provider_ovn_hook': {u'md5': u'73822988042847bab1ea832a6b9fa837'}}, u'after_get_caps': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'5c3a9ab6e06e039bdd220c0216e45809'}, u'ovirt_provider_ovn_hook': {u'md5': u'4c4b1d2d5460e6a65114ae36cb775df6'}}, u'after_nic_hotplug': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'6226fbc4d1602994828a3904fc1b875d'}}, u'before_vm_migrate_destination': {u'50_vhostmd': {u'md5': u'9206bc390bcbf208b06a8e899581be2d'}}, u'after_device_create': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'6226fbc4d1602994828a3904fc1b875d'}}, u'before_vm_dehibernate': {u'50_vhostmd': {u'md5': u'9206bc390bcbf208b06a8e899581be2d'}}, u'before_nic_hotplug': {u'50_vmfex': {u'md5': u'49caba1a5faadd8efacef966f79bc30a'}, u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'ec8d5d7ba063a109f749cd63c7a2b58d'}, u'20_ovirt_provider_ovn_vhostuser_hook': {u'md5': u'a8af653b7386c138b2e6e9738bd6b62c'}, u'10_ovirt_provider_ovn_hook': {u'md5': u'73822988042847bab1ea832a6b9fa837'}}, u'before_network_setup': {u'50_fcoe': {u'md5': u'28c352339c8beef1e1b05c67d106d062'}}, u'after_vm_destroy': {u'delete_vhostuserclient_hook': {u'md5': u'c2f279cc9483a3f842f6c29df13994c1'}, u'50_vhostmd': {u'md5': u'bdf4802c0521cf1bae08f2b90a9559cf'}}}, u'supportsIPv6': True, u'realtimeKernel': False, u'vmTypes': [u'kvm'], u'liveSnapshot': u'true', u'cpuThreads': u'24', u'kdumpStatus': 1, u'networks': {u'ovirtmgmt': {u'iface': u'ovirtmgmt', u'ipv6autoconf': True, u'addr': u'192.168.0.35', u'dhcpv6': False, u'ipv6addrs': [], u'switch': u'legacy', u'bridged': True, u'southbound': u'bond0', u'dhcpv4': False, u'netmask': u'255.255.255.0', u'ipv4defaultroute': True, u'stp': u'off', u'ipv4addrs': [u'192.168.0.35/24'], u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u'192.168.0.1', u'ports': [u'bond0']}, u'streamingnet': {u'iface': u'streamingnet', u'ipv6autoconf': True, u'addr': u'', u'dhcpv6': False, u'ipv6addrs': [], u'switch': u'legacy', u'bridged': True, u'southbound': u'eno4', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'stp': u'off', u'ipv4addrs': [], u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u'', u'ports': [u'eno4']}}, u'kernelArgs': u'BOOT_IMAGE=/vmlinuz-3.10.0-957.10.1.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=de_DE.UTF-8 intel_iommu=on', u'domain_versions': [0, 2, 3, 4, 5], u'bridges': {u'ovirtmgmt': {u'ipv6autoconf': True, u'addr': u'192.168.0.35', u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'255.255.255.0', u'ipv4defaultroute': True, u'stp': u'off', u'ipv4addrs': [u'192.168.0.35/24'], u'ipv6gateway': u'::', u'gateway': u'192.168.0.1', u'opts': {u'multicast_last_member_count': u'2', u'vlan_protocol': u'0x8100', u'hash_elasticity': u'4', u'multicast_query_response_interval': u'1000', u'group_fwd_mask': u'0x0', u'multicast_snooping': u'1', u'multicast_startup_query_interval': u'3125', u'hello_timer': u'0', u'multicast_querier_interval': u'25500', u'max_age': u'2000', u'hash_max': u'512', u'stp_state': u'0', u'topology_change_detected': u'0', u'priority': u'32768', u'multicast_igmp_version': u'2', u'multicast_membership_interval': u'26000', u'root_path_cost': u'0', u'root_port': u'0', u'multicast_stats_enabled': u'0', u'multicast_startup_query_count': u'2', u'nf_call_iptables': u'0', u'vlan_stats_enabled': u'0', u'hello_time': u'200', u'topology_change': u'0', u'bridge_id': u'8000.e41f13b59416', u'topology_change_timer': u'0', u'ageing_time': u'30000', u'nf_call_ip6tables': u'0', u'multicast_mld_version': u'1', u'gc_timer': u'14536', u'root_id': u'8000.e41f13b59416', u'nf_call_arptables': u'0', u'group_addr': u'1:80:c2:0:0:0', u'multicast_last_member_interval': u'100', u'default_pvid': u'1', u'multicast_query_interval': u'12500', u'multicast_query_use_ifaddr': u'0', u'tcn_timer': u'0', u'multicast_router': u'1', u'vlan_filtering': u'0', u'multicast_querier': u'0', u'forward_delay': u'0'}, u'ports': [u'bond0']}, u'streamingnet': {u'ipv6autoconf': True, u'addr': u'', u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'stp': u'off', u'ipv4addrs': [], u'ipv6gateway': u'::', u'gateway': u'', u'opts': {u'multicast_last_member_count': u'2', u'vlan_protocol': u'0x8100', u'hash_elasticity': u'4', u'multicast_query_response_interval': u'1000', u'group_fwd_mask': u'0x0', u'multicast_snooping': u'1', u'multicast_startup_query_interval': u'3125', u'hello_timer': u'0', u'multicast_querier_interval': u'25500', u'max_age': u'2000', u'hash_max': u'512', u'stp_state': u'0', u'topology_change_detected': u'0', u'priority': u'32768', u'multicast_igmp_version': u'2', u'multicast_membership_interval': u'26000', u'root_path_cost': u'0', u'root_port': u'0', u'multicast_stats_enabled': u'0', u'multicast_startup_query_count': u'2', u'nf_call_iptables': u'0', u'vlan_stats_enabled': u'0', u'hello_time': u'200', u'topology_change': u'0', u'bridge_id': u'8000.5cf3fc3918de', u'topology_change_timer': u'0', u'ageing_time': u'30000', u'nf_call_ip6tables': u'0', u'multicast_mld_version': u'1', u'gc_timer': u'18529', u'root_id': u'8000.5cf3fc3918de', u'nf_call_arptables': u'0', u'group_addr': u'1:80:c2:0:0:0', u'multicast_last_member_interval': u'100', u'default_pvid': u'1', u'multicast_query_interval': u'12500', u'multicast_query_use_ifaddr': u'0', u'tcn_timer': u'0', u'multicast_router': u'1', u'vlan_filtering': u'0', u'multicast_querier': u'0', u'forward_delay': u'0'}, u'ports': [u'eno4']}}, u'uuid': u'2650319A-1C29-11E0-8699-E41F13B59414', u'onlineCpus': u'0,1,2,3,4,5,12,13,14,15,16,17,6,7,8,9,10,11,18,19,20,21,22,23', u'nameservers': [u'192.168.0.1'], u'nics': {u'eno1': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e4:1f:13:b5:94:14', u'ipv6gateway': u'::', u'gateway': u''}, u'eno2': {u'permhwaddr': u'e4:1f:13:b5:94:16', u'ipv6autoconf': False, u'addr': u'', u'speed': 1000, u'dhcpv6': False, u'ipv6addrs': [], u'ad_aggregator_id': u'2', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e4:1f:13:b5:94:16', u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u''}, u'eno3': {u'permhwaddr': u'5c:f3:fc:39:18:dc', u'ipv6autoconf': False, u'addr': u'', u'speed': 1000, u'dhcpv6': False, u'ipv6addrs': [], u'ad_aggregator_id': u'2', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e4:1f:13:b5:94:16', u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u''}, u'eno4': {u'ipv6autoconf': False, u'addr': u'', u'speed': 1000, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'5c:f3:fc:39:18:de', u'ipv6gateway': u'::', u'gateway': u''}, u'ens3f3': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'00:1b:21:8a:a6:cb', u'ipv6gateway': u'::', u'gateway': u''}, u'ens3f2': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'00:1b:21:8a:a6:ca', u'ipv6gateway': u'::', u'gateway': u''}, u'enp0s29f0u2': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e6:1f:13:a8:94:17', u'ipv6gateway': u'::', u'gateway': u''}, u'ens3f0': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'00:1b:21:8a:a6:c8', u'ipv6gateway': u'::', u'gateway': u''}, u'ens3f1': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'00:1b:21:8a:a6:c9', u'ipv6gateway': u'::', u'gateway': u''}}, u'software_revision': u'1', u'hostdevPassthrough': u'true', u'clusterLevels': [u'4.1', u'4.2', u'4.3'], u'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,eagerfpu,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,popcnt,aes,lahf_lm,epb,ssbd,ibrs,ibpb,stibp,tpr_shadow,vnmi,flexpriority,ept,vpid,dtherm,arat,spec_ctrl,intel_stibp,flush_l1d,model_Opteron_G2,model_kvm32,model_coreduo,model_Conroe,model_Nehalem,model_Westmere-IBRS,model_Opteron_G1,model_core2duo,model_Nehalem-IBRS,model_qemu32,model_Penryn,model_pentium2,model_pentium3,model_qemu64,model_Westmere,model_kvm64,model_pentium,model_486', u'kernelFeatures': {u'RETP': 1, u'IBRS': 0, u'PTI': 1}, u'ISCSIInitiatorName': u'iqn.1994-05.com.redhat:5ee45db2f0f', u'netConfigDirty': u'False', u'selinux': {u'mode': u'1'}, u'autoNumaBalancing': 1, u'reservedMem': u'321', u'bondings': {u'bond0': {u'ipv6autoconf': False, u'ad_partner_mac': u'ec:b1:d7:11:34:40', u'speed': 2000, u'dhcpv6': False, u'ipv6addrs': [], u'netmask': u'', u'active_slave': u'', u'ad_aggregator_id': u'2', u'dhcpv4': False, u'switch': u'legacy', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e4:1f:13:b5:94:16', u'slaves': [u'eno2', u'eno3'], u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u'', u'opts': {u'mode': u'4', u'xmit_hash_policy': u'2'}, u'addr': u''}}, u'software_version': u'4.30.11', u'supportedENGINEs': [u'4.1', u'4.2', u'4.3'], u'vncEncrypted': True, u'backupEnabled': False, u'cpuSpeed': u'2793.000', u'numaNodes': {u'1': {u'totalMemory': u'98304', u'cpus': [6, 7, 8, 9, 10, 11, 18, 19, 20, 21, 22, 23]}, u'0': {u'totalMemory': u'98291', u'cpus': [0, 1, 2, 3, 4, 5, 12, 13, 14, 15, 16, 17]}}, u'cpuSockets': u'2', u'vlans': {}, u'version_name': u'Snow Man', u'cpuCores': u'12', u'hostedEngineDeployed': False, u'hugepages': [1048576, 2048], u'guestOverhead': u'65', u'additionalFeatures': [u'libgfapi_supported', u'GLUSTER_SNAPSHOT', u'GLUSTER_GEO_REPLICATION', u'GLUSTER_BRICK_MANAGEMENT'], u'kvmEnabled': u'true', u'memSize': u'193330', u'emulatedMachines': [u'pc-i440fx-rhel7.1.0', u'pc-q35-rhel7.3.0', u'rhel6.3.0', u'pc-i440fx-rhel7.5.0', u'pc-i440fx-rhel7.0.0', u'rhel6.1.0', u'pc-q35-rhel7.6.0', u'pc-i440fx-rhel7.4.0', u'rhel6.6.0', u'pc-q35-rhel7.5.0', u'rhel6.2.0', u'pc', u'pc-i440fx-rhel7.3.0', u'q35', u'pc-i440fx-rhel7.2.0', u'rhel6.4.0', u'pc-q35-rhel7.4.0', u'pc-i440fx-rhel7.6.0', u'rhel6.0.0', u'rhel6.5.0'], u'rngSources': [u'hwrng', u'random'], u'operatingSystem': {u'release': u'6.1810.2.el7.centos', u'pretty_name': u'CentOS Linux 7 (Core)', u'version': u'7', u'name': u'RHEL'}}} from=::ffff:192.168.0.19,42536 (api:54) 2019-04-13 22:07:49,180+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getCapabilities succeeded in 25.43 seconds (__init__:312) 2019-04-13 22:07:49,329+0200 INFO (jsonrpc/2) [api.host] START getHardwareInfo() from=::ffff:192.168.0.19,42536 (api:48) 2019-04-13 22:07:49,381+0200 INFO (jsonrpc/2) [api.host] FINISH getHardwareInfo return={'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'System x3650 M3 -[7945L2G]-', 'systemSerialNumber': '99L8940', 'systemFamily': 'System x', 'systemVersion': '00', 'systemUUID': '2650319a-1c29-11e0-8699-e41f13b59414', 'systemManufacturer': 'IBM'}} from=::ffff:192.168.0.19,42536 (api:54) 2019-04-13 22:07:49,381+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getHardwareInfo succeeded in 0.05 seconds (__init__:312) 2019-04-13 22:07:49,567+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,42536, flow_id=dbfb802 (api:48) 2019-04-13 22:07:49,589+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,42536, flow_id=dbfb802, task_id=9f7c8097-cf80-4f60-ac0b-dc6e617059a3 (api:48) 2019-04-13 22:07:49,589+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={} from=::ffff:192.168.0.19,42536, flow_id=dbfb802, task_id=9f7c8097-cf80-4f60-ac0b-dc6e617059a3 (api:54) 2019-04-13 22:07:49,590+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,42536, flow_id=dbfb802, task_id=82c280c4-c239-4b84-887c-6995ddadcb8f (api:48) 2019-04-13 22:07:49,590+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,42536, flow_id=dbfb802, task_id=82c280c4-c239-4b84-887c-6995ddadcb8f (api:54) 2019-04-13 22:07:49,596+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.05', 'cpuIdle': '99.95'}, '1': {'cpuUser': '2.96', 'nodeIndex': 0, 'cpuSys': '0.83', 'cpuIdle': '96.21'}, '0': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '3': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.05', 'cpuIdle': '99.95'}, '5': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.05', 'cpuIdle': '99.95'}, '4': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '7': {'cpuUser': '0.21', 'nodeIndex': 1, 'cpuSys': '0.05', 'cpuIdle': '99.74'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '9': {'cpuUser': '0.05', 'nodeIndex': 1, 'cpuSys': '0.05', 'cpuIdle': '99.90'}, '8': {'cpuUser': '0.16', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.84'}, '11': {'cpuUser': '0.05', 'nodeIndex': 1, 'cpuSys': '0.05', 'cpuIdle': '99.90'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '13': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '17': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '0.05', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.95'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95876'}, '0': {'memPercent': 3, 'memFree': '95374'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186069.588487, 'rx': '12916', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '16609', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186069.588487, 'rx': '64994', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186069.588487, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '24711', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186069.588487, 'rx': '44635', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '33900', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186069.588487, 'rx': '33900', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186069.588487, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '7081', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186069.588487, 'rx': '47556', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '9528', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186069.588487, 'rx': '17438', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1494', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186069.588487, 'rx': '19282', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186069.588487, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186069.588487, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186069.588487, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186069.588487, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186069.588487, 'rx': '11274', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '34', 'ksmPages': 100, 'elapsedTime': '461.05', 'cpuLoad': '0.16', 'cpuSys': '0.05', 'diskStats': {'/var/log': {'free': '47807'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47807'}}, 'cpuUserVdsmd': '3.69', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191871, 'bootTime': '1555185555', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T20:07:49 GMT', 'cpuUser': '0.15', 'memFree': 191615, 'cpuIdle': '99.81', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '1.09'}} from=::ffff:192.168.0.19,42536, flow_id=dbfb802 (api:54) 2019-04-13 22:07:49,599+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 22:07:49,640+0200 INFO (jsonrpc/4) [api.host] START getCapabilities() from=::ffff:192.168.0.19,42536 (api:48) 2019-04-13 22:07:51,749+0200 INFO (jsonrpc/4) [root] managedvolume not supported: Managed Volume Not Supported. Missing package os-brick.: ('Cannot import os_brick',) (caps:149) 2019-04-13 22:07:52,377+0200 INFO (jsonrpc/4) [root] /usr/libexec/vdsm/hooks/after_get_caps/50_openstacknet: rc=0 err= (hooks:114) 2019-04-13 22:07:52,647+0200 INFO (jsonrpc/4) [root] /usr/libexec/vdsm/hooks/after_get_caps/openstacknet_utils.py: rc=0 err= (hooks:114) 2019-04-13 22:07:53,009+0200 INFO (jsonrpc/4) [root] /usr/libexec/vdsm/hooks/after_get_caps/ovirt_provider_ovn_hook: rc=0 err= (hooks:114) 2019-04-13 22:07:53,010+0200 INFO (jsonrpc/4) [api.host] FINISH getCapabilities return={'status': {'message': 'Done', 'code': 0}, 'info': {u'HBAInventory': {u'iSCSI': [{u'InitiatorName': u'iqn.1994-05.com.redhat:5ee45db2f0f'}], u'FC': [{u'wwpn': u'21000024ff2c1d40', u'wwnn': u'20000024ff2c1d40', u'model': u'QLE2562 - QLogic 8Gb FC Dual-port HBA for System x'}, {u'wwpn': u'21000024ff2c1d41', u'wwnn': u'20000024ff2c1d41', u'model': u'QLE2562 - QLogic 8Gb FC Dual-port HBA for System x'}]}, u'packages2': {u'kernel': {u'release': u'957.10.1.el7.x86_64', u'version': u'3.10.0'}, u'glusterfs-rdma': {u'release': u'1.el7', u'version': u'5.5'}, u'glusterfs-fuse': {u'release': u'1.el7', u'version': u'5.5'}, u'spice-server': {u'release': u'6.el7_6.1', u'version': u'0.14.0'}, u'librbd1': {u'release': u'4.el7', u'version': u'10.2.5'}, u'vdsm': {u'release': u'1.el7', u'version': u'4.30.11'}, u'qemu-kvm': {u'release': u'18.el7_6.3.1', u'version': u'2.12.0'}, u'openvswitch': {u'release': u'3.el7', u'version': u'2.10.1'}, u'libvirt': {u'release': u'10.el7_6.6', u'version': u'4.5.0'}, u'ovirt-hosted-engine-ha': {u'release': u'1.el7', u'version': u'2.3.1'}, u'qemu-img': {u'release': u'18.el7_6.3.1', u'version': u'2.12.0'}, u'mom': {u'release': u'1.el7.centos', u'version': u'0.5.12'}, u'glusterfs': {u'release': u'1.el7', u'version': u'5.5'}, u'glusterfs-cli': {u'release': u'1.el7', u'version': u'5.5'}, u'glusterfs-server': {u'release': u'1.el7', u'version': u'5.5'}, u'glusterfs-geo-replication': {u'release': u'1.el7', u'version': u'5.5'}}, u'numaNodeDistance': {u'1': [21, 10], u'0': [10, 21]}, u'cpuModel': u'Intel(R) Xeon(R) CPU X5660 @ 2.80GHz', u'nestedVirtualization': False, u'liveMerge': u'true', u'hooks': {u'after_vm_start': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'ea0a5a715da8c1badbcda28e8b8fa00e'}}, u'after_network_setup': {u'30_ethtool_options': {u'md5': u'ce1fbad7aa0389e3b06231219140bf0d'}}, u'before_vm_start': {u'50_hostedengine': {u'md5': u'95c810cdcfe4195302a59574a5148289'}, u'50_vhostmd': {u'md5': u'9206bc390bcbf208b06a8e899581be2d'}}, u'after_device_migrate_destination': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'6226fbc4d1602994828a3904fc1b875d'}}, u'before_device_migrate_destination': {u'50_vmfex': {u'md5': u'49caba1a5faadd8efacef966f79bc30a'}}, u'before_device_create': {u'50_vmfex': {u'md5': u'49caba1a5faadd8efacef966f79bc30a'}, u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'ec8d5d7ba063a109f749cd63c7a2b58d'}, u'20_ovirt_provider_ovn_vhostuser_hook': {u'md5': u'a8af653b7386c138b2e6e9738bd6b62c'}, u'10_ovirt_provider_ovn_hook': {u'md5': u'73822988042847bab1ea832a6b9fa837'}}, u'after_get_caps': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'5c3a9ab6e06e039bdd220c0216e45809'}, u'ovirt_provider_ovn_hook': {u'md5': u'4c4b1d2d5460e6a65114ae36cb775df6'}}, u'after_nic_hotplug': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'6226fbc4d1602994828a3904fc1b875d'}}, u'before_vm_migrate_destination': {u'50_vhostmd': {u'md5': u'9206bc390bcbf208b06a8e899581be2d'}}, u'after_device_create': {u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'6226fbc4d1602994828a3904fc1b875d'}}, u'before_vm_dehibernate': {u'50_vhostmd': {u'md5': u'9206bc390bcbf208b06a8e899581be2d'}}, u'before_nic_hotplug': {u'50_vmfex': {u'md5': u'49caba1a5faadd8efacef966f79bc30a'}, u'openstacknet_utils.py': {u'md5': u'1ed38ddf30f8a9c7574589e77e2c0b1f'}, u'50_openstacknet': {u'md5': u'ec8d5d7ba063a109f749cd63c7a2b58d'}, u'20_ovirt_provider_ovn_vhostuser_hook': {u'md5': u'a8af653b7386c138b2e6e9738bd6b62c'}, u'10_ovirt_provider_ovn_hook': {u'md5': u'73822988042847bab1ea832a6b9fa837'}}, u'before_network_setup': {u'50_fcoe': {u'md5': u'28c352339c8beef1e1b05c67d106d062'}}, u'after_vm_destroy': {u'delete_vhostuserclient_hook': {u'md5': u'c2f279cc9483a3f842f6c29df13994c1'}, u'50_vhostmd': {u'md5': u'bdf4802c0521cf1bae08f2b90a9559cf'}}}, u'supportsIPv6': True, u'realtimeKernel': False, u'vmTypes': [u'kvm'], u'liveSnapshot': u'true', u'cpuThreads': u'24', u'kdumpStatus': 1, u'networks': {u'ovirtmgmt': {u'iface': u'ovirtmgmt', u'ipv6autoconf': True, u'addr': u'192.168.0.35', u'dhcpv6': False, u'ipv6addrs': [], u'switch': u'legacy', u'bridged': True, u'southbound': u'bond0', u'dhcpv4': False, u'netmask': u'255.255.255.0', u'ipv4defaultroute': True, u'stp': u'off', u'ipv4addrs': [u'192.168.0.35/24'], u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u'192.168.0.1', u'ports': [u'bond0']}, u'streamingnet': {u'iface': u'streamingnet', u'ipv6autoconf': True, u'addr': u'', u'dhcpv6': False, u'ipv6addrs': [], u'switch': u'legacy', u'bridged': True, u'southbound': u'eno4', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'stp': u'off', u'ipv4addrs': [], u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u'', u'ports': [u'eno4']}}, u'kernelArgs': u'BOOT_IMAGE=/vmlinuz-3.10.0-957.10.1.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=de_DE.UTF-8 intel_iommu=on', u'domain_versions': [0, 2, 3, 4, 5], u'bridges': {u'ovirtmgmt': {u'ipv6autoconf': True, u'addr': u'192.168.0.35', u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'255.255.255.0', u'ipv4defaultroute': True, u'stp': u'off', u'ipv4addrs': [u'192.168.0.35/24'], u'ipv6gateway': u'::', u'gateway': u'192.168.0.1', u'opts': {u'multicast_last_member_count': u'2', u'vlan_protocol': u'0x8100', u'hash_elasticity': u'4', u'multicast_query_response_interval': u'1000', u'group_fwd_mask': u'0x0', u'multicast_snooping': u'1', u'multicast_startup_query_interval': u'3125', u'hello_timer': u'0', u'multicast_querier_interval': u'25500', u'max_age': u'2000', u'hash_max': u'512', u'stp_state': u'0', u'topology_change_detected': u'0', u'priority': u'32768', u'multicast_igmp_version': u'2', u'multicast_membership_interval': u'26000', u'root_path_cost': u'0', u'root_port': u'0', u'multicast_stats_enabled': u'0', u'multicast_startup_query_count': u'2', u'nf_call_iptables': u'0', u'vlan_stats_enabled': u'0', u'hello_time': u'200', u'topology_change': u'0', u'bridge_id': u'8000.e41f13b59416', u'topology_change_timer': u'0', u'ageing_time': u'30000', u'nf_call_ip6tables': u'0', u'multicast_mld_version': u'1', u'gc_timer': u'11953', u'root_id': u'8000.e41f13b59416', u'nf_call_arptables': u'0', u'group_addr': u'1:80:c2:0:0:0', u'multicast_last_member_interval': u'100', u'default_pvid': u'1', u'multicast_query_interval': u'12500', u'multicast_query_use_ifaddr': u'0', u'tcn_timer': u'0', u'multicast_router': u'1', u'vlan_filtering': u'0', u'multicast_querier': u'0', u'forward_delay': u'0'}, u'ports': [u'bond0']}, u'streamingnet': {u'ipv6autoconf': True, u'addr': u'', u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'stp': u'off', u'ipv4addrs': [], u'ipv6gateway': u'::', u'gateway': u'', u'opts': {u'multicast_last_member_count': u'2', u'vlan_protocol': u'0x8100', u'hash_elasticity': u'4', u'multicast_query_response_interval': u'1000', u'group_fwd_mask': u'0x0', u'multicast_snooping': u'1', u'multicast_startup_query_interval': u'3125', u'hello_timer': u'0', u'multicast_querier_interval': u'25500', u'max_age': u'2000', u'hash_max': u'512', u'stp_state': u'0', u'topology_change_detected': u'0', u'priority': u'32768', u'multicast_igmp_version': u'2', u'multicast_membership_interval': u'26000', u'root_path_cost': u'0', u'root_port': u'0', u'multicast_stats_enabled': u'0', u'multicast_startup_query_count': u'2', u'nf_call_iptables': u'0', u'vlan_stats_enabled': u'0', u'hello_time': u'200', u'topology_change': u'0', u'bridge_id': u'8000.5cf3fc3918de', u'topology_change_timer': u'0', u'ageing_time': u'30000', u'nf_call_ip6tables': u'0', u'multicast_mld_version': u'1', u'gc_timer': u'15946', u'root_id': u'8000.5cf3fc3918de', u'nf_call_arptables': u'0', u'group_addr': u'1:80:c2:0:0:0', u'multicast_last_member_interval': u'100', u'default_pvid': u'1', u'multicast_query_interval': u'12500', u'multicast_query_use_ifaddr': u'0', u'tcn_timer': u'0', u'multicast_router': u'1', u'vlan_filtering': u'0', u'multicast_querier': u'0', u'forward_delay': u'0'}, u'ports': [u'eno4']}}, u'uuid': u'2650319A-1C29-11E0-8699-E41F13B59414', u'onlineCpus': u'0,1,2,3,4,5,12,13,14,15,16,17,6,7,8,9,10,11,18,19,20,21,22,23', u'nameservers': [u'192.168.0.1'], u'nics': {u'eno1': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e4:1f:13:b5:94:14', u'ipv6gateway': u'::', u'gateway': u''}, u'eno2': {u'permhwaddr': u'e4:1f:13:b5:94:16', u'ipv6autoconf': False, u'addr': u'', u'speed': 1000, u'dhcpv6': False, u'ipv6addrs': [], u'ad_aggregator_id': u'2', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e4:1f:13:b5:94:16', u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u''}, u'eno3': {u'permhwaddr': u'5c:f3:fc:39:18:dc', u'ipv6autoconf': False, u'addr': u'', u'speed': 1000, u'dhcpv6': False, u'ipv6addrs': [], u'ad_aggregator_id': u'2', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e4:1f:13:b5:94:16', u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u''}, u'eno4': {u'ipv6autoconf': False, u'addr': u'', u'speed': 1000, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'5c:f3:fc:39:18:de', u'ipv6gateway': u'::', u'gateway': u''}, u'ens3f3': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'00:1b:21:8a:a6:cb', u'ipv6gateway': u'::', u'gateway': u''}, u'ens3f2': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'00:1b:21:8a:a6:ca', u'ipv6gateway': u'::', u'gateway': u''}, u'enp0s29f0u2': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e6:1f:13:a8:94:17', u'ipv6gateway': u'::', u'gateway': u''}, u'ens3f0': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'00:1b:21:8a:a6:c8', u'ipv6gateway': u'::', u'gateway': u''}, u'ens3f1': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'00:1b:21:8a:a6:c9', u'ipv6gateway': u'::', u'gateway': u''}}, u'software_revision': u'1', u'hostdevPassthrough': u'true', u'clusterLevels': [u'4.1', u'4.2', u'4.3'], u'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,eagerfpu,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,popcnt,aes,lahf_lm,epb,ssbd,ibrs,ibpb,stibp,tpr_shadow,vnmi,flexpriority,ept,vpid,dtherm,arat,spec_ctrl,intel_stibp,flush_l1d,model_Opteron_G2,model_kvm32,model_coreduo,model_Conroe,model_Nehalem,model_Westmere-IBRS,model_Opteron_G1,model_core2duo,model_Nehalem-IBRS,model_qemu32,model_Penryn,model_pentium2,model_pentium3,model_qemu64,model_Westmere,model_kvm64,model_pentium,model_486', u'kernelFeatures': {u'RETP': 1, u'IBRS': 0, u'PTI': 1}, u'ISCSIInitiatorName': u'iqn.1994-05.com.redhat:5ee45db2f0f', u'netConfigDirty': u'False', u'selinux': {u'mode': u'1'}, u'autoNumaBalancing': 1, u'reservedMem': u'321', u'bondings': {u'bond0': {u'ipv6autoconf': False, u'ad_partner_mac': u'ec:b1:d7:11:34:40', u'speed': 2000, u'dhcpv6': False, u'ipv6addrs': [], u'netmask': u'', u'active_slave': u'', u'ad_aggregator_id': u'2', u'dhcpv4': False, u'switch': u'legacy', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'e4:1f:13:b5:94:16', u'slaves': [u'eno2', u'eno3'], u'mtu': u'1500', u'ipv6gateway': u'::', u'gateway': u'', u'opts': {u'mode': u'4', u'xmit_hash_policy': u'2'}, u'addr': u''}}, u'software_version': u'4.30.11', u'supportedENGINEs': [u'4.1', u'4.2', u'4.3'], u'vncEncrypted': True, u'backupEnabled': False, u'cpuSpeed': u'2793.000', u'numaNodes': {u'1': {u'totalMemory': u'98304', u'cpus': [6, 7, 8, 9, 10, 11, 18, 19, 20, 21, 22, 23]}, u'0': {u'totalMemory': u'98291', u'cpus': [0, 1, 2, 3, 4, 5, 12, 13, 14, 15, 16, 17]}}, u'cpuSockets': u'2', u'vlans': {}, u'version_name': u'Snow Man', 'lastClientIface': 'ovirtmgmt', u'cpuCores': u'12', u'hostedEngineDeployed': False, u'hugepages': [1048576, 2048], u'guestOverhead': u'65', u'additionalFeatures': [u'libgfapi_supported', u'GLUSTER_SNAPSHOT', u'GLUSTER_GEO_REPLICATION', u'GLUSTER_BRICK_MANAGEMENT'], u'kvmEnabled': u'true', u'memSize': u'193330', u'emulatedMachines': [u'pc-i440fx-rhel7.1.0', u'pc-q35-rhel7.3.0', u'rhel6.3.0', u'pc-i440fx-rhel7.5.0', u'pc-i440fx-rhel7.0.0', u'rhel6.1.0', u'pc-q35-rhel7.6.0', u'pc-i440fx-rhel7.4.0', u'rhel6.6.0', u'pc-q35-rhel7.5.0', u'rhel6.2.0', u'pc', u'pc-i440fx-rhel7.3.0', u'q35', u'pc-i440fx-rhel7.2.0', u'rhel6.4.0', u'pc-q35-rhel7.4.0', u'pc-i440fx-rhel7.6.0', u'rhel6.0.0', u'rhel6.5.0'], u'rngSources': [u'hwrng', u'random'], u'operatingSystem': {u'release': u'6.1810.2.el7.centos', u'pretty_name': u'CentOS Linux 7 (Core)', u'version': u'7', u'name': u'RHEL'}}} from=::ffff:192.168.0.19,42536 (api:54) 2019-04-13 22:07:53,027+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getCapabilities succeeded in 3.39 seconds (__init__:312) 2019-04-13 22:07:53,090+0200 INFO (jsonrpc/5) [api.host] START getHardwareInfo() from=::ffff:192.168.0.19,42536 (api:48) 2019-04-13 22:07:53,092+0200 INFO (jsonrpc/5) [api.host] FINISH getHardwareInfo return={'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'System x3650 M3 -[7945L2G]-', 'systemSerialNumber': '99L8940', 'systemFamily': 'System x', 'systemVersion': '00', 'systemUUID': '2650319a-1c29-11e0-8699-e41f13b59414', 'systemManufacturer': 'IBM'}} from=::ffff:192.168.0.19,42536 (api:54) 2019-04-13 22:07:53,093+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getHardwareInfo succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:07:53,505+0200 INFO (jsonrpc/6) [vdsm.api] START connectStorageServer(domType=1, spUUID=u'59ef3a18-002f-02d1-0220-000000000124', conList=[{u'tpgt': u'1', u'id': u'2286311a-c1c9-4dda-bea5-9c78651c6e36', u'connection': u'onode-1.discovery.intern:/mnt/store', u'iqn': u'', u'user': u'', u'ipv6_enabled': u'false', u'protocol_version': u'auto', u'password': '********', u'port': u''}], options=None) from=::ffff:192.168.0.19,42536, flow_id=42fd5658, task_id=ac94eb4c-2f9b-496d-a36f-b9ec2d5eae6c (api:48) 2019-04-13 22:07:53,507+0200 INFO (jsonrpc/6) [storage.StorageServer.MountConnection] Creating directory u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store' (storageServer:168) 2019-04-13 22:07:53,507+0200 INFO (jsonrpc/6) [storage.fileUtils] Creating directory: /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store mode: None (fileUtils:199) 2019-04-13 22:07:53,507+0200 INFO (jsonrpc/6) [storage.Mount] mounting onode-1.discovery.intern:/mnt/store at /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store (mount:204) 2019-04-13 22:07:53,877+0200 INFO (jsonrpc/6) [IOProcessClient] (Global) Starting client (__init__:308) 2019-04-13 22:07:53,886+0200 INFO (ioprocess/14764) [IOProcess] (Global) Starting ioprocess (__init__:434) 2019-04-13 22:07:53,903+0200 INFO (jsonrpc/6) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 0, 'id': u'2286311a-c1c9-4dda-bea5-9c78651c6e36'}]} from=::ffff:192.168.0.19,42536, flow_id=42fd5658, task_id=ac94eb4c-2f9b-496d-a36f-b9ec2d5eae6c (api:54) 2019-04-13 22:07:53,904+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 0.40 seconds (__init__:312) 2019-04-13 22:07:53,936+0200 INFO (jsonrpc/0) [vdsm.api] START connectStorageServer(domType=1, spUUID=u'59ef3a18-002f-02d1-0220-000000000124', conList=[{u'tpgt': u'1', u'id': u'b4f4bcf2-e104-468e-b330-ed731ec8ef6c', u'connection': u'nas.discovery.intern:/mnt/d2600/vms', u'iqn': u'', u'user': u'', u'ipv6_enabled': u'false', u'protocol_version': u'auto', u'password': '********', u'port': u''}], options=None) from=::ffff:192.168.0.19,42536, flow_id=42fd5658, task_id=a8292b3b-f18a-4dd8-8136-73c88fd9443c (api:48) 2019-04-13 22:07:53,937+0200 INFO (jsonrpc/0) [storage.StorageServer.MountConnection] Creating directory u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms' (storageServer:168) 2019-04-13 22:07:53,937+0200 INFO (jsonrpc/0) [storage.fileUtils] Creating directory: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms mode: None (fileUtils:199) 2019-04-13 22:07:53,938+0200 INFO (jsonrpc/0) [storage.Mount] mounting nas.discovery.intern:/mnt/d2600/vms at /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms (mount:204) 2019-04-13 22:07:54,115+0200 INFO (jsonrpc/0) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 0, 'id': u'b4f4bcf2-e104-468e-b330-ed731ec8ef6c'}]} from=::ffff:192.168.0.19,42536, flow_id=42fd5658, task_id=a8292b3b-f18a-4dd8-8136-73c88fd9443c (api:54) 2019-04-13 22:07:54,116+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 0.18 seconds (__init__:312) 2019-04-13 22:07:54,137+0200 INFO (jsonrpc/1) [vdsm.api] START connectStorageServer(domType=1, spUUID=u'59ef3a18-002f-02d1-0220-000000000124', conList=[{u'tpgt': u'1', u'id': u'07dd9ff2-afa1-4a33-b401-935ea5b8c9db', u'connection': u'nas.discovery.intern:/mnt/d2600/isos', u'iqn': u'', u'user': u'', u'ipv6_enabled': u'false', u'protocol_version': u'auto', u'password': '********', u'port': u''}], options=None) from=::ffff:192.168.0.19,42536, flow_id=42fd5658, task_id=7c5775f1-00bd-4df3-beea-704e505725d5 (api:48) 2019-04-13 22:07:54,139+0200 INFO (jsonrpc/1) [storage.StorageServer.MountConnection] Creating directory u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos' (storageServer:168) 2019-04-13 22:07:54,139+0200 INFO (jsonrpc/1) [storage.fileUtils] Creating directory: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos mode: None (fileUtils:199) 2019-04-13 22:07:54,139+0200 INFO (jsonrpc/1) [storage.Mount] mounting nas.discovery.intern:/mnt/d2600/isos at /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos (mount:204) 2019-04-13 22:07:54,257+0200 INFO (jsonrpc/1) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 0, 'id': u'07dd9ff2-afa1-4a33-b401-935ea5b8c9db'}]} from=::ffff:192.168.0.19,42536, flow_id=42fd5658, task_id=7c5775f1-00bd-4df3-beea-704e505725d5 (api:54) 2019-04-13 22:07:54,258+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 0.12 seconds (__init__:312) 2019-04-13 22:07:54,277+0200 INFO (jsonrpc/7) [vdsm.api] START connectStorageServer(domType=1, spUUID=u'59ef3a18-002f-02d1-0220-000000000124', conList=[{u'tpgt': u'1', u'id': u'858731bf-2fb1-45a7-a693-235f675a667b', u'connection': u'nas.discovery.intern:/mnt/d2600/sicherung/vmBackups', u'iqn': u'', u'user': u'', u'ipv6_enabled': u'false', u'protocol_version': u'auto', u'password': '********', u'port': u''}], options=None) from=::ffff:192.168.0.19,42536, flow_id=42fd5658, task_id=7bfeeb28-29fc-4ede-9efd-45bfcea76118 (api:48) 2019-04-13 22:07:54,279+0200 INFO (jsonrpc/7) [storage.StorageServer.MountConnection] Creating directory u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups' (storageServer:168) 2019-04-13 22:07:54,279+0200 INFO (jsonrpc/7) [storage.fileUtils] Creating directory: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups mode: None (fileUtils:199) 2019-04-13 22:07:54,280+0200 INFO (jsonrpc/7) [storage.Mount] mounting nas.discovery.intern:/mnt/d2600/sicherung/vmBackups at /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups (mount:204) 2019-04-13 22:07:54,412+0200 INFO (jsonrpc/7) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 0, 'id': u'858731bf-2fb1-45a7-a693-235f675a667b'}]} from=::ffff:192.168.0.19,42536, flow_id=42fd5658, task_id=7bfeeb28-29fc-4ede-9efd-45bfcea76118 (api:54) 2019-04-13 22:07:54,413+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 0.13 seconds (__init__:312) 2019-04-13 22:07:54,482+0200 INFO (jsonrpc/2) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,42536, flow_id=42fd5658, task_id=b80fb3ee-9ef5-4b91-aa8f-53526086112b (api:48) 2019-04-13 22:07:54,483+0200 INFO (jsonrpc/2) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 22:07:54,483+0200 INFO (jsonrpc/2) [storage.StoragePool] updating pool 59ef3a18-002f-02d1-0220-000000000124 backend from type NoneType instance 0x7fdb9b9e5dc0 to type StoragePoolMemoryBackend instance 0x7fdb206dfa48 (sp:157) 2019-04-13 22:07:54,483+0200 INFO (jsonrpc/2) [storage.StoragePool] Connect host #1 to the storage pool 59ef3a18-002f-02d1-0220-000000000124 with master domain: 6a4cde72-6a6a-499e-9b6e-d490d91ca19e (ver = 47) (sp:688) 2019-04-13 22:07:54,734+0200 INFO (itmap/0) [IOProcessClient] (/onode-1.discovery.intern:_mnt_store) Starting client (__init__:308) 2019-04-13 22:07:54,743+0200 INFO (ioprocess/14851) [IOProcess] (/onode-1.discovery.intern:_mnt_store) Starting ioprocess (__init__:434) 2019-04-13 22:07:54,744+0200 INFO (itmap/1) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_vms) Starting client (__init__:308) 2019-04-13 22:07:54,751+0200 INFO (itmap/2) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_isos) Starting client (__init__:308) 2019-04-13 22:07:54,754+0200 INFO (ioprocess/14860) [IOProcess] (/nas.discovery.intern:_mnt_d2600_vms) Starting ioprocess (__init__:434) 2019-04-13 22:07:54,764+0200 INFO (itmap/3) [IOProcessClient] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting client (__init__:308) 2019-04-13 22:07:54,767+0200 INFO (ioprocess/14867) [IOProcess] (/nas.discovery.intern:_mnt_d2600_isos) Starting ioprocess (__init__:434) 2019-04-13 22:07:54,773+0200 INFO (jsonrpc/2) [IOProcessClient] (6a4cde72-6a6a-499e-9b6e-d490d91ca19e) Starting client (__init__:308) 2019-04-13 22:07:54,777+0200 INFO (ioprocess/14874) [IOProcess] (/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups) Starting ioprocess (__init__:434) 2019-04-13 22:07:54,784+0200 INFO (itmap/4) [IOProcessClient] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting client (__init__:308) 2019-04-13 22:07:54,787+0200 INFO (ioprocess/14881) [IOProcess] (6a4cde72-6a6a-499e-9b6e-d490d91ca19e) Starting ioprocess (__init__:434) 2019-04-13 22:07:54,794+0200 INFO (ioprocess/14888) [IOProcess] (/glusterSD/onode-2.discovery.intern:vol-node-2) Starting ioprocess (__init__:434) 2019-04-13 22:07:54,832+0200 INFO (jsonrpc/2) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 22:07:54,833+0200 INFO (jsonrpc/2) [storage.Monitor] Start monitoring 9c3f06cf-7475-448e-819b-f4f52fa7d782 (monitor:183) 2019-04-13 22:07:54,833+0200 INFO (jsonrpc/2) [storage.Monitor] Start monitoring 42bb6772-12f9-4bf2-9245-43caa41c5a2a (monitor:183) 2019-04-13 22:07:54,835+0200 INFO (jsonrpc/2) [storage.Monitor] Start monitoring 93f02ede-ec5c-4641-8e97-2949f98e856d (monitor:183) 2019-04-13 22:07:54,836+0200 INFO (jsonrpc/2) [storage.Monitor] Start monitoring 6a4cde72-6a6a-499e-9b6e-d490d91ca19e (monitor:183) 2019-04-13 22:07:54,839+0200 INFO (jsonrpc/2) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 22:07:54,839+0200 INFO (jsonrpc/2) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 22:07:54,845+0200 INFO (monitor/6a4cde7) [storage.check] Start checking u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/metadata' (check:121) 2019-04-13 22:07:54,871+0200 INFO (monitor/6a4cde7) [storage.Monitor] Domain 6a4cde72-6a6a-499e-9b6e-d490d91ca19e became VALID (monitor:470) 2019-04-13 22:07:54,873+0200 INFO (monitor/9c3f06c) [IOProcessClient] (9c3f06cf-7475-448e-819b-f4f52fa7d782) Starting client (__init__:308) 2019-04-13 22:07:54,877+0200 INFO (monitor/6a4cde7) [storage.SANLock] Acquiring host id for domain 6a4cde72-6a6a-499e-9b6e-d490d91ca19e (id=1, async=True) (clusterlock:294) 2019-04-13 22:07:54,887+0200 INFO (event/1) [vds] vmContainerLock acquired (clientIF:243) 2019-04-13 22:07:54,890+0200 INFO (monitor/42bb677) [IOProcessClient] (42bb6772-12f9-4bf2-9245-43caa41c5a2a) Starting client (__init__:308) 2019-04-13 22:07:54,895+0200 INFO (ioprocess/14915) [IOProcess] (9c3f06cf-7475-448e-819b-f4f52fa7d782) Starting ioprocess (__init__:434) 2019-04-13 22:07:54,906+0200 INFO (monitor/93f02ed) [IOProcessClient] (93f02ede-ec5c-4641-8e97-2949f98e856d) Starting client (__init__:308) 2019-04-13 22:07:54,910+0200 INFO (ioprocess/14927) [IOProcess] (42bb6772-12f9-4bf2-9245-43caa41c5a2a) Starting ioprocess (__init__:434) 2019-04-13 22:07:54,919+0200 INFO (ioprocess/14934) [IOProcess] (93f02ede-ec5c-4641-8e97-2949f98e856d) Starting ioprocess (__init__:434) 2019-04-13 22:07:54,970+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 22:07:54,970+0200 INFO (monitor/42bb677) [storage.check] Start checking u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/dom_md/metadata' (check:121) 2019-04-13 22:07:54,976+0200 INFO (monitor/42bb677) [storage.Monitor] Host id for domain 42bb6772-12f9-4bf2-9245-43caa41c5a2a successfully acquired (id: 1) (monitor:448) 2019-04-13 22:07:54,985+0200 INFO (check/loop) [storage.Monitor] Domain 42bb6772-12f9-4bf2-9245-43caa41c5a2a became VALID (monitor:470) 2019-04-13 22:07:54,987+0200 INFO (event/4) [vds] vmContainerLock acquired (clientIF:243) 2019-04-13 22:07:55,012+0200 WARN (jsonrpc/2) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 22:07:55,016+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 22:07:55,017+0200 INFO (jsonrpc/2) [storage.StoragePool] Creating symlink from /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1247) 2019-04-13 22:07:55,017+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 22:07:55,017+0200 INFO (jsonrpc/2) [storage.StoragePool] Creating symlink from /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1247) 2019-04-13 22:07:55,017+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 22:07:55,017+0200 INFO (jsonrpc/2) [storage.StoragePool] Creating symlink from /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1247) 2019-04-13 22:07:55,017+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 22:07:55,018+0200 INFO (jsonrpc/2) [storage.StoragePool] Creating symlink from /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1247) 2019-04-13 22:07:55,018+0200 INFO (jsonrpc/2) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 22:07:55,018+0200 INFO (jsonrpc/2) [storage.StoragePool] Creating symlink from /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1247) 2019-04-13 22:07:55,037+0200 INFO (jsonrpc/2) [storage.MailBox.HsmMailMonitor] HSM_MailMonitor sending mail to SPM - ['/usr/bin/dd', 'of=/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd/dom_md/inbox', 'iflag=fullblock', 'oflag=direct', 'conv=notrunc', 'bs=4096', 'count=1', 'seek=1'] (mailbox:381) 2019-04-13 22:07:55,044+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,42536 (api:48) 2019-04-13 22:07:55,045+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,42536 (api:54) 2019-04-13 22:07:55,045+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:312) 2019-04-13 22:07:55,050+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 22:07:55,051+0200 INFO (monitor/9c3f06c) [storage.check] Start checking u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/dom_md/metadata' (check:121) 2019-04-13 22:07:55,055+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 22:07:55,056+0200 INFO (monitor/93f02ed) [storage.check] Start checking u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d/dom_md/metadata' (check:121) 2019-04-13 22:07:55,060+0200 INFO (monitor/93f02ed) [storage.Monitor] Host id for domain 93f02ede-ec5c-4641-8e97-2949f98e856d successfully acquired (id: 1) (monitor:448) 2019-04-13 22:07:55,077+0200 INFO (check/loop) [storage.Monitor] Domain 9c3f06cf-7475-448e-819b-f4f52fa7d782 became VALID (monitor:470) 2019-04-13 22:07:55,079+0200 INFO (check/loop) [storage.Monitor] Domain 93f02ede-ec5c-4641-8e97-2949f98e856d became VALID (monitor:470) 2019-04-13 22:07:55,080+0200 INFO (event/7) [vds] vmContainerLock acquired (clientIF:243) 2019-04-13 22:07:55,081+0200 INFO (event/10) [vds] vmContainerLock acquired (clientIF:243) 2019-04-13 22:07:55,082+0200 INFO (jsonrpc/2) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,42536, flow_id=42fd5658, task_id=b80fb3ee-9ef5-4b91-aa8f-53526086112b (api:54) 2019-04-13 22:07:55,083+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.60 seconds (__init__:312) 2019-04-13 22:07:55,085+0200 INFO (event/2) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 22:07:55,085+0200 INFO (event/2) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 22:07:55,087+0200 INFO (event/5) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 22:07:55,088+0200 INFO (event/8) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 22:07:55,089+0200 INFO (event/11) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 22:07:55,097+0200 INFO (jsonrpc/4) [api.host] START getStats() from=::ffff:192.168.0.19,42536, flow_id=42fd5658 (api:48) 2019-04-13 22:07:55,118+0200 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,42536, flow_id=42fd5658, task_id=d2ee2bc1-0e1d-4b53-9231-9d9578d94736 (api:48) 2019-04-13 22:07:55,118+0200 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000804233', 'lastCheck': '0.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000554619', 'lastCheck': '0.1', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000676994', 'lastCheck': '0.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000347955', 'lastCheck': '0.2', 'valid': True}} from=::ffff:192.168.0.19,42536, flow_id=42fd5658, task_id=d2ee2bc1-0e1d-4b53-9231-9d9578d94736 (api:54) 2019-04-13 22:07:55,119+0200 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,42536, flow_id=42fd5658, task_id=becc03a5-de27-40e8-abc5-a3f373f52684 (api:48) 2019-04-13 22:07:55,119+0200 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,42536, flow_id=42fd5658, task_id=becc03a5-de27-40e8-abc5-a3f373f52684 (api:54) 2019-04-13 22:07:55,126+0200 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.05', 'cpuIdle': '99.95'}, '1': {'cpuUser': '2.96', 'nodeIndex': 0, 'cpuSys': '0.83', 'cpuIdle': '96.21'}, '0': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '3': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.05', 'cpuIdle': '99.95'}, '5': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.05', 'cpuIdle': '99.95'}, '4': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '7': {'cpuUser': '0.21', 'nodeIndex': 1, 'cpuSys': '0.05', 'cpuIdle': '99.74'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '9': {'cpuUser': '0.05', 'nodeIndex': 1, 'cpuSys': '0.05', 'cpuIdle': '99.90'}, '8': {'cpuUser': '0.16', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.84'}, '11': {'cpuUser': '0.05', 'nodeIndex': 1, 'cpuSys': '0.05', 'cpuIdle': '99.90'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '13': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '17': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '0.05', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.95'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95876'}, '0': {'memPercent': 3, 'memFree': '95374'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000804233', 'lastCheck': '0.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000676994', 'lastCheck': '0.0', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000347955', 'lastCheck': '0.2', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000554619', 'lastCheck': '0.1', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186075.117793, 'rx': '13046', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '94832', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186075.117793, 'rx': '105510', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186075.117793, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '104058', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186075.117793, 'rx': '122166', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '89220', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186075.117793, 'rx': '89220', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186075.117793, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '7761', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186075.117793, 'rx': '88072', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '87071', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186075.117793, 'rx': '17438', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1494', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186075.117793, 'rx': '19410', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186075.117793, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186075.117793, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186075.117793, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186075.117793, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186075.117793, 'rx': '11412', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '34', 'ksmPages': 100, 'elapsedTime': '466.58', 'cpuLoad': '0.16', 'cpuSys': '0.05', 'diskStats': {'/var/log': {'free': '47807'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47807'}}, 'cpuUserVdsmd': '3.69', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191842, 'bootTime': '1555185555', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T20:07:55 GMT', 'cpuUser': '0.15', 'memFree': 191586, 'cpuIdle': '99.81', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '1.09'}} from=::ffff:192.168.0.19,42536, flow_id=42fd5658 (api:54) 2019-04-13 22:07:55,129+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 22:07:55,478+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.setMOMPolicyParameters succeeded in 0.06 seconds (__init__:312) 2019-04-13 22:07:55,549+0200 INFO (jsonrpc/6) [api.host] START hostdevListByCaps(caps=None) from=::ffff:192.168.0.19,42536, flow_id=4a97a86d (api:48) 2019-04-13 22:07:55,858+0200 INFO (jsonrpc/6) [api.host] FINISH hostdevListByCaps return={'status': {'message': 'Done', 'code': 0}, 'deviceList': {'net_eno1_e4_1f_13_b5_94_14': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'eno1', 'parent': 'pci_0000_0b_00_0'}}, 'pci_0000_00_1d_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB UHCI Controller #3', 'vendor': 'Intel Corporation', 'product_id': '0x3a36', 'parent': 'computer', 'address': {'slot': '29', 'bus': '0', 'domain': '0', 'function': '2'}, 'iommu_group': '14', 'vendor_id': '0x8086', 'driver': 'uhci_hcd'}}, 'pci_0000_00_1d_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB UHCI Controller #2', 'vendor': 'Intel Corporation', 'product_id': '0x3a35', 'parent': 'computer', 'address': {'slot': '29', 'bus': '0', 'domain': '0', 'function': '1'}, 'iommu_group': '14', 'vendor_id': '0x8086', 'driver': 'uhci_hcd'}}, 'pci_0000_00_1d_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB UHCI Controller #1', 'vendor': 'Intel Corporation', 'product_id': '0x3a34', 'parent': 'computer', 'address': {'slot': '29', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '14', 'vendor_id': '0x8086', 'driver': 'uhci_hcd'}}, 'pci_0000_00_1d_7': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB2 EHCI Controller #1', 'vendor': 'Intel Corporation', 'product_id': '0x3a3a', 'parent': 'computer', 'address': {'slot': '29', 'bus': '0', 'domain': '0', 'function': '7'}, 'iommu_group': '14', 'vendor_id': '0x8086', 'driver': 'ehci-pci'}}, 'usb_usb6': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '1.1 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0001', 'parent': 'pci_0000_00_1d_1', 'address': {'device': '1', 'bus': '6'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'pci_0000_00_15_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 Trusted Execution Technology Registers', 'vendor': 'Intel Corporation', 'product_id': '0x342f', 'parent': 'computer', 'address': {'slot': '21', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '10', 'vendor_id': '0x8086'}}, 'usb_2_5_1_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'usbhid', 'parent': 'usb_2_5_1'}}, 'block_sda_3600605b002f69270217fb60a16a05ccf': {'params': {'capability': 'storage', 'is_assignable': 'true', 'product': 'ServeRAID M5015', 'vendor': 'IBM', 'parent': 'scsi_0_2_0_0'}}, 'pci_0000_10_00_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'NetXtreme II BCM5709 Gigabit Ethernet', 'vendor': 'Broadcom Inc. and subsidiaries', 'product_id': '0x1639', 'parent': 'pci_0000_00_02_0', 'address': {'slot': '0', 'bus': '16', 'domain': '0', 'function': '1'}, 'iommu_group': '18', 'vendor_id': '0x14e4', 'driver': 'bnx2'}}, 'pci_0000_10_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'NetXtreme II BCM5709 Gigabit Ethernet', 'vendor': 'Broadcom Inc. and subsidiaries', 'product_id': '0x1639', 'parent': 'pci_0000_00_02_0', 'address': {'slot': '0', 'bus': '16', 'domain': '0', 'function': '0'}, 'iommu_group': '18', 'vendor_id': '0x14e4', 'driver': 'bnx2'}}, 'pci_0000_fe_06_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control', 'vendor': 'Intel Corporation', 'product_id': '0x2db3', 'parent': 'computer', 'address': {'slot': '6', 'bus': '254', 'domain': '0', 'function': '3'}, 'iommu_group': '29', 'vendor_id': '0x8086'}}, 'pci_0000_fe_06_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Rank', 'vendor': 'Intel Corporation', 'product_id': '0x2db2', 'parent': 'computer', 'address': {'slot': '6', 'bus': '254', 'domain': '0', 'function': '2'}, 'iommu_group': '29', 'vendor_id': '0x8086'}}, 'pci_0000_00_11_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500 Physical and Link Layer Registers Port 1', 'vendor': 'Intel Corporation', 'product_id': '0x3427', 'parent': 'computer', 'address': {'slot': '17', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '8', 'vendor_id': '0x8086'}}, 'pci_0000_00_11_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500 Routing & Protocol Layer Register Port 1', 'vendor': 'Intel Corporation', 'product_id': '0x3428', 'parent': 'computer', 'address': {'slot': '17', 'bus': '0', 'domain': '0', 'function': '1'}, 'iommu_group': '8', 'vendor_id': '0x8086'}}, 'usb_2_6_4': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '4-Port USB 2.0 Hub', 'vendor': 'Realtek Semiconductor Corp.', 'product_id': '0x5411', 'parent': 'usb_2_6', 'address': {'device': '6', 'bus': '2'}, 'vendor_id': '0x0bda', 'driver': 'usb'}}, 'net_ens3f1_00_1b_21_8a_a6_c9': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'ens3f1', 'parent': 'pci_0000_24_00_1'}}, 'net_enp0s29f0u2_e6_1f_13_a8_94_17': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'enp0s29f0u2', 'parent': 'usb_5_2_1_0'}}, 'pci_0000_24_00_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82580 Gigabit Network Connection', 'vendor': 'Intel Corporation', 'product_id': '0x150e', 'parent': 'pci_0000_00_09_0', 'address': {'slot': '0', 'bus': '36', 'domain': '0', 'function': '3'}, 'iommu_group': '23', 'vendor_id': '0x8086', 'driver': 'igb'}}, 'pci_0000_00_1f_5': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) 2 port SATA IDE Controller #2', 'vendor': 'Intel Corporation', 'product_id': '0x3a26', 'parent': 'computer', 'address': {'slot': '31', 'bus': '0', 'domain': '0', 'function': '5'}, 'iommu_group': '16', 'vendor_id': '0x8086', 'driver': 'ata_piix'}}, 'pci_0000_24_00_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82580 Gigabit Network Connection', 'vendor': 'Intel Corporation', 'product_id': '0x150e', 'parent': 'pci_0000_00_09_0', 'address': {'slot': '0', 'bus': '36', 'domain': '0', 'function': '2'}, 'iommu_group': '22', 'vendor_id': '0x8086', 'driver': 'igb'}}, 'pci_0000_00_1f_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JIB (ICH10) LPC Interface Controller', 'vendor': 'Intel Corporation', 'product_id': '0x3a18', 'parent': 'computer', 'address': {'slot': '31', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '16', 'vendor_id': '0x8086', 'driver': 'lpc_ich'}}, 'pci_0000_00_1f_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) SMBus Controller', 'vendor': 'Intel Corporation', 'product_id': '0x3a30', 'parent': 'computer', 'address': {'slot': '31', 'bus': '0', 'domain': '0', 'function': '3'}, 'iommu_group': '16', 'vendor_id': '0x8086', 'driver': 'i801_smbus'}}, 'pci_0000_24_00_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82580 Gigabit Network Connection', 'vendor': 'Intel Corporation', 'product_id': '0x150e', 'parent': 'pci_0000_00_09_0', 'address': {'slot': '0', 'bus': '36', 'domain': '0', 'function': '1'}, 'iommu_group': '21', 'vendor_id': '0x8086', 'driver': 'igb'}}, 'pci_0000_00_1f_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) 4 port SATA IDE Controller #1', 'vendor': 'Intel Corporation', 'product_id': '0x3a20', 'parent': 'computer', 'address': {'slot': '31', 'bus': '0', 'domain': '0', 'function': '2'}, 'iommu_group': '16', 'vendor_id': '0x8086', 'driver': 'ata_piix'}}, 'drm_card0': {'params': {'capability': 'drm', 'is_assignable': 'true', 'parent': 'pci_0000_07_00_0'}}, 'pci_0000_00_1e_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '82801 PCI Bridge', 'vendor': 'Intel Corporation', 'product_id': '0x244e', 'parent': 'computer', 'address': {'slot': '30', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '15', 'vendor_id': '0x8086'}}, 'computer': {'params': {'capability': 'system', 'is_assignable': 'true', 'product': 'System x3650 M3 -[7945L2G]-'}}, 'pci_0000_07_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'MGA G200EV', 'vendor': 'Matrox Electronics Systems Ltd.', 'product_id': '0x0530', 'parent': 'pci_0000_06_00_0', 'address': {'slot': '0', 'bus': '7', 'domain': '0', 'function': '0'}, 'iommu_group': '13', 'vendor_id': '0x102b', 'driver': 'mgag200'}}, 'pci_0000_00_1a_7': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB2 EHCI Controller #2', 'vendor': 'Intel Corporation', 'product_id': '0x3a3c', 'parent': 'computer', 'address': {'slot': '26', 'bus': '0', 'domain': '0', 'function': '7'}, 'iommu_group': '12', 'vendor_id': '0x8086', 'driver': 'ehci-pci'}}, 'pci_0000_00_1a_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB UHCI Controller #4', 'vendor': 'Intel Corporation', 'product_id': '0x3a37', 'parent': 'computer', 'address': {'slot': '26', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '12', 'vendor_id': '0x8086', 'driver': 'uhci_hcd'}}, 'pci_0000_00_1a_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82801JI (ICH10 Family) USB UHCI Controller #5', 'vendor': 'Intel Corporation', 'product_id': '0x3a38', 'parent': 'computer', 'address': {'slot': '26', 'bus': '0', 'domain': '0', 'function': '1'}, 'iommu_group': '12', 'vendor_id': '0x8086', 'driver': 'uhci_hcd'}}, 'pci_0000_ff_05_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Address', 'vendor': 'Intel Corporation', 'product_id': '0x2da9', 'parent': 'computer', 'address': {'slot': '5', 'bus': '255', 'domain': '0', 'function': '1'}, 'iommu_group': '34', 'vendor_id': '0x8086'}}, 'pci_0000_ff_05_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Control', 'vendor': 'Intel Corporation', 'product_id': '0x2da8', 'parent': 'computer', 'address': {'slot': '5', 'bus': '255', 'domain': '0', 'function': '0'}, 'iommu_group': '34', 'vendor_id': '0x8086'}}, 'pci_0000_ff_05_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control', 'vendor': 'Intel Corporation', 'product_id': '0x2dab', 'parent': 'computer', 'address': {'slot': '5', 'bus': '255', 'domain': '0', 'function': '3'}, 'iommu_group': '34', 'vendor_id': '0x8086'}}, 'pci_0000_ff_05_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Rank', 'vendor': 'Intel Corporation', 'product_id': '0x2daa', 'parent': 'computer', 'address': {'slot': '5', 'bus': '255', 'domain': '0', 'function': '2'}, 'iommu_group': '34', 'vendor_id': '0x8086'}}, 'usb_4_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb4'}}, 'usb_3_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb3'}}, 'net_eno3_e4_1f_13_b5_94_16': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'eno3', 'parent': 'pci_0000_10_00_0'}}, 'pci_0000_00_1c_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '82801JI (ICH10 Family) PCI Express Root Port 1', 'vendor': 'Intel Corporation', 'product_id': '0x3a40', 'parent': 'computer', 'address': {'slot': '28', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '13', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'pci_0000_00_1c_4': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '82801JI (ICH10 Family) PCI Express Root Port 5', 'vendor': 'Intel Corporation', 'product_id': '0x3a48', 'parent': 'computer', 'address': {'slot': '28', 'bus': '0', 'domain': '0', 'function': '4'}, 'iommu_group': '13', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'usb_6_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb6'}}, 'pci_0000_ff_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QuickPath Architecture Generic Non-core Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2c70', 'parent': 'computer', 'address': {'slot': '0', 'bus': '255', 'domain': '0', 'function': '0'}, 'iommu_group': '30', 'vendor_id': '0x8086'}}, 'pci_0000_ff_00_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QuickPath Architecture System Address Decoder', 'vendor': 'Intel Corporation', 'product_id': '0x2d81', 'parent': 'computer', 'address': {'slot': '0', 'bus': '255', 'domain': '0', 'function': '1'}, 'iommu_group': '30', 'vendor_id': '0x8086'}}, 'net_eno4_5c_f3_fc_39_18_de': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'eno4', 'parent': 'pci_0000_10_00_1'}}, 'net_ens3f3_00_1b_21_8a_a6_cb': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'ens3f3', 'parent': 'pci_0000_24_00_3'}}, 'pci_0000_fe_04_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Address', 'vendor': 'Intel Corporation', 'product_id': '0x2da1', 'parent': 'computer', 'address': {'slot': '4', 'bus': '254', 'domain': '0', 'function': '1'}, 'iommu_group': '27', 'vendor_id': '0x8086'}}, 'pci_0000_fe_04_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Control', 'vendor': 'Intel Corporation', 'product_id': '0x2da0', 'parent': 'computer', 'address': {'slot': '4', 'bus': '254', 'domain': '0', 'function': '0'}, 'iommu_group': '27', 'vendor_id': '0x8086'}}, 'pci_0000_fe_04_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control', 'vendor': 'Intel Corporation', 'product_id': '0x2da3', 'parent': 'computer', 'address': {'slot': '4', 'bus': '254', 'domain': '0', 'function': '3'}, 'iommu_group': '27', 'vendor_id': '0x8086'}}, 'pci_0000_fe_04_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Rank', 'vendor': 'Intel Corporation', 'product_id': '0x2da2', 'parent': 'computer', 'address': {'slot': '4', 'bus': '254', 'domain': '0', 'function': '2'}, 'iommu_group': '27', 'vendor_id': '0x8086'}}, 'usb_2_5_3_1_1_1': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'usbhid', 'parent': 'usb_2_5_3_1'}}, 'usb_2_5_3_1_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'usbhid', 'parent': 'usb_2_5_3_1'}}, 'usb_1_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb1'}}, 'pci_0000_ff_06_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Rank', 'vendor': 'Intel Corporation', 'product_id': '0x2db2', 'parent': 'computer', 'address': {'slot': '6', 'bus': '255', 'domain': '0', 'function': '2'}, 'iommu_group': '35', 'vendor_id': '0x8086'}}, 'pci_0000_ff_06_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control', 'vendor': 'Intel Corporation', 'product_id': '0x2db3', 'parent': 'computer', 'address': {'slot': '6', 'bus': '255', 'domain': '0', 'function': '3'}, 'iommu_group': '35', 'vendor_id': '0x8086'}}, 'pci_0000_ff_06_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Control', 'vendor': 'Intel Corporation', 'product_id': '0x2db0', 'parent': 'computer', 'address': {'slot': '6', 'bus': '255', 'domain': '0', 'function': '0'}, 'iommu_group': '35', 'vendor_id': '0x8086'}}, 'pci_0000_ff_06_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Address', 'vendor': 'Intel Corporation', 'product_id': '0x2db1', 'parent': 'computer', 'address': {'slot': '6', 'bus': '255', 'domain': '0', 'function': '1'}, 'iommu_group': '35', 'vendor_id': '0x8086'}}, 'pci_0000_fe_02_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Mirror Port Link 1', 'vendor': 'Intel Corporation', 'product_id': '0x2d93', 'parent': 'computer', 'address': {'slot': '2', 'bus': '254', 'domain': '0', 'function': '3'}, 'iommu_group': '25', 'vendor_id': '0x8086'}}, 'pci_0000_fe_02_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Mirror Port Link 0', 'vendor': 'Intel Corporation', 'product_id': '0x2d92', 'parent': 'computer', 'address': {'slot': '2', 'bus': '254', 'domain': '0', 'function': '2'}, 'iommu_group': '25', 'vendor_id': '0x8086'}}, 'pci_0000_fe_02_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Physical 0', 'vendor': 'Intel Corporation', 'product_id': '0x2d91', 'parent': 'computer', 'address': {'slot': '2', 'bus': '254', 'domain': '0', 'function': '1'}, 'iommu_group': '25', 'vendor_id': '0x8086'}}, 'pci_0000_fe_02_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Link 0', 'vendor': 'Intel Corporation', 'product_id': '0x2d90', 'parent': 'computer', 'address': {'slot': '2', 'bus': '254', 'domain': '0', 'function': '0'}, 'iommu_group': '25', 'vendor_id': '0x8086'}}, 'pci_0000_fe_02_5': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Physical 1', 'vendor': 'Intel Corporation', 'product_id': '0x2d95', 'parent': 'computer', 'address': {'slot': '2', 'bus': '254', 'domain': '0', 'function': '5'}, 'iommu_group': '25', 'vendor_id': '0x8086'}}, 'net_ens3f2_00_1b_21_8a_a6_ca': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'ens3f2', 'parent': 'pci_0000_24_00_2'}}, 'scsi_0_2_0_0': {'params': {'product': 'ServeRAID M5015', 'vendor': 'IBM', 'parent': 'scsi_target0_2_0', 'driver': 'sd', 'capability': 'scsi', 'is_assignable': 'true', 'udev_path': '/dev/sg0', 'address': {'bus': '2', 'host': '0', 'target': '0', 'lun': '0'}}}, 'usb_5_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb5'}}, 'pci_0000_fe_00_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QuickPath Architecture System Address Decoder', 'vendor': 'Intel Corporation', 'product_id': '0x2d81', 'parent': 'computer', 'address': {'slot': '0', 'bus': '254', 'domain': '0', 'function': '1'}, 'iommu_group': '24', 'vendor_id': '0x8086'}}, 'pci_0000_00_01_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '5520/5500/X58 I/O Hub PCI Express Root Port 1', 'vendor': 'Intel Corporation', 'product_id': '0x3408', 'parent': 'computer', 'address': {'slot': '1', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '1', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'pci_0000_fe_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QuickPath Architecture Generic Non-core Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2c70', 'parent': 'computer', 'address': {'slot': '0', 'bus': '254', 'domain': '0', 'function': '0'}, 'iommu_group': '24', 'vendor_id': '0x8086'}}, 'pci_0000_0b_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'NetXtreme II BCM5709 Gigabit Ethernet', 'vendor': 'Broadcom Inc. and subsidiaries', 'product_id': '0x1639', 'parent': 'pci_0000_00_01_0', 'address': {'slot': '0', 'bus': '11', 'domain': '0', 'function': '0'}, 'iommu_group': '17', 'vendor_id': '0x14e4', 'driver': 'bnx2'}}, 'usb_7_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb7'}}, 'usb_2_0_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_usb2'}}, 'pci_0000_ff_02_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Physical 0', 'vendor': 'Intel Corporation', 'product_id': '0x2d91', 'parent': 'computer', 'address': {'slot': '2', 'bus': '255', 'domain': '0', 'function': '1'}, 'iommu_group': '31', 'vendor_id': '0x8086'}}, 'scsi_target0_2_0': {'params': {'capability': 'scsi_target', 'is_assignable': 'true', 'parent': 'scsi_host0'}}, 'pci_0000_00_14_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 I/O Hub Throttle Registers', 'vendor': 'Intel Corporation', 'product_id': '0x3438', 'parent': 'computer', 'address': {'slot': '20', 'bus': '0', 'domain': '0', 'function': '3'}, 'iommu_group': '9', 'vendor_id': '0x8086'}}, 'pci_0000_00_14_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 I/O Hub Control Status and RAS Registers', 'vendor': 'Intel Corporation', 'product_id': '0x3423', 'parent': 'computer', 'address': {'slot': '20', 'bus': '0', 'domain': '0', 'function': '2'}, 'iommu_group': '9', 'vendor_id': '0x8086'}}, 'pci_0000_00_14_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers', 'vendor': 'Intel Corporation', 'product_id': '0x3422', 'parent': 'computer', 'address': {'slot': '20', 'bus': '0', 'domain': '0', 'function': '1'}, 'iommu_group': '9', 'vendor_id': '0x8086'}}, 'pci_0000_00_14_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 I/O Hub System Management Registers', 'vendor': 'Intel Corporation', 'product_id': '0x342e', 'parent': 'computer', 'address': {'slot': '20', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '9', 'vendor_id': '0x8086', 'driver': 'i7core_edac'}}, 'pci_0000_ff_04_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Control', 'vendor': 'Intel Corporation', 'product_id': '0x2da0', 'parent': 'computer', 'address': {'slot': '4', 'bus': '255', 'domain': '0', 'function': '0'}, 'iommu_group': '33', 'vendor_id': '0x8086'}}, 'pci_0000_ff_04_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Address', 'vendor': 'Intel Corporation', 'product_id': '0x2da1', 'parent': 'computer', 'address': {'slot': '4', 'bus': '255', 'domain': '0', 'function': '1'}, 'iommu_group': '33', 'vendor_id': '0x8086'}}, 'pci_0000_ff_04_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Rank', 'vendor': 'Intel Corporation', 'product_id': '0x2da2', 'parent': 'computer', 'address': {'slot': '4', 'bus': '255', 'domain': '0', 'function': '2'}, 'iommu_group': '33', 'vendor_id': '0x8086'}}, 'pci_0000_ff_04_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control', 'vendor': 'Intel Corporation', 'product_id': '0x2da3', 'parent': 'computer', 'address': {'slot': '4', 'bus': '255', 'domain': '0', 'function': '3'}, 'iommu_group': '33', 'vendor_id': '0x8086'}}, 'pci_0000_00_05_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '5520/X58 I/O Hub PCI Express Root Port 5', 'vendor': 'Intel Corporation', 'product_id': '0x340c', 'parent': 'computer', 'address': {'slot': '5', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '4', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'pci_0000_ff_02_4': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Link 1', 'vendor': 'Intel Corporation', 'product_id': '0x2d94', 'parent': 'computer', 'address': {'slot': '2', 'bus': '255', 'domain': '0', 'function': '4'}, 'iommu_group': '31', 'vendor_id': '0x8086'}}, 'pci_0000_ff_02_5': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Physical 1', 'vendor': 'Intel Corporation', 'product_id': '0x2d95', 'parent': 'computer', 'address': {'slot': '2', 'bus': '255', 'domain': '0', 'function': '5'}, 'iommu_group': '31', 'vendor_id': '0x8086'}}, 'pci_0000_ff_02_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Mirror Port Link 0', 'vendor': 'Intel Corporation', 'product_id': '0x2d92', 'parent': 'computer', 'address': {'slot': '2', 'bus': '255', 'domain': '0', 'function': '2'}, 'iommu_group': '31', 'vendor_id': '0x8086'}}, 'pci_0000_ff_02_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Mirror Port Link 1', 'vendor': 'Intel Corporation', 'product_id': '0x2d93', 'parent': 'computer', 'address': {'slot': '2', 'bus': '255', 'domain': '0', 'function': '3'}, 'iommu_group': '31', 'vendor_id': '0x8086'}}, 'pci_0000_ff_02_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Link 0', 'vendor': 'Intel Corporation', 'product_id': '0x2d90', 'parent': 'computer', 'address': {'slot': '2', 'bus': '255', 'domain': '0', 'function': '0'}, 'iommu_group': '31', 'vendor_id': '0x8086'}}, 'pci_0000_0b_00_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'NetXtreme II BCM5709 Gigabit Ethernet', 'vendor': 'Broadcom Inc. and subsidiaries', 'product_id': '0x1639', 'parent': 'pci_0000_00_01_0', 'address': {'slot': '0', 'bus': '11', 'domain': '0', 'function': '1'}, 'iommu_group': '17', 'vendor_id': '0x14e4', 'driver': 'bnx2'}}, 'pci_0000_00_16_5': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x342a', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '5'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_00_16_4': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x3429', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '4'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_00_16_7': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x342c', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '7'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_00_16_6': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x342b', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '6'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_00_16_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x3431', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '1'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_00_16_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x3430', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_00_16_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x3433', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '3'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'pci_0000_01_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'MegaRAID SAS 2108 [Liberator]', 'vendor': 'LSI Logic / Symbios Logic', 'product_id': '0x0079', 'parent': 'pci_0000_00_1c_0', 'address': {'slot': '0', 'bus': '1', 'domain': '0', 'function': '0'}, 'iommu_group': '13', 'vendor_id': '0x1000', 'driver': 'megaraid_sas'}}, 'scsi_target0_2_1': {'params': {'capability': 'scsi_target', 'is_assignable': 'true', 'parent': 'scsi_host0'}}, 'pci_0000_fe_02_4': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series QPI Link 1', 'vendor': 'Intel Corporation', 'product_id': '0x2d94', 'parent': 'computer', 'address': {'slot': '2', 'bus': '254', 'domain': '0', 'function': '4'}, 'iommu_group': '25', 'vendor_id': '0x8086'}}, 'pci_0000_fe_05_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Control', 'vendor': 'Intel Corporation', 'product_id': '0x2da8', 'parent': 'computer', 'address': {'slot': '5', 'bus': '254', 'domain': '0', 'function': '0'}, 'iommu_group': '28', 'vendor_id': '0x8086'}}, 'pci_0000_fe_05_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Address', 'vendor': 'Intel Corporation', 'product_id': '0x2da9', 'parent': 'computer', 'address': {'slot': '5', 'bus': '254', 'domain': '0', 'function': '1'}, 'iommu_group': '28', 'vendor_id': '0x8086'}}, 'pci_0000_fe_05_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Rank', 'vendor': 'Intel Corporation', 'product_id': '0x2daa', 'parent': 'computer', 'address': {'slot': '5', 'bus': '254', 'domain': '0', 'function': '2'}, 'iommu_group': '28', 'vendor_id': '0x8086'}}, 'pci_0000_fe_05_3': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control', 'vendor': 'Intel Corporation', 'product_id': '0x2dab', 'parent': 'computer', 'address': {'slot': '5', 'bus': '254', 'domain': '0', 'function': '3'}, 'iommu_group': '28', 'vendor_id': '0x8086'}}, 'pci_0000_fe_06_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Address', 'vendor': 'Intel Corporation', 'product_id': '0x2db1', 'parent': 'computer', 'address': {'slot': '6', 'bus': '254', 'domain': '0', 'function': '1'}, 'iommu_group': '29', 'vendor_id': '0x8086'}}, 'scsi_generic_sg1': {'params': {'capability': 'scsi_generic', 'is_assignable': 'true', 'udev_path': '/dev/sg1', 'parent': 'scsi_0_2_1_0'}}, 'scsi_generic_sg0': {'params': {'capability': 'scsi_generic', 'is_assignable': 'true', 'udev_path': '/dev/sg0', 'parent': 'scsi_0_2_0_0'}}, 'pci_0000_fe_06_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Channel 2 Control', 'vendor': 'Intel Corporation', 'product_id': '0x2db0', 'parent': 'computer', 'address': {'slot': '6', 'bus': '254', 'domain': '0', 'function': '0'}, 'iommu_group': '29', 'vendor_id': '0x8086'}}, 'pci_0000_fe_03_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller RAS Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2d9a', 'parent': 'computer', 'address': {'slot': '3', 'bus': '254', 'domain': '0', 'function': '2'}, 'iommu_group': '26', 'vendor_id': '0x8086'}}, 'pci_0000_fe_03_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2d98', 'parent': 'computer', 'address': {'slot': '3', 'bus': '254', 'domain': '0', 'function': '0'}, 'iommu_group': '26', 'vendor_id': '0x8086'}}, 'pci_0000_fe_03_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Target Address Decoder', 'vendor': 'Intel Corporation', 'product_id': '0x2d99', 'parent': 'computer', 'address': {'slot': '3', 'bus': '254', 'domain': '0', 'function': '1'}, 'iommu_group': '26', 'vendor_id': '0x8086'}}, 'pci_0000_fe_03_4': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Test Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2d9c', 'parent': 'computer', 'address': {'slot': '3', 'bus': '254', 'domain': '0', 'function': '4'}, 'iommu_group': '26', 'vendor_id': '0x8086'}}, 'usb_usb7': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '1.1 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0001', 'parent': 'pci_0000_00_1d_2', 'address': {'device': '1', 'bus': '7'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'scsi_0_2_1_0': {'params': {'product': 'ServeRAID M5015', 'vendor': 'IBM', 'parent': 'scsi_target0_2_1', 'driver': 'sd', 'capability': 'scsi', 'is_assignable': 'true', 'udev_path': '/dev/sg1', 'address': {'bus': '2', 'host': '0', 'target': '1', 'lun': '0'}}}, 'usb_usb5': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '1.1 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0001', 'parent': 'pci_0000_00_1d_0', 'address': {'device': '1', 'bus': '5'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'usb_usb4': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '1.1 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0001', 'parent': 'pci_0000_00_1a_1', 'address': {'device': '1', 'bus': '4'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'usb_usb3': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '1.1 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0001', 'parent': 'pci_0000_00_1a_0', 'address': {'device': '1', 'bus': '3'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'usb_usb2': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '2.0 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0002', 'parent': 'pci_0000_00_1d_7', 'address': {'device': '1', 'bus': '2'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'usb_usb1': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': '2.0 root hub', 'vendor': 'Linux Foundation', 'product_id': '0x0002', 'parent': 'pci_0000_00_1a_7', 'address': {'device': '1', 'bus': '1'}, 'vendor_id': '0x1d6b', 'driver': 'usb'}}, 'usb_2_5_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_2_5'}}, 'pci_0000_00_03_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '5520/5500/X58 I/O Hub PCI Express Root Port 3', 'vendor': 'Intel Corporation', 'product_id': '0x340a', 'parent': 'computer', 'address': {'slot': '3', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '3', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'usb_2_5_3_1': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'CS682 2-Port USB 2.0 DVI KVM Switch', 'vendor': 'ATEN International Co., Ltd', 'product_id': '0x2213', 'parent': 'usb_2_5_3', 'address': {'device': '9', 'bus': '2'}, 'vendor_id': '0x0557', 'driver': 'usb'}}, 'usb_2_6': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'Hub', 'vendor': 'Terminus Technology Inc.', 'product_id': '0x0101', 'parent': 'usb_usb2', 'address': {'device': '4', 'bus': '2'}, 'vendor_id': '0x1a40', 'driver': 'usb'}}, 'usb_2_5': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'Hub', 'vendor': 'Genesys Logic, Inc.', 'product_id': '0x0608', 'parent': 'usb_usb2', 'address': {'device': '3', 'bus': '2'}, 'vendor_id': '0x05e3', 'driver': 'usb'}}, 'usb_2_5_1': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'Uninterruptible Power Supply', 'vendor': 'American Power Conversion', 'product_id': '0x0002', 'parent': 'usb_2_5', 'address': {'device': '5', 'bus': '2'}, 'vendor_id': '0x051d', 'driver': 'usb'}}, 'usb_2_5_3': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'Hub', 'vendor': 'ATEN International Co., Ltd', 'product_id': '0x8021', 'parent': 'usb_2_5', 'address': {'device': '8', 'bus': '2'}, 'vendor_id': '0x0557', 'driver': 'usb'}}, 'usb_2_5_2': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'HASP copy protection dongle', 'vendor': 'Aladdin Knowledge Systems', 'product_id': '0x0001', 'parent': 'usb_2_5', 'address': {'device': '7', 'bus': '2'}, 'vendor_id': '0x0529', 'driver': 'usb'}}, 'pci_0000_00_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520 I/O Hub to ESI Port', 'vendor': 'Intel Corporation', 'product_id': '0x3406', 'parent': 'computer', 'address': {'slot': '0', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '0', 'vendor_id': '0x8086'}}, 'pci_0000_00_09_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '7500/5520/5500/X58 I/O Hub PCI Express Root Port 9', 'vendor': 'Intel Corporation', 'product_id': '0x3410', 'parent': 'computer', 'address': {'slot': '9', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '6', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'pci_0000_1f_00_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'ISP2532-based 8Gb Fibre Channel to PCI Express HBA', 'vendor': 'QLogic Corp.', 'product_id': '0x2532', 'parent': 'pci_0000_00_07_0', 'address': {'slot': '0', 'bus': '31', 'domain': '0', 'function': '1'}, 'iommu_group': '19', 'vendor_id': '0x1077', 'driver': 'qla2xxx'}}, 'pci_0000_1f_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'ISP2532-based 8Gb Fibre Channel to PCI Express HBA', 'vendor': 'QLogic Corp.', 'product_id': '0x2532', 'parent': 'pci_0000_00_07_0', 'address': {'slot': '0', 'bus': '31', 'domain': '0', 'function': '0'}, 'iommu_group': '19', 'vendor_id': '0x1077', 'driver': 'qla2xxx'}}, 'pci_0000_24_00_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '82580 Gigabit Network Connection', 'vendor': 'Intel Corporation', 'product_id': '0x150e', 'parent': 'pci_0000_00_09_0', 'address': {'slot': '0', 'bus': '36', 'domain': '0', 'function': '0'}, 'iommu_group': '20', 'vendor_id': '0x8086', 'driver': 'igb'}}, 'usb_2_5_2_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'parent': 'usb_2_5_2'}}, 'pci_0000_00_07_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '5520/5500/X58 I/O Hub PCI Express Root Port 7', 'vendor': 'Intel Corporation', 'product_id': '0x340e', 'parent': 'computer', 'address': {'slot': '7', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '5', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'net_eno2_e4_1f_13_b5_94_16': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'eno2', 'parent': 'pci_0000_0b_00_1'}}, 'usb_2_5_3_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_2_5_3'}}, 'usb_5_2_1_1': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'cdc_ether', 'parent': 'usb_5_2'}}, 'usb_5_2_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'cdc_ether', 'parent': 'usb_5_2'}}, 'usb_2_6_4_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_2_6_4'}}, 'pci_0000_00_02_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': '5520/5500/X58 I/O Hub PCI Express Root Port 2', 'vendor': 'Intel Corporation', 'product_id': '0x3409', 'parent': 'computer', 'address': {'slot': '2', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '2', 'vendor_id': '0x8086', 'driver': 'pcieport'}}, 'pci_0000_ff_03_4': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Test Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2d9c', 'parent': 'computer', 'address': {'slot': '3', 'bus': '255', 'domain': '0', 'function': '4'}, 'iommu_group': '32', 'vendor_id': '0x8086'}}, 'pci_0000_ff_03_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller RAS Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2d9a', 'parent': 'computer', 'address': {'slot': '3', 'bus': '255', 'domain': '0', 'function': '2'}, 'iommu_group': '32', 'vendor_id': '0x8086'}}, 'pci_0000_ff_03_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Target Address Decoder', 'vendor': 'Intel Corporation', 'product_id': '0x2d99', 'parent': 'computer', 'address': {'slot': '3', 'bus': '255', 'domain': '0', 'function': '1'}, 'iommu_group': '32', 'vendor_id': '0x8086'}}, 'pci_0000_ff_03_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': 'Xeon 5600 Series Integrated Memory Controller Registers', 'vendor': 'Intel Corporation', 'product_id': '0x2d98', 'parent': 'computer', 'address': {'slot': '3', 'bus': '255', 'domain': '0', 'function': '0'}, 'iommu_group': '32', 'vendor_id': '0x8086'}}, 'pci_0000_00_10_1': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 Routing and Protocol Layer Registers Port 0', 'vendor': 'Intel Corporation', 'product_id': '0x3426', 'parent': 'computer', 'address': {'slot': '16', 'bus': '0', 'domain': '0', 'function': '1'}, 'iommu_group': '7', 'vendor_id': '0x8086'}}, 'pci_0000_00_10_0': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '7500/5520/5500/X58 Physical and Link Layer Registers Port 0', 'vendor': 'Intel Corporation', 'product_id': '0x3425', 'parent': 'computer', 'address': {'slot': '16', 'bus': '0', 'domain': '0', 'function': '0'}, 'iommu_group': '7', 'vendor_id': '0x8086'}}, 'pci_0000_06_00_0': {'params': {'capability': 'pci', 'is_assignable': 'false', 'product': 'VSC452 [SuperBMC]', 'vendor': 'Vitesse Semiconductor', 'product_id': '0x0452', 'parent': 'pci_0000_00_1c_4', 'address': {'slot': '0', 'bus': '6', 'domain': '0', 'function': '0'}, 'iommu_group': '13', 'vendor_id': '0x101b'}}, 'scsi_host3': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_00_1f_5'}}, 'scsi_host2': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_00_1f_2'}}, 'scsi_host1': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_00_1f_2'}}, 'scsi_host0': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_01_00_0'}}, 'scsi_host6': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_1f_00_1'}}, 'scsi_host5': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_1f_00_0'}}, 'scsi_host4': {'params': {'capability': 'scsi_host', 'is_assignable': 'true', 'parent': 'pci_0000_00_1f_5'}}, 'usb_2_6_1_0': {'params': {'capability': 'usb', 'is_assignable': 'true', 'driver': 'hub', 'parent': 'usb_2_6'}}, 'pci_0000_00_16_2': {'params': {'capability': 'pci', 'is_assignable': 'true', 'product': '5520/5500/X58 Chipset QuickData Technology Device', 'vendor': 'Intel Corporation', 'product_id': '0x3432', 'parent': 'computer', 'address': {'slot': '22', 'bus': '0', 'domain': '0', 'function': '2'}, 'iommu_group': '11', 'vendor_id': '0x8086', 'driver': 'ioatdma'}}, 'net_lo_00_00_00_00_00_00': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'lo', 'parent': 'computer'}}, 'net_ens3f0_00_1b_21_8a_a6_c8': {'params': {'capability': 'net', 'is_assignable': 'true', 'interface': 'ens3f0', 'parent': 'pci_0000_24_00_0'}}, 'usb_5_2': {'params': {'capability': 'usb_device', 'is_assignable': 'true', 'product': 'RNDIS/CDC ETHER', 'vendor': 'IBM Corp.', 'product_id': '0x4010', 'parent': 'usb_usb5', 'address': {'device': '2', 'bus': '5'}, 'vendor_id': '0x04b3', 'driver': 'usb'}}, 'block_sdb_3600605b002f6927021946993801eb9a6': {'params': {'capability': 'storage', 'is_assignable': 'true', 'product': 'ServeRAID M5015', 'vendor': 'IBM', 'parent': 'scsi_0_2_1_0'}}}} from=::ffff:192.168.0.19,42536, flow_id=4a97a86d (api:54) 2019-04-13 22:07:55,868+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.hostdevListByCaps succeeded in 0.32 seconds (__init__:312) 2019-04-13 22:07:56,444+0200 INFO (jsonrpc/0) [vdsm.api] START connectStoragePool(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', hostID=1, msdUUID=u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', masterVersion=47, domainsMap={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'active'}, options=None) from=::ffff:192.168.0.19,42536, task_id=08382c9c-3ef9-492a-964d-f76d94e17b6d (api:48) 2019-04-13 22:07:56,444+0200 INFO (jsonrpc/0) [storage.StoragePoolMemoryBackend] new storage pool master version 47 and domains map {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': u'Active', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': u'Active', u'93f02ede-ec5c-4641-8e97-2949f98e856d': u'Active', u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': u'Active'} (spbackends:455) 2019-04-13 22:07:56,717+0200 INFO (jsonrpc/0) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 22:07:56,718+0200 INFO (jsonrpc/0) [storage.StoragePool] Creating pool directory '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124' (sp:1275) 2019-04-13 22:07:56,718+0200 INFO (jsonrpc/0) [storage.fileUtils] Creating directory: /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124 mode: None (fileUtils:199) 2019-04-13 22:07:56,872+0200 WARN (jsonrpc/0) [storage.LVM] Reloading VGs failed (vgs=[u'9c3f06cf-7475-448e-819b-f4f52fa7d782', u'42bb6772-12f9-4bf2-9245-43caa41c5a2a', u'93f02ede-ec5c-4641-8e97-2949f98e856d'] rc=5 out=[] err=[' Volume group "9c3f06cf-7475-448e-819b-f4f52fa7d782" not found', ' Cannot process volume group 9c3f06cf-7475-448e-819b-f4f52fa7d782', ' Volume group "42bb6772-12f9-4bf2-9245-43caa41c5a2a" not found', ' Cannot process volume group 42bb6772-12f9-4bf2-9245-43caa41c5a2a', ' Volume group "93f02ede-ec5c-4641-8e97-2949f98e856d" not found', ' Cannot process volume group 93f02ede-ec5c-4641-8e97-2949f98e856d']) (lvm:442) 2019-04-13 22:07:56,877+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782 to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782 (sp:1230) 2019-04-13 22:07:56,877+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/42bb6772-12f9-4bf2-9245-43caa41c5a2a (sp:1230) 2019-04-13 22:07:56,877+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/93f02ede-ec5c-4641-8e97-2949f98e856d (sp:1230) 2019-04-13 22:07:56,877+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/6a4cde72-6a6a-499e-9b6e-d490d91ca19e (sp:1230) 2019-04-13 22:07:56,877+0200 INFO (jsonrpc/0) [storage.StoragePool] Linking /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e to /rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/mastersd (sp:1230) 2019-04-13 22:07:56,878+0200 INFO (jsonrpc/0) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.0.19,42536, task_id=08382c9c-3ef9-492a-964d-f76d94e17b6d (api:54) 2019-04-13 22:07:56,878+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect succeeded in 0.44 seconds (__init__:312) 2019-04-13 22:07:56,888+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,42536, task_id=56048a58-1fcc-46c6-af75-68c66345cf14 (api:48) 2019-04-13 22:07:56,941+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} from=::ffff:192.168.0.19,42536, task_id=56048a58-1fcc-46c6-af75-68c66345cf14 (api:54) 2019-04-13 22:07:56,941+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.05 seconds (__init__:312) 2019-04-13 22:07:56,966+0200 INFO (jsonrpc/7) [vdsm.api] START spmStart(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', prevID=-1, prevLVER=u'-1', maxHostID=250, domVersion=u'4', options=None) from=::ffff:192.168.0.19,42536, task_id=25b68fdb-2dd8-4aed-89a5-0eda39cd92c9 (api:48) 2019-04-13 22:07:56,967+0200 INFO (jsonrpc/7) [vdsm.api] FINISH spmStart return=None from=::ffff:192.168.0.19,42536, task_id=25b68fdb-2dd8-4aed-89a5-0eda39cd92c9 (api:54) 2019-04-13 22:07:56,967+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.spmStart succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:07:56,967+0200 INFO (tasks/0) [storage.ThreadPool.WorkerThread] START task 25b68fdb-2dd8-4aed-89a5-0eda39cd92c9 (cmd=>, args=None) (threadPool:208) 2019-04-13 22:07:56,972+0200 INFO (tasks/0) [storage.SANLock] Acquiring host id for domain 6a4cde72-6a6a-499e-9b6e-d490d91ca19e (id=1, async=False) (clusterlock:294) 2019-04-13 22:07:57,986+0200 INFO (jsonrpc/3) [vdsm.api] START getTaskStatus(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=91292722-d3fe-40fd-92cf-1ce4b0308c76 (api:48) 2019-04-13 22:07:57,986+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '25b68fdb-2dd8-4aed-89a5-0eda39cd92c9'}} from=::ffff:192.168.0.19,42536, task_id=91292722-d3fe-40fd-92cf-1ce4b0308c76 (api:54) 2019-04-13 22:07:57,986+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:07:58,946+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:07:58,946+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:07:58,946+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:07:58,995+0200 INFO (jsonrpc/4) [vdsm.api] START getTaskStatus(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=774f4e3a-0201-4c4d-b358-087060e3c43b (api:48) 2019-04-13 22:07:58,995+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '25b68fdb-2dd8-4aed-89a5-0eda39cd92c9'}} from=::ffff:192.168.0.19,42536, task_id=774f4e3a-0201-4c4d-b358-087060e3c43b (api:54) 2019-04-13 22:07:58,995+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:07:59,367+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=95480f32-4689-4510-8a0d-3561035ff44b (api:48) 2019-04-13 22:07:59,367+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000804233', 'lastCheck': '4.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000554619', 'lastCheck': '4.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000676994', 'lastCheck': '4.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000347955', 'lastCheck': '4.5', 'valid': True}} from=internal, task_id=95480f32-4689-4510-8a0d-3561035ff44b (api:54) 2019-04-13 22:07:59,754+0200 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.0.19,42536 (api:48) 2019-04-13 22:07:59,775+0200 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,42536, task_id=d30d12c5-6ce8-4e73-81b4-14b5cccaf8d5 (api:48) 2019-04-13 22:07:59,776+0200 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000804233', 'lastCheck': '4.7', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000554619', 'lastCheck': '4.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000676994', 'lastCheck': '4.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000347955', 'lastCheck': '4.9', 'valid': True}} from=::ffff:192.168.0.19,42536, task_id=d30d12c5-6ce8-4e73-81b4-14b5cccaf8d5 (api:54) 2019-04-13 22:07:59,776+0200 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,42536, task_id=31b4a812-f400-45a9-b36c-7599bfba8021 (api:48) 2019-04-13 22:07:59,777+0200 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,42536, task_id=31b4a812-f400-45a9-b36c-7599bfba8021 (api:54) 2019-04-13 22:07:59,781+0200 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.19', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.68'}, '21': {'cpuUser': '0.32', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.55'}, '22': {'cpuUser': '0.06', 'nodeIndex': 1, 'cpuSys': '0.06', 'cpuIdle': '99.88'}, '23': {'cpuUser': '0.06', 'nodeIndex': 1, 'cpuSys': '0.06', 'cpuIdle': '99.88'}, '1': {'cpuUser': '6.80', 'nodeIndex': 0, 'cpuSys': '3.04', 'cpuIdle': '90.16'}, '0': {'cpuUser': '0.39', 'nodeIndex': 0, 'cpuSys': '0.91', 'cpuIdle': '98.70'}, '3': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.26', 'cpuIdle': '99.61'}, '2': {'cpuUser': '0.65', 'nodeIndex': 0, 'cpuSys': '0.58', 'cpuIdle': '98.77'}, '5': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.19', 'cpuIdle': '99.81'}, '4': {'cpuUser': '0.06', 'nodeIndex': 0, 'cpuSys': '0.19', 'cpuIdle': '99.75'}, '7': {'cpuUser': '4.28', 'nodeIndex': 1, 'cpuSys': '0.84', 'cpuIdle': '94.88'}, '6': {'cpuUser': '0.32', 'nodeIndex': 1, 'cpuSys': '0.26', 'cpuIdle': '99.42'}, '9': {'cpuUser': '0.97', 'nodeIndex': 1, 'cpuSys': '0.45', 'cpuIdle': '98.58'}, '8': {'cpuUser': '0.84', 'nodeIndex': 1, 'cpuSys': '0.39', 'cpuIdle': '98.77'}, '11': {'cpuUser': '0.32', 'nodeIndex': 1, 'cpuSys': '0.19', 'cpuIdle': '99.49'}, '10': {'cpuUser': '0.32', 'nodeIndex': 1, 'cpuSys': '0.19', 'cpuIdle': '99.49'}, '13': {'cpuUser': '3.37', 'nodeIndex': 0, 'cpuSys': '0.52', 'cpuIdle': '96.11'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.45', 'nodeIndex': 0, 'cpuSys': '0.06', 'cpuIdle': '99.49'}, '14': {'cpuUser': '0.19', 'nodeIndex': 0, 'cpuSys': '0.32', 'cpuIdle': '99.49'}, '17': {'cpuUser': '0.06', 'nodeIndex': 0, 'cpuSys': '0.06', 'cpuIdle': '99.88'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.06', 'cpuIdle': '99.94'}, '19': {'cpuUser': '8.23', 'nodeIndex': 1, 'cpuSys': '2.01', 'cpuIdle': '89.76'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.06', 'cpuIdle': '99.94'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95862'}, '0': {'memPercent': 4, 'memFree': '95337'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000804233', 'lastCheck': '4.7', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000676994', 'lastCheck': '4.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000347955', 'lastCheck': '4.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000554619', 'lastCheck': '4.8', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186079.775111, 'rx': '13176', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '173207', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186079.775111, 'rx': '156011', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186079.775111, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '163394', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186079.775111, 'rx': '130142', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '2209407', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186079.775111, 'rx': '2209407', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186079.775111, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '7761', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186079.775111, 'rx': '138459', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '165446', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186079.775111, 'rx': '17552', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1494', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186079.775111, 'rx': '19602', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186079.775111, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186079.775111, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186079.775111, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186079.775111, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186079.775111, 'rx': '11504', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '42', 'ksmPages': 100, 'elapsedTime': '471.24', 'cpuLoad': '0.17', 'cpuSys': '0.45', 'diskStats': {'/var/log': {'free': '47799'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47799'}}, 'cpuUserVdsmd': '6.80', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191840, 'bootTime': '1555185555', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T20:07:59 GMT', 'cpuUser': '1.16', 'memFree': 191584, 'cpuIdle': '98.39', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '2.01'}} from=::ffff:192.168.0.19,42536 (api:54) 2019-04-13 22:07:59,784+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 22:08:00,003+0200 INFO (jsonrpc/6) [vdsm.api] START getTaskStatus(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=168c535a-00d4-4967-9418-ff0f63a0acee (api:48) 2019-04-13 22:08:00,003+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '25b68fdb-2dd8-4aed-89a5-0eda39cd92c9'}} from=::ffff:192.168.0.19,42536, task_id=168c535a-00d4-4967-9418-ff0f63a0acee (api:54) 2019-04-13 22:08:00,003+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:01,011+0200 INFO (jsonrpc/0) [vdsm.api] START getTaskStatus(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=96f76d1e-aea1-40b6-a3dd-d17cbeb6749c (api:48) 2019-04-13 22:08:01,011+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '25b68fdb-2dd8-4aed-89a5-0eda39cd92c9'}} from=::ffff:192.168.0.19,42536, task_id=96f76d1e-aea1-40b6-a3dd-d17cbeb6749c (api:54) 2019-04-13 22:08:01,011+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:02,023+0200 INFO (jsonrpc/1) [vdsm.api] START getTaskStatus(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=685a76c7-6709-4fcb-b499-eb0309c31e16 (api:48) 2019-04-13 22:08:02,023+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '25b68fdb-2dd8-4aed-89a5-0eda39cd92c9'}} from=::ffff:192.168.0.19,42536, task_id=685a76c7-6709-4fcb-b499-eb0309c31e16 (api:54) 2019-04-13 22:08:02,023+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:03,030+0200 INFO (jsonrpc/7) [vdsm.api] START getTaskStatus(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=4fbc9cd8-f921-41f4-bb44-cfae4e6036e3 (api:48) 2019-04-13 22:08:03,030+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '25b68fdb-2dd8-4aed-89a5-0eda39cd92c9'}} from=::ffff:192.168.0.19,42536, task_id=4fbc9cd8-f921-41f4-bb44-cfae4e6036e3 (api:54) 2019-04-13 22:08:03,030+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:04,037+0200 INFO (jsonrpc/3) [vdsm.api] START getTaskStatus(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=cd9f6f18-3bc4-49f8-8e14-6ee68e4dbc48 (api:48) 2019-04-13 22:08:04,037+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '25b68fdb-2dd8-4aed-89a5-0eda39cd92c9'}} from=::ffff:192.168.0.19,42536, task_id=cd9f6f18-3bc4-49f8-8e14-6ee68e4dbc48 (api:54) 2019-04-13 22:08:04,038+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:04,989+0200 INFO (monitor/42bb677) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 22:08:05,046+0200 INFO (jsonrpc/2) [vdsm.api] START getTaskStatus(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=0d1e32da-cbe1-40d5-8bfe-f236284f29d6 (api:48) 2019-04-13 22:08:05,046+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '25b68fdb-2dd8-4aed-89a5-0eda39cd92c9'}} from=::ffff:192.168.0.19,42536, task_id=0d1e32da-cbe1-40d5-8bfe-f236284f29d6 (api:54) 2019-04-13 22:08:05,046+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:05,076+0200 INFO (monitor/9c3f06c) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 22:08:05,080+0200 INFO (monitor/9c3f06c) [storage.SANLock] Acquiring host id for domain 9c3f06cf-7475-448e-819b-f4f52fa7d782 (id=1, async=True) (clusterlock:294) 2019-04-13 22:08:05,094+0200 INFO (monitor/93f02ed) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-13 22:08:06,054+0200 INFO (jsonrpc/4) [vdsm.api] START getTaskStatus(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=0193791b-8800-4d15-bcf1-45f7ee8271a9 (api:48) 2019-04-13 22:08:06,054+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '25b68fdb-2dd8-4aed-89a5-0eda39cd92c9'}} from=::ffff:192.168.0.19,42536, task_id=0193791b-8800-4d15-bcf1-45f7ee8271a9 (api:54) 2019-04-13 22:08:06,054+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 22:08:07,061+0200 INFO (jsonrpc/5) [vdsm.api] START getTaskStatus(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=33dffb1b-47b8-4f3a-8cb5-e07d5590d0c8 (api:48) 2019-04-13 22:08:07,062+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '25b68fdb-2dd8-4aed-89a5-0eda39cd92c9'}} from=::ffff:192.168.0.19,42536, task_id=33dffb1b-47b8-4f3a-8cb5-e07d5590d0c8 (api:54) 2019-04-13 22:08:07,062+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:08,069+0200 INFO (jsonrpc/6) [vdsm.api] START getTaskStatus(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=611c2d67-a521-477a-8518-84904696029b (api:48) 2019-04-13 22:08:08,069+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '25b68fdb-2dd8-4aed-89a5-0eda39cd92c9'}} from=::ffff:192.168.0.19,42536, task_id=611c2d67-a521-477a-8518-84904696029b (api:54) 2019-04-13 22:08:08,069+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:09,076+0200 INFO (jsonrpc/0) [vdsm.api] START getTaskStatus(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=e967dc65-07c4-4320-8b70-288af6710b5b (api:48) 2019-04-13 22:08:09,076+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '25b68fdb-2dd8-4aed-89a5-0eda39cd92c9'}} from=::ffff:192.168.0.19,42536, task_id=e967dc65-07c4-4320-8b70-288af6710b5b (api:54) 2019-04-13 22:08:09,077+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:10,069+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,42536 (api:48) 2019-04-13 22:08:10,069+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,42536 (api:54) 2019-04-13 22:08:10,070+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:10,084+0200 INFO (jsonrpc/7) [vdsm.api] START getTaskStatus(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=c1df2805-950a-4198-b49e-38041e00f9a3 (api:48) 2019-04-13 22:08:10,084+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '25b68fdb-2dd8-4aed-89a5-0eda39cd92c9'}} from=::ffff:192.168.0.19,42536, task_id=c1df2805-950a-4198-b49e-38041e00f9a3 (api:54) 2019-04-13 22:08:10,084+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 22:08:11,092+0200 INFO (jsonrpc/3) [vdsm.api] START getTaskStatus(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=30ce7482-0310-4918-8b83-ddfd707c1f78 (api:48) 2019-04-13 22:08:11,092+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '25b68fdb-2dd8-4aed-89a5-0eda39cd92c9'}} from=::ffff:192.168.0.19,42536, task_id=30ce7482-0310-4918-8b83-ddfd707c1f78 (api:54) 2019-04-13 22:08:11,093+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:12,100+0200 INFO (jsonrpc/2) [vdsm.api] START getTaskStatus(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=2bf77e93-1860-4089-aa90-a5dea03ff691 (api:48) 2019-04-13 22:08:12,100+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '25b68fdb-2dd8-4aed-89a5-0eda39cd92c9'}} from=::ffff:192.168.0.19,42536, task_id=2bf77e93-1860-4089-aa90-a5dea03ff691 (api:54) 2019-04-13 22:08:12,100+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:13,109+0200 INFO (jsonrpc/4) [vdsm.api] START getTaskStatus(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=9854ff53-5d57-4f23-b363-9f3f12c89d96 (api:48) 2019-04-13 22:08:13,109+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '25b68fdb-2dd8-4aed-89a5-0eda39cd92c9'}} from=::ffff:192.168.0.19,42536, task_id=9854ff53-5d57-4f23-b363-9f3f12c89d96 (api:54) 2019-04-13 22:08:13,109+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:13,964+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:08:13,965+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:08:13,965+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:14,116+0200 INFO (jsonrpc/6) [vdsm.api] START getTaskStatus(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=48b119a3-4dd4-486a-a244-7c655a67b7c2 (api:48) 2019-04-13 22:08:14,116+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '25b68fdb-2dd8-4aed-89a5-0eda39cd92c9'}} from=::ffff:192.168.0.19,42536, task_id=48b119a3-4dd4-486a-a244-7c655a67b7c2 (api:54) 2019-04-13 22:08:14,117+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:14,392+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=5ddb2a3e-6dbb-4e77-bc39-70a41adcd475 (api:48) 2019-04-13 22:08:14,393+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000775833', 'lastCheck': '9.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00057121', 'lastCheck': '9.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000430439', 'lastCheck': '9.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000480413', 'lastCheck': '9.5', 'valid': True}} from=internal, task_id=5ddb2a3e-6dbb-4e77-bc39-70a41adcd475 (api:54) 2019-04-13 22:08:14,815+0200 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.0.19,42536 (api:48) 2019-04-13 22:08:14,835+0200 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,42536, task_id=fbe109c5-e1e7-43e9-8051-b09f148026b9 (api:48) 2019-04-13 22:08:14,835+0200 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000775833', 'lastCheck': '9.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00057121', 'lastCheck': '9.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000430439', 'lastCheck': '9.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000480413', 'lastCheck': '10.0', 'valid': True}} from=::ffff:192.168.0.19,42536, task_id=fbe109c5-e1e7-43e9-8051-b09f148026b9 (api:54) 2019-04-13 22:08:14,835+0200 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,42536, task_id=776a5565-3da6-4cae-a1a1-bb162db28b02 (api:48) 2019-04-13 22:08:14,836+0200 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,42536, task_id=776a5565-3da6-4cae-a1a1-bb162db28b02 (api:54) 2019-04-13 22:08:14,840+0200 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '0.80', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '98.73'}, '0': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '3': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '5': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '4': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '9': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '8': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.80'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '13': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '17': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '0.27', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.53'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95863'}, '0': {'memPercent': 4, 'memFree': '95338'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000775833', 'lastCheck': '9.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000430439', 'lastCheck': '9.7', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000480413', 'lastCheck': '10.0', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00057121', 'lastCheck': '9.8', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186094.834562, 'rx': '13566', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '205300', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186094.834562, 'rx': '182001', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186094.834562, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '194953', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186094.834562, 'rx': '154040', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '2217468', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186094.834562, 'rx': '2217468', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186094.834562, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '7761', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186094.834562, 'rx': '164231', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '197539', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186094.834562, 'rx': '17770', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1494', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186094.834562, 'rx': '20268', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186094.834562, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186094.834562, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186094.834562, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186094.834562, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186094.834562, 'rx': '11826', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '44', 'ksmPages': 100, 'elapsedTime': '486.30', 'cpuLoad': '0.16', 'cpuSys': '0.04', 'diskStats': {'/var/log': {'free': '47799'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47799'}}, 'cpuUserVdsmd': '0.80', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191841, 'bootTime': '1555185555', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T20:08:14 GMT', 'cpuUser': '0.06', 'memFree': 191585, 'cpuIdle': '99.90', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.27'}} from=::ffff:192.168.0.19,42536 (api:54) 2019-04-13 22:08:14,843+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.02 seconds (__init__:312) 2019-04-13 22:08:15,123+0200 INFO (jsonrpc/1) [vdsm.api] START getTaskStatus(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=3c91408d-91c2-4483-8866-2809dcfb0193 (api:48) 2019-04-13 22:08:15,124+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '25b68fdb-2dd8-4aed-89a5-0eda39cd92c9'}} from=::ffff:192.168.0.19,42536, task_id=3c91408d-91c2-4483-8866-2809dcfb0193 (api:54) 2019-04-13 22:08:15,124+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 22:08:15,975+0200 INFO (tasks/0) [storage.SANLock] Host id for domain 6a4cde72-6a6a-499e-9b6e-d490d91ca19e successfully acquired (id=1, async=False) (clusterlock:317) 2019-04-13 22:08:15,976+0200 INFO (tasks/0) [storage.SANLock] Acquiring Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576) for host id 1 (clusterlock:387) 2019-04-13 22:08:16,029+0200 INFO (tasks/0) [storage.SANLock] Successfully acquired Lease(name='SDM', path=u'/rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/dom_md/leases', offset=1048576) for host id 1 (clusterlock:425) 2019-04-13 22:08:16,031+0200 INFO (monitor/6a4cde7) [storage.Monitor] Host id for domain 6a4cde72-6a6a-499e-9b6e-d490d91ca19e successfully acquired (id: 1) (monitor:448) 2019-04-13 22:08:16,033+0200 INFO (tasks/0) [storage.StorageDomain] Creating vms dir: /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/master/vms (sd:1048) 2019-04-13 22:08:16,038+0200 INFO (tasks/0) [storage.StorageDomain] Creating task dir: /rhev/data-center/mnt/onode-1.discovery.intern:_mnt_store/6a4cde72-6a6a-499e-9b6e-d490d91ca19e/master/tasks (sd:1051) 2019-04-13 22:08:16,045+0200 INFO (tasks/0) [storage.StorageDomain] Creating vms dir: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d/master/vms (sd:1048) 2019-04-13 22:08:16,051+0200 INFO (tasks/0) [storage.StorageDomain] Creating task dir: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_sicherung_vmBackups/93f02ede-ec5c-4641-8e97-2949f98e856d/master/tasks (sd:1051) 2019-04-13 22:08:16,112+0200 INFO (tasks/0) [storage.ThreadPool.WorkerThread] FINISH task 25b68fdb-2dd8-4aed-89a5-0eda39cd92c9 (threadPool:210) 2019-04-13 22:08:16,132+0200 INFO (jsonrpc/7) [vdsm.api] START getTaskStatus(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=d92e8812-ce42-47d9-8161-17acc3fa7982 (api:48) 2019-04-13 22:08:16,132+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getTaskStatus return={'taskStatus': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': '25b68fdb-2dd8-4aed-89a5-0eda39cd92c9'}} from=::ffff:192.168.0.19,42536, task_id=d92e8812-ce42-47d9-8161-17acc3fa7982 (api:54) 2019-04-13 22:08:16,133+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Task.getStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:16,140+0200 INFO (jsonrpc/3) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,42536, task_id=8ce97bb3-9ac8-463b-b1bf-0f598c457982 (api:48) 2019-04-13 22:08:16,145+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 29L}} from=::ffff:192.168.0.19,42536, task_id=8ce97bb3-9ac8-463b-b1bf-0f598c457982 (api:54) 2019-04-13 22:08:16,146+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 22:08:16,156+0200 INFO (jsonrpc/2) [vdsm.api] START clearTask(taskID=u'25b68fdb-2dd8-4aed-89a5-0eda39cd92c9', spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=d87a824f-be1c-4370-a284-f276e1d199c3 (api:48) 2019-04-13 22:08:16,156+0200 INFO (jsonrpc/2) [vdsm.api] FINISH clearTask return=None from=::ffff:192.168.0.19,42536, task_id=d87a824f-be1c-4370-a284-f276e1d199c3 (api:54) 2019-04-13 22:08:16,156+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Task.clear succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:16,184+0200 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::ffff:192.168.0.19:42542 (protocoldetector:61) 2019-04-13 22:08:16,286+0200 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::ffff:192.168.0.19:42542 (protocoldetector:125) 2019-04-13 22:08:16,286+0200 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompserver:95) 2019-04-13 22:08:16,287+0200 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompserver:124) 2019-04-13 22:08:16,328+0200 INFO (jsonrpc/4) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,42542, task_id=20730cd3-3cf9-46a4-b58d-18aa43c2adb1 (api:48) 2019-04-13 22:08:16,336+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 29L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,42542, task_id=20730cd3-3cf9-46a4-b58d-18aa43c2adb1 (api:54) 2019-04-13 22:08:16,337+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 22:08:16,412+0200 INFO (jsonrpc/5) [vdsm.api] START getAllTasksInfo(spUUID=None, options=None) from=::ffff:192.168.0.19,42536, task_id=f966f5de-cb2c-4180-99fa-5c9b49a5c30e (api:48) 2019-04-13 22:08:16,412+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getAllTasksInfo return={'allTasksInfo': {}} from=::ffff:192.168.0.19,42536, task_id=f966f5de-cb2c-4180-99fa-5c9b49a5c30e (api:54) 2019-04-13 22:08:16,412+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllTasksInfo succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:25,082+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,42536 (api:48) 2019-04-13 22:08:25,082+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,42536 (api:54) 2019-04-13 22:08:25,082+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:26,421+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,42536, task_id=8fdd70b3-1b35-4629-93d1-5c4a5458b140 (api:48) 2019-04-13 22:08:26,426+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 29L}} from=::ffff:192.168.0.19,42536, task_id=8fdd70b3-1b35-4629-93d1-5c4a5458b140 (api:54) 2019-04-13 22:08:26,427+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 22:08:26,443+0200 INFO (jsonrpc/1) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,42542, task_id=68b1b48d-9123-4e9a-9ec2-3e8107109c14 (api:48) 2019-04-13 22:08:26,451+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 29L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,42542, task_id=68b1b48d-9123-4e9a-9ec2-3e8107109c14 (api:54) 2019-04-13 22:08:26,451+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 2019-04-13 22:08:28,984+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:08:28,984+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:08:28,985+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:29,421+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=9be80fdf-cde6-4c2a-bbf6-adf5610be88c (api:48) 2019-04-13 22:08:29,421+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000646564', 'lastCheck': '4.3', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00060189', 'lastCheck': '4.4', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00055297', 'lastCheck': '4.3', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000181452', 'lastCheck': '3.4', 'valid': True}} from=internal, task_id=9be80fdf-cde6-4c2a-bbf6-adf5610be88c (api:54) 2019-04-13 22:08:29,899+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.19,42536 (api:48) 2019-04-13 22:08:29,919+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,42536, task_id=1e18f7ac-0b1a-42f0-a114-76bd2c61f136 (api:48) 2019-04-13 22:08:29,919+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000646564', 'lastCheck': '4.8', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00060189', 'lastCheck': '4.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00055297', 'lastCheck': '4.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000181452', 'lastCheck': '3.9', 'valid': True}} from=::ffff:192.168.0.19,42536, task_id=1e18f7ac-0b1a-42f0-a114-76bd2c61f136 (api:54) 2019-04-13 22:08:29,920+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,42536, task_id=af80e539-5908-401e-870a-456dc1f137ac (api:48) 2019-04-13 22:08:29,920+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,42536, task_id=af80e539-5908-401e-870a-456dc1f137ac (api:54) 2019-04-13 22:08:29,925+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '1': {'cpuUser': '1.20', 'nodeIndex': 0, 'cpuSys': '1.20', 'cpuIdle': '97.60'}, '0': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '3': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '5': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '4': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '6': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '9': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '8': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.73'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '13': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '17': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '0.27', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.46'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95855'}, '0': {'memPercent': 4, 'memFree': '95335'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000646564', 'lastCheck': '4.8', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00055297', 'lastCheck': '4.8', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000181452', 'lastCheck': '3.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00060189', 'lastCheck': '4.9', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186109.918839, 'rx': '13956', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '241718', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186109.918839, 'rx': '1301079', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186109.918839, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '229526', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186109.918839, 'rx': '1225118', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '18897003', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186109.918839, 'rx': '18897003', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186109.918839, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '7889', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186109.918839, 'rx': '1283053', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '233829', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186109.918839, 'rx': '18026', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1494', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186109.918839, 'rx': '20780', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186109.918839, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186109.918839, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186109.918839, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186109.918839, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186109.918839, 'rx': '12194', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '44', 'ksmPages': 100, 'elapsedTime': '501.38', 'cpuLoad': '0.17', 'cpuSys': '0.08', 'diskStats': {'/var/log': {'free': '47799'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47799'}}, 'cpuUserVdsmd': '1.06', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191834, 'bootTime': '1555185555', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T20:08:29 GMT', 'cpuUser': '0.08', 'memFree': 191578, 'cpuIdle': '99.84', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.60'}} from=::ffff:192.168.0.19,42536 (api:54) 2019-04-13 22:08:29,927+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 22:08:35,091+0200 INFO (monitor/9c3f06c) [storage.Monitor] Host id for domain 9c3f06cf-7475-448e-819b-f4f52fa7d782 successfully acquired (id: 1) (monitor:448) 2019-04-13 22:08:36,535+0200 INFO (jsonrpc/2) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,42536, task_id=2d7ca9e5-508f-4730-a728-6d530bc3241c (api:48) 2019-04-13 22:08:36,540+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 29L}} from=::ffff:192.168.0.19,42536, task_id=2d7ca9e5-508f-4730-a728-6d530bc3241c (api:54) 2019-04-13 22:08:36,540+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:36,554+0200 INFO (jsonrpc/4) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,42542, task_id=a208f729-962a-4338-90d0-23293cc34546 (api:48) 2019-04-13 22:08:36,562+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 29L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,42542, task_id=a208f729-962a-4338-90d0-23293cc34546 (api:54) 2019-04-13 22:08:36,562+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:40,093+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.19,42536 (api:48) 2019-04-13 22:08:40,093+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.19,42536 (api:54) 2019-04-13 22:08:40,093+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:44,007+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,59530 (api:48) 2019-04-13 22:08:44,008+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,59530 (api:54) 2019-04-13 22:08:44,008+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:44,458+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=192849d6-3597-46ea-bf8b-84d2eaffc52e (api:48) 2019-04-13 22:08:44,459+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000729298', 'lastCheck': '9.4', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000889308', 'lastCheck': '9.5', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000646977', 'lastCheck': '9.4', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000231074', 'lastCheck': '8.4', 'valid': True}} from=internal, task_id=192849d6-3597-46ea-bf8b-84d2eaffc52e (api:54) 2019-04-13 22:08:44,976+0200 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.0.19,42536 (api:48) 2019-04-13 22:08:44,997+0200 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.19,42536, task_id=01089a2b-63d9-4b1d-a5d4-7454b576b783 (api:48) 2019-04-13 22:08:44,998+0200 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000729298', 'lastCheck': '9.9', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000674376', 'lastCheck': '0.0', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000646977', 'lastCheck': '9.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000233191', 'lastCheck': '0.1', 'valid': True}} from=::ffff:192.168.0.19,42536, task_id=01089a2b-63d9-4b1d-a5d4-7454b576b783 (api:54) 2019-04-13 22:08:44,998+0200 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.0.19,42536, task_id=59470087-6e9a-4c15-8e8c-2d03f9f24bcd (api:48) 2019-04-13 22:08:44,999+0200 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.19,42536, task_id=59470087-6e9a-4c15-8e8c-2d03f9f24bcd (api:54) 2019-04-13 22:08:45,005+0200 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '0.86', 'nodeIndex': 0, 'cpuSys': '0.93', 'cpuIdle': '98.21'}, '0': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '3': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '5': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '4': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '8': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.73'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '10': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '13': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '17': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.54'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 3, 'memFree': '95855'}, '0': {'memPercent': 4, 'memFree': '95335'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '1', 'storageDomains': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000729298', 'lastCheck': '9.9', 'valid': True}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000646977', 'lastCheck': '9.9', 'valid': True}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000233191', 'lastCheck': '0.1', 'valid': True}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000674376', 'lastCheck': '0.0', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp0s29f0u2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp0s29f0u2', 'tx': '180', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186124.997155, 'rx': '14346', 'state': 'up'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '2', 'name': 'bond0', 'tx': '255232', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186124.997155, 'rx': '1306925', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186124.997155, 'rx': '0', 'state': 'down'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirtmgmt', 'tx': '242954', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186124.997155, 'rx': '1230650', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '30295846', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186124.997155, 'rx': '30295846', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186124.997155, 'rx': '0', 'state': 'down'}, 'eno2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno2', 'tx': '8077', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186124.997155, 'rx': '1288681', 'state': 'up'}, 'eno3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno3', 'tx': '247155', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186124.997155, 'rx': '18244', 'state': 'up'}, 'eno4': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1', 'name': 'eno4', 'tx': '1494', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1555186124.997155, 'rx': '21446', 'state': 'up'}, 'ens3f3': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f3', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186124.997155, 'rx': '0', 'state': 'down'}, 'ens3f2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186124.997155, 'rx': '0', 'state': 'down'}, 'ens3f1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186124.997155, 'rx': '0', 'state': 'down'}, 'ens3f0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ens3f0', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186124.997155, 'rx': '0', 'state': 'down'}, 'streamingnet': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'streamingnet', 'tx': '656', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1555186124.997155, 'rx': '12562', 'state': 'up'}}, 'txDropped': '5', 'anonHugePages': '44', 'ksmPages': 100, 'elapsedTime': '516.46', 'cpuLoad': '0.16', 'cpuSys': '0.06', 'diskStats': {'/var/log': {'free': '47799'}, '/var/run/vdsm/': {'free': '96655'}, '/tmp': {'free': '47799'}}, 'cpuUserVdsmd': '0.80', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 191835, 'bootTime': '1555185555', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2019-04-13T20:08:45 GMT', 'cpuUser': '0.06', 'memFree': 191579, 'cpuIdle': '99.88', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.40'}} from=::ffff:192.168.0.19,42536 (api:54) 2019-04-13 22:08:45,008+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312) 2019-04-13 22:08:46,633+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,42536, task_id=0d6cd9ef-9878-4dca-b95b-f1d239b9fc74 (api:48) 2019-04-13 22:08:46,638+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 29L}} from=::ffff:192.168.0.19,42536, task_id=0d6cd9ef-9878-4dca-b95b-f1d239b9fc74 (api:54) 2019-04-13 22:08:46,638+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:312) 2019-04-13 22:08:46,654+0200 INFO (jsonrpc/7) [vdsm.api] START getStoragePoolInfo(spUUID=u'59ef3a18-002f-02d1-0220-000000000124', options=None) from=::ffff:192.168.0.19,42542, task_id=1bfed00c-b982-498f-93ee-615c2d999434 (api:48) 2019-04-13 22:08:46,661+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 29L, 'domains': u'9c3f06cf-7475-448e-819b-f4f52fa7d782:Active,42bb6772-12f9-4bf2-9245-43caa41c5a2a:Active,93f02ede-ec5c-4641-8e97-2949f98e856d:Active,6a4cde72-6a6a-499e-9b6e-d490d91ca19e:Active', 'master_uuid': u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 47}, 'dominfo': {u'9c3f06cf-7475-448e-819b-f4f52fa7d782': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '3897524486144', 'version': 4}, u'42bb6772-12f9-4bf2-9245-43caa41c5a2a': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_isos/42bb6772-12f9-4bf2-9245-43caa41c5a2a/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '3719176388608', 'version': 0}, u'93f02ede-ec5c-4641-8e97-2949f98e856d': {'status': u'Active', 'diskfree': '3706427801600', 'isoprefix': '', 'alerts': [], 'disktotal': '4363815747584', 'version': 0}, u'6a4cde72-6a6a-499e-9b6e-d490d91ca19e': {'status': u'Active', 'diskfree': '997471551488', 'isoprefix': '', 'alerts': [], 'disktotal': '997511397376', 'version': 4}}} from=::ffff:192.168.0.19,42542, task_id=1bfed00c-b982-498f-93ee-615c2d999434 (api:54) 2019-04-13 22:08:46,662+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:312) 2019-04-13 22:08:51,652+0200 INFO (jsonrpc/3) [api.virt] START create(vmParams={u'xml': u'gogscb6ca3d9-ff5b-4c5d-91f9-08a980af1de620971522097152838860816oVirtOS-NAME:OS-VERSION:HOST-SERIAL:cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6Westmere
0066d0ee-95b6-4554-bf20-84337780dfe8
hvm20484.359ef3a18-002f-02d1-0220-000000000124ada24be6-3165-4a99-a002-e951d8807deb0066d0ee-95b6-4554-bf20-84337780dfe89c3f06cf-7475-448e-819b-f4f52fa7d782falseauto_resume
'}) from=::ffff:192.168.0.19,42536, flow_id=6180f9cb-8bd6-4267-923b-6adef080adb4, vmId= (api:48) 2019-04-13 22:08:51,667+0200 INFO (jsonrpc/3) [api.virt] FINISH create return={'status': {'message': 'Done', 'code': 0}, 'vmList': {'status': 'WaitForLaunch', 'maxMemSize': 8192, 'acpiEnable': 'true', 'emulatedMachine': 'pc-i440fx-rhel7.6.0', 'tabletEnable': 'true', 'vmId': 'cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6', 'memGuaranteedSize': 2048, 'timeOffset': '0', 'smpThreadsPerCore': '1', 'cpuType': 'Westmere', 'guestDiskMapping': {}, 'arch': 'x86_64', 'smp': '1', 'guestNumaNodes': [{'nodeIndex': 0, 'cpus': '0', 'memory': '2048'}], u'xml': u'gogscb6ca3d9-ff5b-4c5d-91f9-08a980af1de620971522097152838860816oVirtOS-NAME:OS-VERSION:HOST-SERIAL:cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6Westmere
0066d0ee-95b6-4554-bf20-84337780dfe8
hvm20484.359ef3a18-002f-02d1-0220-000000000124ada24be6-3165-4a99-a002-e951d8807deb0066d0ee-95b6-4554-bf20-84337780dfe89c3f06cf-7475-448e-819b-f4f52fa7d782falseauto_resume
', 'smpCoresPerSocket': '1', 'kvmEnable': 'true', 'bootMenuEnable': 'false', 'devices': [], 'custom': {}, 'maxVCpus': '16', 'statusTime': '4295243570', 'vmName': 'gogs', 'maxMemSlots': 16}} from=::ffff:192.168.0.19,42536, flow_id=6180f9cb-8bd6-4267-923b-6adef080adb4, vmId= (api:54) 2019-04-13 22:08:51,667+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call VM.create succeeded in 0.02 seconds (__init__:312) 2019-04-13 22:08:51,669+0200 INFO (vm/cb6ca3d9) [virt.vm] (vmId='cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6') VM wrapper has started (vm:2746) 2019-04-13 22:08:51,704+0200 INFO (vm/cb6ca3d9) [vdsm.api] START getVolumeSize(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', volUUID='ada24be6-3165-4a99-a002-e951d8807deb', options=None) from=internal, task_id=4689be95-5e27-4a11-8615-fc7f03b94ec7 (api:48) 2019-04-13 22:08:51,719+0200 INFO (vm/cb6ca3d9) [vdsm.api] FINISH getVolumeSize return={'truesize': '2116492288', 'apparentsize': '3948150784'} from=internal, task_id=4689be95-5e27-4a11-8615-fc7f03b94ec7 (api:54) 2019-04-13 22:08:51,719+0200 INFO (vm/cb6ca3d9) [vdsm.api] START prepareImage(sdUUID='9c3f06cf-7475-448e-819b-f4f52fa7d782', spUUID='59ef3a18-002f-02d1-0220-000000000124', imgUUID='0066d0ee-95b6-4554-bf20-84337780dfe8', leafUUID='ada24be6-3165-4a99-a002-e951d8807deb', allowIllegal=False) from=internal, task_id=94fa36d2-606e-4815-9ed0-eb9d5b0198a5 (api:48) 2019-04-13 22:08:52,403+0200 INFO (vm/cb6ca3d9) [storage.StorageDomain] Fixing permissions on /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/0066d0ee-95b6-4554-bf20-84337780dfe8/ada24be6-3165-4a99-a002-e951d8807deb (fileSD:629) 2019-04-13 22:08:52,404+0200 INFO (vm/cb6ca3d9) [storage.StorageDomain] Creating domain run directory u'/var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782' (fileSD:583) 2019-04-13 22:08:52,404+0200 INFO (vm/cb6ca3d9) [storage.fileUtils] Creating directory: /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782 mode: None (fileUtils:199) 2019-04-13 22:08:52,405+0200 INFO (vm/cb6ca3d9) [storage.StorageDomain] Creating symlink from /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/0066d0ee-95b6-4554-bf20-84337780dfe8 to /var/run/vdsm/storage/9c3f06cf-7475-448e-819b-f4f52fa7d782/0066d0ee-95b6-4554-bf20-84337780dfe8 (fileSD:586) 2019-04-13 22:08:52,410+0200 INFO (vm/cb6ca3d9) [vdsm.api] FINISH prepareImage return={'info': {'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/0066d0ee-95b6-4554-bf20-84337780dfe8/ada24be6-3165-4a99-a002-e951d8807deb', 'type': 'file'}, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/0066d0ee-95b6-4554-bf20-84337780dfe8/ada24be6-3165-4a99-a002-e951d8807deb', 'imgVolumesInfo': [{'domainID': '9c3f06cf-7475-448e-819b-f4f52fa7d782', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/0066d0ee-95b6-4554-bf20-84337780dfe8/ada24be6-3165-4a99-a002-e951d8807deb', 'volumeID': u'ada24be6-3165-4a99-a002-e951d8807deb', 'leasePath': u'/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/0066d0ee-95b6-4554-bf20-84337780dfe8/ada24be6-3165-4a99-a002-e951d8807deb.lease', 'imageID': '0066d0ee-95b6-4554-bf20-84337780dfe8'}]} from=internal, task_id=94fa36d2-606e-4815-9ed0-eb9d5b0198a5 (api:54) 2019-04-13 22:08:52,411+0200 INFO (vm/cb6ca3d9) [vds] prepared volume path: /rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/0066d0ee-95b6-4554-bf20-84337780dfe8/ada24be6-3165-4a99-a002-e951d8807deb (clientIF:499) 2019-04-13 22:08:52,411+0200 INFO (vm/cb6ca3d9) [virt.vm] (vmId='cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6') Enabling drive monitoring (drivemonitor:56) 2019-04-13 22:08:52,440+0200 INFO (vm/cb6ca3d9) [virt.vm] (vmId='cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6') drive 'vda' path: 'file=/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/0066d0ee-95b6-4554-bf20-84337780dfe8/ada24be6-3165-4a99-a002-e951d8807deb' -> u'*file=/rhev/data-center/mnt/nas.discovery.intern:_mnt_d2600_vms/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/0066d0ee-95b6-4554-bf20-84337780dfe8/ada24be6-3165-4a99-a002-e951d8807deb' (storagexml:333) 2019-04-13 22:08:52,631+0200 INFO (vm/cb6ca3d9) [root] /usr/libexec/vdsm/hooks/before_device_create/10_ovirt_provider_ovn_hook: rc=0 err= (hooks:114) 2019-04-13 22:08:52,806+0200 INFO (vm/cb6ca3d9) [root] /usr/libexec/vdsm/hooks/before_device_create/20_ovirt_provider_ovn_vhostuser_hook: rc=0 err= (hooks:114) 2019-04-13 22:08:53,085+0200 INFO (vm/cb6ca3d9) [root] /usr/libexec/vdsm/hooks/before_device_create/50_openstacknet: rc=0 err= (hooks:114) 2019-04-13 22:08:53,267+0200 INFO (vm/cb6ca3d9) [root] /usr/libexec/vdsm/hooks/before_device_create/50_vmfex: rc=0 err= (hooks:114) 2019-04-13 22:08:53,538+0200 INFO (vm/cb6ca3d9) [root] /usr/libexec/vdsm/hooks/before_device_create/openstacknet_utils.py: rc=0 err= (hooks:114) 2019-04-13 22:08:53,790+0200 INFO (vm/cb6ca3d9) [root] /usr/libexec/vdsm/hooks/before_vm_start/50_hostedengine: rc=0 err= (hooks:114) 2019-04-13 22:08:53,965+0200 INFO (vm/cb6ca3d9) [root] /usr/libexec/vdsm/hooks/before_vm_start/50_vhostmd: rc=0 err= (hooks:114) 2019-04-13 22:08:53,970+0200 INFO (vm/cb6ca3d9) [virt.vm] (vmId='cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6') gogs cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6 2097152 2097152 8388608 16 oVirt oVirt Node 7-6.1810.2.el7.centos 2650319A-1C29-11E0-8699-E41F13B59414 cb6ca3d9-ff5b-4c5d-91f9-08a980af1de6 Westmere