<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">Hi,</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Errors occurred at 10.43 AM and 10.46 AM</div><div class="">node00.starlett.lv - 192.168.0.4 - oVirt host engine (separate PC, not hosted)</div><div class="">node10.starlett.lv - 192.168.0.5 - host #1 of DC #1, export domain from which was detached from DC #1</div><div class="">node11.starlett.lv - 192.168.0.6 - host #1 of DC #2, </div><div class=""><br class=""></div>Logs from DC#2 node11, to which I’m trying to attach export domain located at NFS share node10.<div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class="" style="word-wrap: break-word; line-break: after-white-space;"><div class=""><div class=""><font color="#aa7942" class=""><b class="">grep 570ec5d9-fff5-4656-afbd-<wbr class="">90b3207a616e</b></font></div></div></div></blockquote></div></div></blockquote>within vdsm.log returned nothing, so I did</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><font color="#942192" class=""><b class="">grep -n 10:43 vdsm.log | tail -1000</b></font><br class="">1011:2018-03-21 06:10:43,077+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br class="">1012:2018-03-21 06:10:43,077+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br class="">1013:2018-03-21 06:10:43,077+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573)<br class="">1014:2018-03-21 06:10:43,868+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br class="">1015:2018-03-21 06:10:43,868+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br class="">1016:2018-03-21 06:10:43,868+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br class="">19729:2018-03-21 09:10:43,641+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br class="">19730:2018-03-21 09:10:43,641+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br class="">19731:2018-03-21 09:10:43,642+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br class="">29302:2018-03-21 10:43:00,085+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=7b690c8a-5470-44da-a8f3-7e7e9b018e88 (api:46)<br class="">29303:2018-03-21 10:43:00,085+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000477038', 'lastCheck': '0.5', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': '0.4', 'valid': True}} from=internal, task_id=7b690c8a-5470-44da-a8f3-7e7e9b018e88 (api:52)<br class="">29304:2018-03-21 10:43:00,086+0200 INFO (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=9a625d43-a03c-429c-856f-9aa8e5ff65b5 (api:46)<br class="">29305:2018-03-21 10:43:00,086+0200 INFO (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=9a625d43-a03c-429c-856f-9aa8e5ff65b5 (api:52)<br class="">29306:2018-03-21 10:43:05,382+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br class="">29307:2018-03-21 10:43:05,383+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br class="">29308:2018-03-21 10:43:05,383+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br class="">29309:2018-03-21 10:43:06,248+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)<br class="">29310:2018-03-21 10:43:06,249+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=f80f9bb8-2e84-4f45-ab58-09a88e808cf3 (api:46)<br class="">29311:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000477038', 'lastCheck': '6.7', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': '6.6', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=f80f9bb8-2e84-4f45-ab58-09a88e808cf3 (api:52)<br class="">29312:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=ddd7064d-cd12-43bf-a177-b2a93cc9035e (api:46)<br class="">29313:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=ddd7064d-cd12-43bf-a177-b2a93cc9035e (api:52)<br class="">29314:2018-03-21 10:43:06,256+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '10': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.47'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '12': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '15': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.47', 'cpuIdle': '99.33'}, '14': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '1': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': '1.60', 'cpuIdle': '97.27'}, '0': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.26'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '2': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.54'}, '5': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.74'}, '4': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '7': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.60', 'cpuIdle': '99.27'}, '6': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '8': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15142'}, '0': {'memPercent': 5, 'memFree': '15349'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000477038', 'lastCheck': '6.7', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': '6.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'enp3s0f0', 'tx': '1091287978', 'txDropped': '0', 'rx': '11914868563', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1439'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'ovirtmgmt', 'tx': '1048639112', 'txDropped': '0', 'rx': '11615209092', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'lo', 'tx': '58057075945', 'txDropped': '0', 'rx': '58057075945', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621780.072418, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621780.072418, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621780.072418, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76345.09', 'cpuLoad': '0.16', 'cpuSys': '0.36', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31081, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1439', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:43:06 GMT', 'cpuUser': '0.21', 'memFree': 31337, 'cpuIdle': '99.43', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.93'}} from=::ffff:192.168.0.4,49914 (api:52)<br class="">29315:2018-03-21 10:43:06,258+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)<br class="">29316:2018-03-21 10:43:06,632+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br class="">29317:2018-03-21 10:43:06,632+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br class="">29318:2018-03-21 10:43:06,633+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br class="">29319:2018-03-21 10:43:07,415+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=276c13c7-61ec-4782-a0d9-5cd0f0b0a729 (api:46)<br class="">29320:2018-03-21 10:43:07,420+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=276c13c7-61ec-4782-a0d9-5cd0f0b0a729 (api:52)<br class="">29321:2018-03-21 10:43:07,420+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)<br class="">29322:2018-03-21 10:43:07,427+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=344b18df-a428-429a-b49f-381f901d1618 (api:46)<br class="">29323:2018-03-21 10:43:07,434+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=344b18df-a428-429a-b49f-381f901d1618 (api:52)<br class="">29324:2018-03-21 10:43:07,435+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)<br class="">29325:2018-03-21 10:43:13,839+0200 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/1 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524', u'connection': u'node10.starlett.lv:/vmdata/nfs/exports', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'protocol_version': u'auto', u'password': '********', u'port': u''}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': u'5f98dfd5-998f-44a6-8594-e5f40144b0ed'} at 0x4076150> timeout=60, duration=120 at 0x402fe50> task#=3754 at 0x353e690>, traceback:<br class="">29376:2018-03-21 10:43:15,104+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=672a883e-1d60-4981-9eee-bdcac2ec30c1 (api:46)<br class="">29377:2018-03-21 10:43:15,105+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000488777', 'lastCheck': '5.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000481962', 'lastCheck': '5.4', 'valid': True}} from=internal, task_id=672a883e-1d60-4981-9eee-bdcac2ec30c1 (api:52)<br class="">29378:2018-03-21 10:43:15,105+0200 INFO (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=55f1b797-da2a-4b07-affe-5b9e41a74307 (api:46)<br class="">29379:2018-03-21 10:43:15,106+0200 INFO (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=55f1b797-da2a-4b07-affe-5b9e41a74307 (api:52)<br class="">29380:2018-03-21 10:43:17,645+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=52a6ddeb-3c20-4f3f-b27d-94d19580b473 (api:46)<br class="">29381:2018-03-21 10:43:17,649+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=52a6ddeb-3c20-4f3f-b27d-94d19580b473 (api:52)<br class="">29382:2018-03-21 10:43:17,650+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)<br class="">29383:2018-03-21 10:43:17,656+0200 INFO (jsonrpc/2) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=1a103883-6a6a-4d61-ade6-96559591ddb0 (api:46)<br class="">29384:2018-03-21 10:43:17,663+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=1a103883-6a6a-4d61-ade6-96559591ddb0 (api:52)<br class="">29385:2018-03-21 10:43:17,663+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)<br class="">29386:2018-03-21 10:43:18,882+0200 ERROR (jsonrpc/1) [storage.HSM] Could not connect to storageServer (hsm:2407)<br class="">29406:2018-03-21 10:43:18,883+0200 INFO (jsonrpc/1) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 477, 'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524'}]} from=::ffff:192.168.0.4,49914, flow_id=4a53b512, task_id=e598cbe0-cde8-4c73-b526-4398df05e67f (api:52)<br class="">29407:2018-03-21 10:43:18,883+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 125.04 seconds (__init__:573)<br class="">29408:2018-03-21 10:43:20,387+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br class="">29409:2018-03-21 10:43:20,388+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br class="">29410:2018-03-21 10:43:20,388+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573)<br class="">29411:2018-03-21 10:43:21,497+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)<br class="">29412:2018-03-21 10:43:21,498+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=a7469c1f-3d6d-42e7-8798-5c5335da4a9b (api:46)<br class="">29413:2018-03-21 10:43:21,498+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000445096', 'lastCheck': '1.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000296586', 'lastCheck': '1.8', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=a7469c1f-3d6d-42e7-8798-5c5335da4a9b (api:52)<br class="">29414:2018-03-21 10:43:21,499+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=9fa5dae3-e2bd-4186-9a1d-afb0aa6e2511 (api:46)<br class="">29415:2018-03-21 10:43:21,499+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=9fa5dae3-e2bd-4186-9a1d-afb0aa6e2511 (api:52)<br class="">29416:2018-03-21 10:43:21,505+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '10': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '12': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '15': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.60'}, '1': {'cpuUser': '1.00', 'nodeIndex': 1, 'cpuSys': '1.00', 'cpuIdle': '98.00'}, '0': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '3': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '2': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.66'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '7': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '6': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '8': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15142'}, '0': {'memPercent': 5, 'memFree': '15351'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000445096', 'lastCheck': '1.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000296586', 'lastCheck': '1.8', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'enp3s0f0', 'tx': '1091295628', 'txDropped': '0', 'rx': '11914875210', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1442'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'ovirtmgmt', 'tx': '1048646480', 'txDropped': '0', 'rx': '11615214750', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'lo', 'tx': '58066397668', 'txDropped': '0', 'rx': '58066397668', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621795.091263, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621795.091263, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621795.091263, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76360.34', 'cpuLoad': '0.15', 'cpuSys': '0.18', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31031, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1442', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:43:21 GMT', 'cpuUser': '0.15', 'memFree': 31287, 'cpuIdle': '99.67', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.67'}} from=::ffff:192.168.0.4,49914 (api:52)<br class="">29417:2018-03-21 10:43:21,508+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.02 seconds (__init__:573)<br class="">29418:2018-03-21 10:43:21,651+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br class="">29419:2018-03-21 10:43:21,651+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br class="">29420:2018-03-21 10:43:21,652+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br class="">29421:2018-03-21 10:43:27,764+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=46d19cab-8903-4c52-9bb2-4dd8f370997e (api:46)<br class="">29422:2018-03-21 10:43:27,769+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=46d19cab-8903-4c52-9bb2-4dd8f370997e (api:52)<br class="">29423:2018-03-21 10:43:27,770+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)<br class="">29424:2018-03-21 10:43:27,815+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=290acc73-8f25-4fd7-9263-7bd05d46430a (api:46)<br class="">29425:2018-03-21 10:43:27,822+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=290acc73-8f25-4fd7-9263-7bd05d46430a (api:52)<br class="">29426:2018-03-21 10:43:27,823+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)<br class="">29427:2018-03-21 10:43:30,121+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=0abdebcf-2b0b-4d33-b552-c0f2ed3e567b (api:46)<br class="">29428:2018-03-21 10:43:30,122+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000402077', 'lastCheck': '0.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': '0.5', 'valid': True}} from=internal, task_id=0abdebcf-2b0b-4d33-b552-c0f2ed3e567b (api:52)<br class="">29429:2018-03-21 10:43:30,122+0200 INFO (periodic/3) [vdsm.api] START multipath_health() from=internal, task_id=acde7e7d-ab4a-4811-934f-4d312da33208 (api:46)<br class="">29430:2018-03-21 10:43:30,123+0200 INFO (periodic/3) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=acde7e7d-ab4a-4811-934f-4d312da33208 (api:52)<br class="">29431:2018-03-21 10:43:35,393+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br class="">29432:2018-03-21 10:43:35,394+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br class="">29433:2018-03-21 10:43:35,394+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br class="">29434:2018-03-21 10:43:36,671+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br class="">29435:2018-03-21 10:43:36,671+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br class="">29436:2018-03-21 10:43:36,672+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br class="">29437:2018-03-21 10:43:37,356+0200 INFO (jsonrpc/1) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)<br class="">29438:2018-03-21 10:43:37,357+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=bc14e253-d3d4-4cc3-9ae1-c0e6ac6e37f7 (api:46)<br class="">29439:2018-03-21 10:43:37,357+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000402077', 'lastCheck': '7.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': '7.7', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=bc14e253-d3d4-4cc3-9ae1-c0e6ac6e37f7 (api:52)<br class="">29440:2018-03-21 10:43:37,358+0200 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=23c698b8-d5bd-4746-b738-97b10f3e4bfb (api:46)<br class="">29441:2018-03-21 10:43:37,358+0200 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=23c698b8-d5bd-4746-b738-97b10f3e4bfb (api:52)<br class="">29442:2018-03-21 10:43:37,365+0200 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '10': {'cpuUser': '3.06', 'nodeIndex': 0, 'cpuSys': '0.80', 'cpuIdle': '96.14'}, '13': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '12': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '15': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '14': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.33'}, '1': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': '1.33', 'cpuIdle': '97.54'}, '0': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.20'}, '3': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '2': {'cpuUser': '0.67', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '98.93'}, '5': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.74'}, '4': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '7': {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': '0.73', 'cpuIdle': '98.94'}, '6': {'cpuUser': '5.39', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '94.14'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '8': {'cpuUser': '0.53', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.07'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15143'}, '0': {'memPercent': 5, 'memFree': '15348'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000402077', 'lastCheck': '7.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': '7.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'enp3s0f0', 'tx': '1091305295', 'txDropped': '0', 'rx': '11914880953', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1442'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'ovirtmgmt', 'tx': '1048655897', 'txDropped': '0', 'rx': '11615220341', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'lo', 'tx': '58078891247', 'txDropped': '0', 'rx': '58078891247', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621810.108511, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621810.108511, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621810.108511, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76376.19', 'cpuLoad': '0.16', 'cpuSys': '0.39', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.00', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1442', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:43:37 GMT', 'cpuUser': '0.79', 'memFree': 31336, 'cpuIdle': '98.82', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.73'}} from=::ffff:192.168.0.4,49914 (api:52)<br class="">29443:2018-03-21 10:43:37,367+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)<br class="">29444:2018-03-21 10:43:38,124+0200 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=9d2a82dd-c76c-4eb5-a1e1-b49bf004384e (api:46)<br class="">29445:2018-03-21 10:43:38,128+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=9d2a82dd-c76c-4eb5-a1e1-b49bf004384e (api:52)<br class="">29446:2018-03-21 10:43:38,129+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)<br class="">29447:2018-03-21 10:43:38,171+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=62d44b0b-20b2-47ae-bfb8-d4717e301b69 (api:46)<br class="">29448:2018-03-21 10:43:38,177+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=62d44b0b-20b2-47ae-bfb8-d4717e301b69 (api:52)<br class="">29449:2018-03-21 10:43:38,178+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)<br class="">29450:2018-03-21 10:43:45,147+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=49c33285-a222-464d-8984-4e75a5c6354a (api:46)<br class="">29451:2018-03-21 10:43:45,148+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000454102', 'lastCheck': '5.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000347381', 'lastCheck': '5.5', 'valid': True}} from=internal, task_id=49c33285-a222-464d-8984-4e75a5c6354a (api:52)<br class="">29452:2018-03-21 10:43:45,148+0200 INFO (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=eed6b564-76b1-4f9b-80a2-1f0f6d9bbe32 (api:46)<br class="">29453:2018-03-21 10:43:45,149+0200 INFO (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=eed6b564-76b1-4f9b-80a2-1f0f6d9bbe32 (api:52)<br class="">29454:2018-03-21 10:43:48,405+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=84651a37-7d13-41a9-9dcf-2169a9c46fc5 (api:46)<br class="">29455:2018-03-21 10:43:48,410+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=84651a37-7d13-41a9-9dcf-2169a9c46fc5 (api:52)<br class="">29456:2018-03-21 10:43:48,411+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)<br class="">29457:2018-03-21 10:43:48,417+0200 INFO (jsonrpc/4) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=4ebd1d1a-11ac-41ee-b459-7e45f008776b (api:46)<br class="">29458:2018-03-21 10:43:48,423+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=4ebd1d1a-11ac-41ee-b459-7e45f008776b (api:52)<br class="">29459:2018-03-21 10:43:48,424+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)<br class="">29460:2018-03-21 10:43:50,399+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br class="">29461:2018-03-21 10:43:50,400+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br class="">29462:2018-03-21 10:43:50,400+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br class="">29463:2018-03-21 10:43:51,694+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br class="">29464:2018-03-21 10:43:51,694+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br class="">29465:2018-03-21 10:43:51,695+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br class="">29466:2018-03-21 10:43:52,516+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)<br class="">29467:2018-03-21 10:43:52,517+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=0a88f0dc-eccf-4d24-9ac2-22e079a1480a (api:46)<br class="">29468:2018-03-21 10:43:52,517+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000487937', 'lastCheck': '3.0', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000455956', 'lastCheck': '2.9', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=0a88f0dc-eccf-4d24-9ac2-22e079a1480a (api:52)<br class="">29469:2018-03-21 10:43:52,518+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=a7a44ff7-a137-41b6-9ad5-2c0b32ab79ec (api:46)<br class="">29470:2018-03-21 10:43:52,518+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=a7a44ff7-a137-41b6-9ad5-2c0b32ab79ec (api:52)<br class="">29471:2018-03-21 10:43:52,524+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '10': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '13': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '15': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '1': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': '1.13', 'cpuIdle': '97.74'}, '0': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '5': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '4': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.60'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '99.67'}, '6': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '8': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15143'}, '0': {'memPercent': 5, 'memFree': '15349'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000487937', 'lastCheck': '3.0', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000455956', 'lastCheck': '2.9', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'enp3s0f0', 'tx': '1091312739', 'txDropped': '0', 'rx': '11914899051', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1443'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'ovirtmgmt', 'tx': '1048663107', 'txDropped': '0', 'rx': '11615236520', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'lo', 'tx': '58090295494', 'txDropped': '0', 'rx': '58090295494', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621825.13588, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621825.13588, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621825.13588, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76391.35', 'cpuLoad': '0.15', 'cpuSys': '0.20', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.13', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1443', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:43:52 GMT', 'cpuUser': '0.14', 'memFree': 31336, 'cpuIdle': '99.66', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.73'}} from=::ffff:192.168.0.4,49914 (api:52)<br class="">29472:2018-03-21 10:43:52,526+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)<br class="">29473:2018-03-21 10:43:58,686+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=d6d29cf6-c083-4639-91a2-f062eae4e629 (api:46)<br class="">29474:2018-03-21 10:43:58,690+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=d6d29cf6-c083-4639-91a2-f062eae4e629 (api:52)<br class="">29475:2018-03-21 10:43:58,690+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)<br class="">29476:2018-03-21 10:43:58,697+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=9dbd91df-dd9c-469c-8308-bfbfc228937c (api:46)<br class="">29477:2018-03-21 10:43:58,703+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=9dbd91df-dd9c-469c-8308-bfbfc228937c (api:52)<br class="">29478:2018-03-21 10:43:58,704+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)<br class=""><br class=""><br class=""><br class=""><b class=""><font color="#942192" class="">grep -n 10:46 vdsm.log | tail -1000</font></b><br class="">1017:2018-03-21 06:10:46,882+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)<br class="">1018:2018-03-21 06:10:46,883+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=6e045b00-88b2-469b-80d6-af38303f8c32 (api:46)<br class="">1019:2018-03-21 06:10:46,883+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000332785', 'lastCheck': '7.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000203914', 'lastCheck': '5.6', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=6e045b00-88b2-469b-80d6-af38303f8c32 (api:52)<br class="">1020:2018-03-21 06:10:46,884+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=475a5b40-4a43-42ee-9524-2bdd220aed83 (api:46)<br class="">1021:2018-03-21 06:10:46,884+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=475a5b40-4a43-42ee-9524-2bdd220aed83 (api:52)<br class="">1022:2018-03-21 06:10:46,889+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '10': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '12': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '15': {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.40'}, '14': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '1': {'cpuUser': '3.00', 'nodeIndex': 1, 'cpuSys': '1.40', 'cpuIdle': '95.60'}, '0': {'cpuUser': '0.73', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.07'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '2': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '4': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '7': {'cpuUser': '0.80', 'nodeIndex': 1, 'cpuSys': '0.80', 'cpuIdle': '98.40'}, '6': {'cpuUser': '5.46', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '94.27'}, '9': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '8': {'cpuUser': '1.27', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '98.26'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15128'}, '0': {'memPercent': 5, 'memFree': '15366'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000332785', 'lastCheck': '7.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000203914', 'lastCheck': '5.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'enp3s0f0', 'tx': '1082264809', 'txDropped': '0', 'rx': '11907188594', 'rxErrors': '0', 'speed': '100', 'rxDropped': '289'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'ovirtmgmt', 'tx': '1039869554', 'txDropped': '0', 'rx': '11608697383', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'lo', 'tx': '45587320171', 'txDropped': '0', 'rx': '45587320171', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '53620', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521605437.203776, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521605437.203776, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521605437.203776, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '60005.72', 'cpuLoad': '0.06', 'cpuSys': '0.38', 'diskStats': {'/var/log': {'free': '7357'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31084, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '289', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T04:10:46 GMT', 'cpuUser': '0.83', 'memFree': 31340, 'cpuIdle': '98.79', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.80'}} from=::ffff:192.168.0.4,49914 (api:52)<br class="">1023:2018-03-21 06:10:46,891+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)<br class="">13522:2018-03-21 08:10:46,827+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br class="">13523:2018-03-21 08:10:46,828+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br class="">13524:2018-03-21 08:10:46,828+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br class="">29734:2018-03-21 10:46:00,338+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=af099ce6-0f44-461b-9e8e-99de63b3884f (api:46)<br class="">29735:2018-03-21 10:46:00,339+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470245', 'lastCheck': '0.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000476935', 'lastCheck': '0.7', 'valid': True}} from=internal, task_id=af099ce6-0f44-461b-9e8e-99de63b3884f (api:52)<br class="">29736:2018-03-21 10:46:00,339+0200 INFO (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=4a5709d9-e322-44eb-a8a4-bbe3f5e5e1bd (api:46)<br class="">29737:2018-03-21 10:46:00,340+0200 INFO (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=4a5709d9-e322-44eb-a8a4-bbe3f5e5e1bd (api:52)<br class="">29738:2018-03-21 10:46:04,637+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=5b9300f7-f8c4-4f63-a49c-45ca8119d3b9 (api:46)<br class="">29739:2018-03-21 10:46:04,642+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=5b9300f7-f8c4-4f63-a49c-45ca8119d3b9 (api:52)<br class="">29740:2018-03-21 10:46:04,642+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)<br class="">29741:2018-03-21 10:46:04,648+0200 INFO (jsonrpc/1) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=f4db5bfa-e131-4f9a-baaa-e7555c996ab0 (api:46)<br class="">29742:2018-03-21 10:46:04,655+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=f4db5bfa-e131-4f9a-baaa-e7555c996ab0 (api:52)<br class="">29743:2018-03-21 10:46:04,656+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:573)<br class="">29744:2018-03-21 10:46:05,451+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br class="">29745:2018-03-21 10:46:05,452+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br class="">29746:2018-03-21 10:46:05,452+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br class="">29747:2018-03-21 10:46:06,903+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br class="">29748:2018-03-21 10:46:06,904+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br class="">29749:2018-03-21 10:46:06,904+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br class="">29750:2018-03-21 10:46:08,791+0200 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/6 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524', u'connection': u'node10.starlett.lv:/vmdata/nfs/exports', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'protocol_version': u'auto', u'password': '********', u'port': u''}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': u'71143eef-aac6-4996-9783-3e0e3da180c3'} at 0x3604fd0> timeout=60, duration=120 at 0x3604e10> task#=3762 at 0x3541650>, traceback:<br class="">29801:2018-03-21 10:46:12,852+0200 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)<br class="">29802:2018-03-21 10:46:12,853+0200 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=391fdab9-d3c2-4075-9c34-177a07a21ec3 (api:46)<br class="">29803:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000492277', 'lastCheck': '3.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': '3.2', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=391fdab9-d3c2-4075-9c34-177a07a21ec3 (api:52)<br class="">29804:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=7e53adbc-3974-4f51-bfc8-8fbd3ca6b749 (api:46)<br class="">29805:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=7e53adbc-3974-4f51-bfc8-8fbd3ca6b749 (api:52)<br class="">29806:2018-03-21 10:46:12,861+0200 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '10': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '12': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '15': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '14': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '1': {'cpuUser': '1.00', 'nodeIndex': 1, 'cpuSys': '1.33', 'cpuIdle': '97.67'}, '0': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '2': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.47'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '7': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.53'}, '6': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '8': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15142'}, '0': {'memPercent': 5, 'memFree': '15348'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000492277', 'lastCheck': '3.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': '3.2', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'enp3s0f0', 'tx': '1091387790', 'txDropped': '0', 'rx': '11914959714', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1451'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'ovirtmgmt', 'tx': '1048735814', 'txDropped': '0', 'rx': '11615289173', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'lo', 'tx': '58192068637', 'txDropped': '0', 'rx': '58192068637', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621960.326184, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621960.326184, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621960.326184, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76531.69', 'cpuLoad': '0.17', 'cpuSys': '0.32', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.00', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31081, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1451', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:46:12 GMT', 'cpuUser': '0.21', 'memFree': 31337, 'cpuIdle': '99.46', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.80'}} from=::ffff:192.168.0.4,49914 (api:52)<br class="">29807:2018-03-21 10:46:12,863+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)<br class="">29808:2018-03-21 10:46:13,829+0200 ERROR (jsonrpc/6) [storage.HSM] Could not connect to storageServer (hsm:2407)<br class="">29828:2018-03-21 10:46:13,829+0200 INFO (jsonrpc/6) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 477, 'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524'}]} from=::ffff:192.168.0.4,49914, flow_id=1435fc81, task_id=0a828d2c-d9f4-4f83-a9e9-7393159d5323 (api:52)<br class="">29829:2018-03-21 10:46:13,830+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 125.04 seconds (__init__:573)<br class="">29830:2018-03-21 10:46:14,767+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=49cd2ae6-7943-47c9-ad86-9e0b7e58bca3 (api:46)<br class="">29831:2018-03-21 10:46:14,772+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=49cd2ae6-7943-47c9-ad86-9e0b7e58bca3 (api:52)<br class="">29832:2018-03-21 10:46:14,773+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)<br class="">29833:2018-03-21 10:46:14,815+0200 INFO (jsonrpc/2) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=19909616-efb4-490d-982c-66eda9ca4381 (api:46)<br class="">29834:2018-03-21 10:46:14,822+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=19909616-efb4-490d-982c-66eda9ca4381 (api:52)<br class="">29835:2018-03-21 10:46:14,823+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)<br class="">29836:2018-03-21 10:46:15,360+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=da74ea9f-164b-450d-b836-6818caa3fdc5 (api:46)<br class="">29837:2018-03-21 10:46:15,360+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000492277', 'lastCheck': '5.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': '5.7', 'valid': True}} from=internal, task_id=da74ea9f-164b-450d-b836-6818caa3fdc5 (api:52)<br class="">29838:2018-03-21 10:46:15,361+0200 INFO (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=06245e2b-e8ca-41c4-90bf-4294d0c699b8 (api:46)<br class="">29839:2018-03-21 10:46:15,361+0200 INFO (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=06245e2b-e8ca-41c4-90bf-4294d0c699b8 (api:52)<br class="">29840:2018-03-21 10:46:20,456+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br class="">29841:2018-03-21 10:46:20,457+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br class="">29842:2018-03-21 10:46:20,457+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573)<br class="">29843:2018-03-21 10:46:21,925+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br class="">29844:2018-03-21 10:46:21,926+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br class="">29845:2018-03-21 10:46:21,926+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br class="">29846:2018-03-21 10:46:24,930+0200 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=d964535d-e5eb-42bc-b324-432a05c364da (api:46)<br class="">29847:2018-03-21 10:46:24,935+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=d964535d-e5eb-42bc-b324-432a05c364da (api:52)<br class="">29848:2018-03-21 10:46:24,935+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:573)<br class="">29849:2018-03-21 10:46:24,987+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=167dcd23-04b5-4de3-9c81-7d25ba56407a (api:46)<br class="">29850:2018-03-21 10:46:25,022+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=167dcd23-04b5-4de3-9c81-7d25ba56407a (api:52)<br class="">29851:2018-03-21 10:46:25,023+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.03 seconds (__init__:573)<br class="">29852:2018-03-21 10:46:28,010+0200 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)<br class="">29853:2018-03-21 10:46:28,011+0200 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=011616ce-53c9-4cfc-8b29-247b1be03409 (api:46)<br class="">29854:2018-03-21 10:46:28,011+0200 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000218955', 'lastCheck': '8.4', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000351741', 'lastCheck': '8.4', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=011616ce-53c9-4cfc-8b29-247b1be03409 (api:52)<br class="">29855:2018-03-21 10:46:28,012+0200 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=85b42a8d-1bf1-495d-ad32-a1f9710a6468 (api:46)<br class="">29856:2018-03-21 10:46:28,012+0200 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=85b42a8d-1bf1-495d-ad32-a1f9710a6468 (api:52)<br class="">29857:2018-03-21 10:46:28,018+0200 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '10': {'cpuUser': '1.73', 'nodeIndex': 0, 'cpuSys': '0.53', 'cpuIdle': '97.74'}, '13': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '12': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '15': {'cpuUser': '0.67', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.06'}, '14': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.47'}, '1': {'cpuUser': '1.26', 'nodeIndex': 1, 'cpuSys': '1.33', 'cpuIdle': '97.41'}, '0': {'cpuUser': '0.40', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.53'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.73'}, '5': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '4': {'cpuUser': '13.78', 'nodeIndex': 0, 'cpuSys': '0.80', 'cpuIdle': '85.42'}, '7': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.67', 'cpuIdle': '98.86'}, '6': {'cpuUser': '1.07', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '98.66'}, '9': {'cpuUser': '1.20', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '98.53'}, '8': {'cpuUser': '0.80', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '98.87'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15136'}, '0': {'memPercent': 5, 'memFree': '15357'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000218955', 'lastCheck': '8.4', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000351741', 'lastCheck': '8.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'enp3s0f0', 'tx': '1091397599', 'txDropped': '0', 'rx': '11914966591', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1454'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'ovirtmgmt', 'tx': '1048745347', 'txDropped': '0', 'rx': '11615294897', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'lo', 'tx': '58205583384', 'txDropped': '0', 'rx': '58205583384', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621975.346633, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621975.346633, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621975.346633, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76546.85', 'cpuLoad': '0.18', 'cpuSys': '0.39', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.20', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31031, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1454', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:46:28 GMT', 'cpuUser': '1.38', 'memFree': 31287, 'cpuIdle': '98.23', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.87'}} from=::ffff:192.168.0.4,49914 (api:52)<br class="">29858:2018-03-21 10:46:28,020+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)<br class="">29859:2018-03-21 10:46:29,667+0200 INFO (itmap/0) [IOProcessClient] Starting client ioprocess-575 (__init__:308)<br class="">29860:2018-03-21 10:46:29,700+0200 INFO (itmap/1) [IOProcessClient] Starting client ioprocess-576 (__init__:308)<br class="">29861:2018-03-21 10:46:29,714+0200 INFO (ioprocess/1711202) [IOProcess] Starting ioprocess (__init__:437)<br class="">29862:2018-03-21 10:46:29,725+0200 INFO (ioprocess/1711208) [IOProcess] Starting ioprocess (__init__:437)<br class="">29863:2018-03-21 10:46:30,375+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=bdacf1c0-3fcb-43b1-82e2-a041a26d0d7f (api:46)<br class="">29864:2018-03-21 10:46:30,376+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00031111', 'lastCheck': '0.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00045594', 'lastCheck': '0.6', 'valid': True}} from=internal, task_id=bdacf1c0-3fcb-43b1-82e2-a041a26d0d7f (api:52)<br class="">29865:2018-03-21 10:46:30,376+0200 INFO (periodic/0) [vdsm.api] START multipath_health() from=internal, task_id=3d3cad70-e5c8-49b2-9f5d-92a249cc102d (api:46)<br class="">29866:2018-03-21 10:46:30,377+0200 INFO (periodic/0) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=3d3cad70-e5c8-49b2-9f5d-92a249cc102d (api:52)<br class="">29867:2018-03-21 10:46:35,108+0200 INFO (jsonrpc/6) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=304d7e7c-03fc-44d6-be32-d083ada09b30 (api:46)<br class="">29868:2018-03-21 10:46:35,114+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=304d7e7c-03fc-44d6-be32-d083ada09b30 (api:52)<br class="">29869:2018-03-21 10:46:35,114+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:573)<br class="">29870:2018-03-21 10:46:35,170+0200 INFO (jsonrpc/4) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=76736a93-d4c4-4b2f-9848-7b8731fe2b67 (api:46)<br class="">29871:2018-03-21 10:46:35,176+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=76736a93-d4c4-4b2f-9848-7b8731fe2b67 (api:52)<br class="">29872:2018-03-21 10:46:35,177+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)<br class="">29873:2018-03-21 10:46:35,462+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br class="">29874:2018-03-21 10:46:35,462+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br class="">29875:2018-03-21 10:46:35,463+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br class="">29876:2018-03-21 10:46:36,948+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br class="">29877:2018-03-21 10:46:36,948+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br class="">29878:2018-03-21 10:46:36,949+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br class="">29879:2018-03-21 10:46:43,179+0200 INFO (jsonrpc/1) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)<br class="">29880:2018-03-21 10:46:43,180+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=83bbc95c-6d59-4eec-82b1-e8993be28759 (api:46)<br class="">29881:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000486101', 'lastCheck': '3.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': '3.4', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=83bbc95c-6d59-4eec-82b1-e8993be28759 (api:52)<br class="">29882:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=9b781324-b50b-49a9-bf31-5eca835dc75c (api:46)<br class="">29883:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=9b781324-b50b-49a9-bf31-5eca835dc75c (api:52)<br class="">29884:2018-03-21 10:46:43,188+0200 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.40', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.33'}, '10': {'cpuUser': '0.60', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '98.93'}, '13': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.73'}, '12': {'cpuUser': '1.46', 'nodeIndex': 0, 'cpuSys': '0.60', 'cpuIdle': '97.94'}, '15': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '99.47'}, '14': {'cpuUser': '0.40', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '99.13'}, '1': {'cpuUser': '1.33', 'nodeIndex': 1, 'cpuSys': '1.93', 'cpuIdle': '96.74'}, '0': {'cpuUser': '0.60', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.13'}, '3': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.40'}, '2': {'cpuUser': '6.99', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '92.61'}, '5': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '4': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.46'}, '7': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.60', 'cpuIdle': '98.93'}, '6': {'cpuUser': '0.53', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.20'}, '9': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '99.20'}, '8': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15131'}, '0': {'memPercent': 5, 'memFree': '15357'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000486101', 'lastCheck': '3.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': '3.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'enp3s0f0', 'tx': '1091404888', 'txDropped': '0', 'rx': '11914970667', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1455'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'ovirtmgmt', 'tx': '1048752406', 'txDropped': '0', 'rx': '11615298347', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'lo', 'tx': '58215969455', 'txDropped': '0', 'rx': '58215969455', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621990.363726, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621990.363726, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621990.363726, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '206', 'ksmPages': 100, 'elapsedTime': '76562.02', 'cpuLoad': '0.18', 'cpuSys': '0.45', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.27', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1455', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:46:43 GMT', 'cpuUser': '0.87', 'memFree': 31336, 'cpuIdle': '98.68', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '1.20'}} from=::ffff:192.168.0.4,49914 (api:52)<br class="">29885:2018-03-21 10:46:43,190+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)<br class="">29886:2018-03-21 10:46:45,288+0200 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=ffe57349-2532-4b64-b32c-8a558273d8e5 (api:46)<br class="">29887:2018-03-21 10:46:45,292+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=ffe57349-2532-4b64-b32c-8a558273d8e5 (api:52)<br class="">29888:2018-03-21 10:46:45,293+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:573)<br class="">29889:2018-03-21 10:46:45,298+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=b2ea28f0-abf0-4258-8152-6035d93aa5dc (api:46)<br class="">29890:2018-03-21 10:46:45,305+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=b2ea28f0-abf0-4258-8152-6035d93aa5dc (api:52)<br class="">29891:2018-03-21 10:46:45,306+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:573)<br class="">29892:2018-03-21 10:46:45,395+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=4010bf72-5c86-4daf-a8f9-12ec229a86e4 (api:46)<br class="">29893:2018-03-21 10:46:45,395+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000486101', 'lastCheck': '5.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': '5.7', 'valid': True}} from=internal, task_id=4010bf72-5c86-4daf-a8f9-12ec229a86e4 (api:52)<br class="">29894:2018-03-21 10:46:45,396+0200 INFO (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=37a1132f-452b-4805-9cc0-b8e74e8d6b02 (api:46)<br class="">29895:2018-03-21 10:46:45,396+0200 INFO (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=37a1132f-452b-4805-9cc0-b8e74e8d6b02 (api:52)<br class="">29896:2018-03-21 10:46:50,467+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br class="">29897:2018-03-21 10:46:50,468+0200 INFO (jsonrpc/0) [throttled] Current getAllVmStats: {} (throttledlog:103)<br class="">29898:2018-03-21 10:46:50,468+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br class="">29899:2018-03-21 10:46:50,468+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573)<br class="">29900:2018-03-21 10:46:51,967+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br class="">29901:2018-03-21 10:46:51,968+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br class="">29902:2018-03-21 10:46:51,968+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br class="">29903:2018-03-21 10:46:55,648+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=a975635f-59d7-4f40-b9a7-a5ea7150368e (api:46)<br class="">29904:2018-03-21 10:46:55,653+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=a975635f-59d7-4f40-b9a7-a5ea7150368e (api:52)<br class="">29905:2018-03-21 10:46:55,653+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:573)<br class="">29906:2018-03-21 10:46:55,659+0200 INFO (jsonrpc/2) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=97d4ae6c-3719-4149-85fd-28e924ad8883 (api:46)<br class="">29907:2018-03-21 10:46:55,666+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=97d4ae6c-3719-4149-85fd-28e924ad8883 (api:52)<br class="">29908:2018-03-21 10:46:55,667+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:573)<br class="">29909:2018-03-21 10:46:59,473+0200 INFO (jsonrpc/7) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)<br class="">29910:2018-03-21 10:46:59,474+0200 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=177b551b-9bd3-45a4-84d0-5244630c08e5 (api:46)<br class="">29911:2018-03-21 10:46:59,474+0200 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470906', 'lastCheck': '9.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000334928', 'lastCheck': '9.7', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=177b551b-9bd3-45a4-84d0-5244630c08e5 (api:52)<br class="">29912:2018-03-21 10:46:59,475+0200 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=30753d5c-9bfd-4b0f-ac83-e32278d82b63 (api:46)<br class="">29913:2018-03-21 10:46:59,475+0200 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=30753d5c-9bfd-4b0f-ac83-e32278d82b63 (api:52)<br class="">29914:2018-03-21 10:46:59,482+0200 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '10': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '15': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '1': {'cpuUser': '1.07', 'nodeIndex': 1, 'cpuSys': '1.07', 'cpuIdle': '97.86'}, '0': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '4': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.26'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.47', 'cpuIdle': '99.53'}, '6': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.73'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '8': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15125'}, '0': {'memPercent': 5, 'memFree': '15364'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470906', 'lastCheck': '9.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000334928', 'lastCheck': '9.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'enp3s0f0', 'tx': '1091412332', 'txDropped': '0', 'rx': '11914976557', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1456'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'ovirtmgmt', 'tx': '1048761326', 'txDropped': '0', 'rx': '11615304242', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'lo', 'tx': '58228460774', 'txDropped': '0', 'rx': '58228460774', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521622005.383289, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521622005.383289, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521622005.383289, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '208', 'ksmPages': 100, 'elapsedTime': '76578.31', 'cpuLoad': '0.17', 'cpuSys': '0.20', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.13', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31078, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1456', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:46:59 GMT', 'cpuUser': '0.15', 'memFree': 31334, 'cpuIdle': '99.65', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.60'}} from=::ffff:192.168.0.4,49914 (api:52)<br class="">29915:2018-03-21 10:46:59,484+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 21 Mar 2018, at 11:33, Fred Rolland <<a href="mailto:frolland@redhat.com" class="">frolland@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="">Can you provide the vdsm logs from the host.<br class=""></div>It looks the vdsm failed to connect to the server.<br class=""></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Wed, Mar 21, 2018 at 11:21 AM, Andrei Verovski <span dir="ltr" class=""><<a href="mailto:andreil1@starlett.lv" target="_blank" class="">andreil1@starlett.lv</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space" class="">Hi,<div class=""><br class=""></div><div class="">I have 2-host oVirt setup with 2 Data Centers, one with local storage domain (DC #1) for VMs + Export domain on NFS, another with all NFS shared (DC #2).</div><div class="">Trying to export VMs from DC #1 to DC #2.</div><div class="">VMs are exported to DC #1 export domain (NFS), then domain put into maintenance mode and detached from DC #1.</div><div class=""><br class=""></div><div class="">Unfortunately, attaching it to DC #2 failed. Logs attached. Tried to run this command twice.</div><div class="">Workaround are possible in order to accomplish this task, yet it would be better to do in a way as it was designed.<br class="">
Thanks.<br class=""><br class=""><br class=""><div class="">2018-03-21 10:43:24,024+02 ERROR [org.ovirt.engine.core.dal.<wbr class="">dbbroker.auditloghandling.<wbr class="">AuditLogDirector] (EE-ManagedThreadFactory-<wbr class="">engine-Thread-128904) [<font color="#942192" class="">570ec5d9-fff5-4656-afbd-<wbr class="">90b3207a616e</font>] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_<wbr class="">POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz)<br class="">2018-03-21 10:46:16,512+02 ERROR [org.ovirt.engine.core.dal.<wbr class="">dbbroker.auditloghandling.<wbr class="">AuditLogDirector] (EE-ManagedThreadFactory-<wbr class="">engine-Thread-128956) [1435fc81] EVENT_ID: VDS_STORAGES_CONNECTION_<wbr class="">FAILED(188), Failed to connect Host node11 to the Storage Domains node10-NFS-EXPORTS.<br class="">2018-03-21 10:46:16,651+02 ERROR [org.ovirt.engine.core.dal.<wbr class="">dbbroker.auditloghandling.<wbr class="">AuditLogDirector] (EE-ManagedThreadFactory-<wbr class="">engine-Thread-128955) [<font color="#942192" class="">a81ffa4a-5a58-41a0-888a-<wbr class="">f0edc321609b</font>] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_<wbr class="">POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz)<br class=""><br class=""><br class=""><font color="#aa7942" class=""><b class="">tail -n 1000 engine.log | grep 570ec5d9-fff5-4656-afbd-<wbr class="">90b3207a616e</b></font><br class="">2018-03-21 10:41:14,643+02 INFO [org.ovirt.engine.core.bll.<wbr class="">storage.domain.<wbr class="">AttachStorageDomainToPoolComma<wbr class="">nd] (default task-2) [570ec5d9-fff5-4656-afbd-<wbr class="">90b3207a616e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[<wbr class="">1d7208ce-d3a1-4406-9638-<wbr class="">fe7051562994=STORAGE]', sharedLocks=''}'<br class="">2018-03-21 10:41:16,129+02 INFO [org.ovirt.engine.core.bll.<wbr class="">storage.domain.<wbr class="">AttachStorageDomainToPoolComma<wbr class="">nd] (EE-ManagedThreadFactory-<wbr class="">engine-Thread-128904) [570ec5d9-fff5-4656-afbd-<wbr class="">90b3207a616e] Running command: AttachStorageDomainToPoolComma<wbr class="">nd internal: false. Entities affected : ID: 1d7208ce-d3a1-4406-9638-<wbr class="">fe7051562994 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: 80cc922f-8dea-4fed-b951-<wbr class="">1060ba116ad5 Type: StoragePoolAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN<br class="">2018-03-21 10:43:23,564+02 ERROR [org.ovirt.engine.core.bll.<wbr class="">storage.domain.<wbr class="">AttachStorageDomainToPoolComma<wbr class="">nd] (EE-ManagedThreadFactory-<wbr class="">engine-Thread-128904) [570ec5d9-fff5-4656-afbd-<wbr class="">90b3207a616e] Cannot connect storage connection server, aborting attach storage domain operation.<br class="">2018-03-21 10:43:23,567+02 INFO [org.ovirt.engine.core.bll.<wbr class="">CommandCompensator] (EE-ManagedThreadFactory-<wbr class="">engine-Thread-128904) [570ec5d9-fff5-4656-afbd-<wbr class="">90b3207a616e] Command [id=921ca7cd-4f93-46aa-8de2-<wbr class="">91b13b8f96cb]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.<wbr class="">businessentities.<wbr class="">StoragePoolIsoMap; snapshot: StoragePoolIsoMapId:{<wbr class="">storagePoolId='80cc922f-8dea-<wbr class="">4fed-b951-1060ba116ad5', storageId='1d7208ce-d3a1-4406-<wbr class="">9638-fe7051562994'}.<br class="">2018-03-21 10:43:24,024+02 ERROR [org.ovirt.engine.core.dal.<wbr class="">dbbroker.auditloghandling.<wbr class="">AuditLogDirector] (EE-ManagedThreadFactory-<wbr class="">engine-Thread-128904) [570ec5d9-fff5-4656-afbd-<wbr class="">90b3207a616e] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_<wbr class="">POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz)<br class="">2018-03-21 10:43:24,114+02 INFO [org.ovirt.engine.core.bll.<wbr class="">storage.domain.<wbr class="">AttachStorageDomainToPoolComma<wbr class="">nd] (EE-ManagedThreadFactory-<wbr class="">engine-Thread-128904) [570ec5d9-fff5-4656-afbd-<wbr class="">90b3207a616e] Lock freed to object 'EngineLock:{exclusiveLocks='[<wbr class="">1d7208ce-d3a1-4406-9638-<wbr class="">fe7051562994=STORAGE]', sharedLocks=''}'<br class=""><br class=""><br class=""><br class=""><font color="#aa7942" class=""><b class="">[root@node00 ovirt-engine]# tail -n 1000 engine.log | grep a81ffa4a-5a58-41a0-888a-<wbr class="">f0edc321609b</b></font><br class="">2018-03-21 10:44:11,025+02 INFO [org.ovirt.engine.core.bll.<wbr class="">storage.domain.<wbr class="">AttachStorageDomainToPoolComma<wbr class="">nd] (default task-16) [a81ffa4a-5a58-41a0-888a-<wbr class="">f0edc321609b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[<wbr class="">1d7208ce-d3a1-4406-9638-<wbr class="">fe7051562994=STORAGE]', sharedLocks=''}'<br class="">2018-03-21 10:44:11,236+02 INFO [org.ovirt.engine.core.bll.<wbr class="">storage.domain.<wbr class="">AttachStorageDomainToPoolComma<wbr class="">nd] (EE-ManagedThreadFactory-<wbr class="">engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-<wbr class="">f0edc321609b] Running command: AttachStorageDomainToPoolComma<wbr class="">nd internal: false. Entities affected : ID: 1d7208ce-d3a1-4406-9638-<wbr class="">fe7051562994 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: 80cc922f-8dea-4fed-b951-<wbr class="">1060ba116ad5 Type: StoragePoolAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN<br class="">2018-03-21 10:46:16,567+02 ERROR [org.ovirt.engine.core.bll.<wbr class="">storage.domain.<wbr class="">AttachStorageDomainToPoolComma<wbr class="">nd] (EE-ManagedThreadFactory-<wbr class="">engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-<wbr class="">f0edc321609b] Cannot connect storage connection server, aborting attach storage domain operation.<br class="">2018-03-21 10:46:16,568+02 INFO [org.ovirt.engine.core.bll.<wbr class="">CommandCompensator] (EE-ManagedThreadFactory-<wbr class="">engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-<wbr class="">f0edc321609b] Command [id=b5c25100-1a8a-4db0-9509-<wbr class="">99cfa60995b1]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.<wbr class="">businessentities.<wbr class="">StoragePoolIsoMap; snapshot: StoragePoolIsoMapId:{<wbr class="">storagePoolId='80cc922f-8dea-<wbr class="">4fed-b951-1060ba116ad5', storageId='1d7208ce-d3a1-4406-<wbr class="">9638-fe7051562994'}.<br class="">2018-03-21 10:46:16,651+02 ERROR [org.ovirt.engine.core.dal.<wbr class="">dbbroker.auditloghandling.<wbr class="">AuditLogDirector] (EE-ManagedThreadFactory-<wbr class="">engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-<wbr class="">f0edc321609b] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_<wbr class="">POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz)<br class="">2018-03-21 10:46:16,681+02 INFO [org.ovirt.engine.core.bll.<wbr class="">storage.domain.<wbr class="">AttachStorageDomainToPoolComma<wbr class="">nd] (EE-ManagedThreadFactory-<wbr class="">engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-<wbr class="">f0edc321609b] Lock freed to object 'EngineLock:{exclusiveLocks='[<wbr class="">1d7208ce-d3a1-4406-9638-<wbr class="">fe7051562994=STORAGE]', sharedLocks='’}'</div><div class=""><br class=""></div></div></div><br class="">______________________________<wbr class="">_________________<br class="">
Users mailing list<br class="">
<a href="mailto:Users@ovirt.org" class="">Users@ovirt.org</a><br class="">
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank" class="">http://lists.ovirt.org/<wbr class="">mailman/listinfo/users</a><br class="">
<br class=""></blockquote></div><br class=""></div>
</div></blockquote></div><br class=""></div></body></html>