<div dir="ltr"><div><div>node11 fails to mount the export domain from node 10.<br><br></div>You can try manually to see if you have access from node 11.<br></div>Look at this page to debug the NFS connection:<br><a href="https://www.ovirt.org/documentation/how-to/troubleshooting/troubleshooting-nfs-storage-issues/">https://www.ovirt.org/documentation/how-to/troubleshooting/troubleshooting-nfs-storage-issues/</a><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 21, 2018 at 12:04 PM, Andrei Verovski <span dir="ltr"><<a href="mailto:andreil1@starlett.lv" target="_blank">andreil1@starlett.lv</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space"><div>Hi,</div><div><br></div><div><br></div><div>Errors occurred at 10.43 AM and 10.46 AM</div><div><a href="http://node00.starlett.lv" target="_blank">node00.starlett.lv</a> - 192.168.0.4 - oVirt host engine (separate PC, not hosted)</div><div><a href="http://node10.starlett.lv" target="_blank">node10.starlett.lv</a> - 192.168.0.5 - host #1 of DC #1, export domain from which was detached from DC #1</div><div><a href="http://node11.starlett.lv" target="_blank">node11.starlett.lv</a> - 192.168.0.6 - host #1 of DC #2, </div><div><br></div>Logs from DC#2 node11, to which I’m trying to attach export domain located at NFS share node10.<div><br></div><div><blockquote type="cite"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space"><div><div><font color="#aa7942"><b>grep 570ec5d9-fff5-4656-afbd-90b320<wbr>7a616e</b></font></div></div></div></blockquote></div></div></blockquote>within vdsm.log returned nothing, so I did</div><div><br></div><div><br></div><div><font color="#942192"><b>grep -n 10:43 vdsm.log | tail -1000</b></font><br>1011:2018-03-21 06:10:43,077+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br>1012:2018-03-21 06:10:43,077+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br>1013:2018-03-21 06:10:43,077+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573)<br>1014:2018-03-21 06:10:43,868+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br>1015:2018-03-21 06:10:43,868+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br>1016:2018-03-21 06:10:43,868+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br>19729:2018-03-21 09:10:43,641+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br>19730:2018-03-21 09:10:43,641+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br>19731:2018-03-21 09:10:43,642+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br>29302:2018-03-21 10:43:00,085+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=7b690c8a-5470-44da-<wbr>a8f3-7e7e9b018e88 (api:46)<br>29303:2018-03-21 10:43:00,085+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-<wbr>b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000477038', 'lastCheck': '0.5', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': '0.4', 'valid': True}} from=internal, task_id=7b690c8a-5470-44da-<wbr>a8f3-7e7e9b018e88 (api:52)<br>29304:2018-03-21 10:43:00,086+0200 INFO (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=9a625d43-a03c-429c-<wbr>856f-9aa8e5ff65b5 (api:46)<br>29305:2018-03-21 10:43:00,086+0200 INFO (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=9a625d43-a03c-429c-<wbr>856f-9aa8e5ff65b5 (api:52)<br>29306:2018-03-21 10:43:05,382+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br>29307:2018-03-21 10:43:05,383+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br>29308:2018-03-21 10:43:05,383+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br>29309:2018-03-21 10:43:06,248+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)<br>29310:2018-03-21 10:43:06,249+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=f80f9bb8-2e84-4f45-<wbr>ab58-09a88e808cf3 (api:46)<br>29311:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-<wbr>b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000477038', 'lastCheck': '6.7', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': '6.6', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=f80f9bb8-2e84-4f45-<wbr>ab58-09a88e808cf3 (api:52)<br>29312:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=ddd7064d-cd12-43bf-<wbr>a177-b2a93cc9035e (api:46)<br>29313:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=ddd7064d-cd12-43bf-<wbr>a177-b2a93cc9035e (api:52)<br>29314:2018-03-21 10:43:06,256+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '10': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.47'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '12': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '15': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.47', 'cpuIdle': '99.33'}, '14': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '1': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': '1.60', 'cpuIdle': '97.27'}, '0': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.26'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '2': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.54'}, '5': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.74'}, '4': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '7': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.60', 'cpuIdle': '99.27'}, '6': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '8': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15142'}, '0': {'memPercent': 5, 'memFree': '15349'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000477038', 'lastCheck': '6.7', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': '6.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'enp3s0f0', 'tx': '1091287978', 'txDropped': '0', 'rx': '11914868563', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1439'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'ovirtmgmt', 'tx': '1048639112', 'txDropped': '0', 'rx': '11615209092', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'lo', 'tx': '58057075945', 'txDropped': '0', 'rx': '58057075945', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621780.072418, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621780.072418, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621780.072418, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76345.09', 'cpuLoad': '0.16', 'cpuSys': '0.36', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31081, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1439', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:43:06 GMT', 'cpuUser': '0.21', 'memFree': 31337, 'cpuIdle': '99.43', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.93'}} from=::ffff:192.168.0.4,49914 (api:52)<br>29315:2018-03-21 10:43:06,258+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)<br>29316:2018-03-21 10:43:06,632+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br>29317:2018-03-21 10:43:06,632+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br>29318:2018-03-21 10:43:06,633+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br>29319:2018-03-21 10:43:07,415+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=276c13c7-61ec-4782-<wbr>a0d9-5cd0f0b0a729 (api:46)<br>29320:2018-03-21 10:43:07,420+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=276c13c7-61ec-4782-<wbr>a0d9-5cd0f0b0a729 (api:52)<br>29321:2018-03-21 10:43:07,420+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)<br>29322:2018-03-21 10:43:07,427+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=344b18df-a428-429a-<wbr>b49f-381f901d1618 (api:46)<br>29323:2018-03-21 10:43:07,434+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8:Active,8a2c304b-<wbr>c8ae-438b-af54-fc8797ea149f:<wbr>Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=344b18df-a428-429a-<wbr>b49f-381f901d1618 (api:52)<br>29324:2018-03-21 10:43:07,435+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)<br>29325:2018-03-21 10:43:13,839+0200 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/1 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'id': u'461f65a9-3a81-4f3f-a46d-<wbr>c5ed12520524', u'connection': u'node10.starlett.lv:/vmdata/<wbr>nfs/exports', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'protocol_version': u'auto', u'password': '********', u'port': u''}], u'storagepoolID': u'00000000-0000-0000-0000-<wbr>000000000000', u'domainType': 1}, 'jsonrpc': '2.0', 'method': u'StoragePool.<wbr>connectStorageServer', 'id': u'5f98dfd5-998f-44a6-8594-<wbr>e5f40144b0ed'} at 0x4076150> timeout=60, duration=120 at 0x402fe50> task#=3754 at 0x353e690>, traceback:<br>29376:2018-03-21 10:43:15,104+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=672a883e-1d60-4981-<wbr>9eee-bdcac2ec30c1 (api:46)<br>29377:2018-03-21 10:43:15,105+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-<wbr>b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000488777', 'lastCheck': '5.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000481962', 'lastCheck': '5.4', 'valid': True}} from=internal, task_id=672a883e-1d60-4981-<wbr>9eee-bdcac2ec30c1 (api:52)<br>29378:2018-03-21 10:43:15,105+0200 INFO (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=55f1b797-da2a-4b07-<wbr>affe-5b9e41a74307 (api:46)<br>29379:2018-03-21 10:43:15,106+0200 INFO (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=55f1b797-da2a-4b07-<wbr>affe-5b9e41a74307 (api:52)<br>29380:2018-03-21 10:43:17,645+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=52a6ddeb-3c20-4f3f-<wbr>b27d-94d19580b473 (api:46)<br>29381:2018-03-21 10:43:17,649+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=52a6ddeb-3c20-4f3f-<wbr>b27d-94d19580b473 (api:52)<br>29382:2018-03-21 10:43:17,650+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)<br>29383:2018-03-21 10:43:17,656+0200 INFO (jsonrpc/2) [vdsm.api] START getStoragePoolInfo(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=1a103883-6a6a-4d61-<wbr>ade6-96559591ddb0 (api:46)<br>29384:2018-03-21 10:43:17,663+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8:Active,8a2c304b-<wbr>c8ae-438b-af54-fc8797ea149f:<wbr>Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=1a103883-6a6a-4d61-<wbr>ade6-96559591ddb0 (api:52)<br>29385:2018-03-21 10:43:17,663+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)<br>29386:2018-03-21 10:43:18,882+0200 ERROR (jsonrpc/1) [storage.HSM] Could not connect to storageServer (hsm:2407)<br>29406:2018-03-21 10:43:18,883+0200 INFO (jsonrpc/1) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 477, 'id': u'461f65a9-3a81-4f3f-a46d-<wbr>c5ed12520524'}]} from=::ffff:192.168.0.4,49914, flow_id=4a53b512, task_id=e598cbe0-cde8-4c73-<wbr>b526-4398df05e67f (api:52)<br>29407:2018-03-21 10:43:18,883+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.<wbr>connectStorageServer succeeded in 125.04 seconds (__init__:573)<br>29408:2018-03-21 10:43:20,387+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br>29409:2018-03-21 10:43:20,388+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br>29410:2018-03-21 10:43:20,388+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573)<br>29411:2018-03-21 10:43:21,497+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)<br>29412:2018-03-21 10:43:21,498+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=a7469c1f-3d6d-42e7-<wbr>8798-5c5335da4a9b (api:46)<br>29413:2018-03-21 10:43:21,498+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-<wbr>b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000445096', 'lastCheck': '1.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000296586', 'lastCheck': '1.8', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=a7469c1f-3d6d-42e7-<wbr>8798-5c5335da4a9b (api:52)<br>29414:2018-03-21 10:43:21,499+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=9fa5dae3-e2bd-4186-<wbr>9a1d-afb0aa6e2511 (api:46)<br>29415:2018-03-21 10:43:21,499+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=9fa5dae3-e2bd-4186-<wbr>9a1d-afb0aa6e2511 (api:52)<br>29416:2018-03-21 10:43:21,505+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '10': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '12': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '15': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.60'}, '1': {'cpuUser': '1.00', 'nodeIndex': 1, 'cpuSys': '1.00', 'cpuIdle': '98.00'}, '0': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '3': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '2': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.66'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '7': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '6': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '8': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15142'}, '0': {'memPercent': 5, 'memFree': '15351'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000445096', 'lastCheck': '1.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000296586', 'lastCheck': '1.8', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'enp3s0f0', 'tx': '1091295628', 'txDropped': '0', 'rx': '11914875210', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1442'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'ovirtmgmt', 'tx': '1048646480', 'txDropped': '0', 'rx': '11615214750', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'lo', 'tx': '58066397668', 'txDropped': '0', 'rx': '58066397668', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621795.091263, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621795.091263, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621795.091263, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76360.34', 'cpuLoad': '0.15', 'cpuSys': '0.18', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31031, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1442', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:43:21 GMT', 'cpuUser': '0.15', 'memFree': 31287, 'cpuIdle': '99.67', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.67'}} from=::ffff:192.168.0.4,49914 (api:52)<br>29417:2018-03-21 10:43:21,508+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.02 seconds (__init__:573)<br>29418:2018-03-21 10:43:21,651+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br>29419:2018-03-21 10:43:21,651+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br>29420:2018-03-21 10:43:21,652+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br>29421:2018-03-21 10:43:27,764+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=46d19cab-8903-4c52-<wbr>9bb2-4dd8f370997e (api:46)<br>29422:2018-03-21 10:43:27,769+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=46d19cab-8903-4c52-<wbr>9bb2-4dd8f370997e (api:52)<br>29423:2018-03-21 10:43:27,770+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)<br>29424:2018-03-21 10:43:27,815+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=290acc73-8f25-4fd7-<wbr>9263-7bd05d46430a (api:46)<br>29425:2018-03-21 10:43:27,822+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8:Active,8a2c304b-<wbr>c8ae-438b-af54-fc8797ea149f:<wbr>Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=290acc73-8f25-4fd7-<wbr>9263-7bd05d46430a (api:52)<br>29426:2018-03-21 10:43:27,823+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)<br>29427:2018-03-21 10:43:30,121+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=0abdebcf-2b0b-4d33-<wbr>b552-c0f2ed3e567b (api:46)<br>29428:2018-03-21 10:43:30,122+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-<wbr>b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000402077', 'lastCheck': '0.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': '0.5', 'valid': True}} from=internal, task_id=0abdebcf-2b0b-4d33-<wbr>b552-c0f2ed3e567b (api:52)<br>29429:2018-03-21 10:43:30,122+0200 INFO (periodic/3) [vdsm.api] START multipath_health() from=internal, task_id=acde7e7d-ab4a-4811-<wbr>934f-4d312da33208 (api:46)<br>29430:2018-03-21 10:43:30,123+0200 INFO (periodic/3) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=acde7e7d-ab4a-4811-<wbr>934f-4d312da33208 (api:52)<br>29431:2018-03-21 10:43:35,393+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br>29432:2018-03-21 10:43:35,394+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br>29433:2018-03-21 10:43:35,394+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br>29434:2018-03-21 10:43:36,671+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br>29435:2018-03-21 10:43:36,671+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br>29436:2018-03-21 10:43:36,672+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br>29437:2018-03-21 10:43:37,356+0200 INFO (jsonrpc/1) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)<br>29438:2018-03-21 10:43:37,357+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=bc14e253-d3d4-4cc3-<wbr>9ae1-c0e6ac6e37f7 (api:46)<br>29439:2018-03-21 10:43:37,357+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-<wbr>b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000402077', 'lastCheck': '7.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': '7.7', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=bc14e253-d3d4-4cc3-<wbr>9ae1-c0e6ac6e37f7 (api:52)<br>29440:2018-03-21 10:43:37,358+0200 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=23c698b8-d5bd-4746-<wbr>b738-97b10f3e4bfb (api:46)<br>29441:2018-03-21 10:43:37,358+0200 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=23c698b8-d5bd-4746-<wbr>b738-97b10f3e4bfb (api:52)<br>29442:2018-03-21 10:43:37,365+0200 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '10': {'cpuUser': '3.06', 'nodeIndex': 0, 'cpuSys': '0.80', 'cpuIdle': '96.14'}, '13': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '12': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '15': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '14': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.33'}, '1': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': '1.33', 'cpuIdle': '97.54'}, '0': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.20'}, '3': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '2': {'cpuUser': '0.67', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '98.93'}, '5': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.74'}, '4': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '7': {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': '0.73', 'cpuIdle': '98.94'}, '6': {'cpuUser': '5.39', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '94.14'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '8': {'cpuUser': '0.53', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.07'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15143'}, '0': {'memPercent': 5, 'memFree': '15348'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000402077', 'lastCheck': '7.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': '7.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'enp3s0f0', 'tx': '1091305295', 'txDropped': '0', 'rx': '11914880953', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1442'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'ovirtmgmt', 'tx': '1048655897', 'txDropped': '0', 'rx': '11615220341', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'lo', 'tx': '58078891247', 'txDropped': '0', 'rx': '58078891247', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621810.108511, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621810.108511, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621810.108511, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76376.19', 'cpuLoad': '0.16', 'cpuSys': '0.39', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.00', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1442', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:43:37 GMT', 'cpuUser': '0.79', 'memFree': 31336, 'cpuIdle': '98.82', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.73'}} from=::ffff:192.168.0.4,49914 (api:52)<br>29443:2018-03-21 10:43:37,367+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)<br>29444:2018-03-21 10:43:38,124+0200 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=9d2a82dd-c76c-4eb5-<wbr>a1e1-b49bf004384e (api:46)<br>29445:2018-03-21 10:43:38,128+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=9d2a82dd-c76c-4eb5-<wbr>a1e1-b49bf004384e (api:52)<br>29446:2018-03-21 10:43:38,129+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)<br>29447:2018-03-21 10:43:38,171+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=62d44b0b-20b2-47ae-<wbr>bfb8-d4717e301b69 (api:46)<br>29448:2018-03-21 10:43:38,177+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8:Active,8a2c304b-<wbr>c8ae-438b-af54-fc8797ea149f:<wbr>Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=62d44b0b-20b2-47ae-<wbr>bfb8-d4717e301b69 (api:52)<br>29449:2018-03-21 10:43:38,178+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)<br>29450:2018-03-21 10:43:45,147+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=49c33285-a222-464d-<wbr>8984-4e75a5c6354a (api:46)<br>29451:2018-03-21 10:43:45,148+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-<wbr>b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000454102', 'lastCheck': '5.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000347381', 'lastCheck': '5.5', 'valid': True}} from=internal, task_id=49c33285-a222-464d-<wbr>8984-4e75a5c6354a (api:52)<br>29452:2018-03-21 10:43:45,148+0200 INFO (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=eed6b564-76b1-4f9b-<wbr>80a2-1f0f6d9bbe32 (api:46)<br>29453:2018-03-21 10:43:45,149+0200 INFO (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=eed6b564-76b1-4f9b-<wbr>80a2-1f0f6d9bbe32 (api:52)<br>29454:2018-03-21 10:43:48,405+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=84651a37-7d13-41a9-<wbr>9dcf-2169a9c46fc5 (api:46)<br>29455:2018-03-21 10:43:48,410+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=84651a37-7d13-41a9-<wbr>9dcf-2169a9c46fc5 (api:52)<br>29456:2018-03-21 10:43:48,411+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)<br>29457:2018-03-21 10:43:48,417+0200 INFO (jsonrpc/4) [vdsm.api] START getStoragePoolInfo(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=4ebd1d1a-11ac-41ee-<wbr>b459-7e45f008776b (api:46)<br>29458:2018-03-21 10:43:48,423+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8:Active,8a2c304b-<wbr>c8ae-438b-af54-fc8797ea149f:<wbr>Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=4ebd1d1a-11ac-41ee-<wbr>b459-7e45f008776b (api:52)<br>29459:2018-03-21 10:43:48,424+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)<br>29460:2018-03-21 10:43:50,399+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br>29461:2018-03-21 10:43:50,400+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br>29462:2018-03-21 10:43:50,400+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br>29463:2018-03-21 10:43:51,694+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br>29464:2018-03-21 10:43:51,694+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br>29465:2018-03-21 10:43:51,695+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br>29466:2018-03-21 10:43:52,516+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)<br>29467:2018-03-21 10:43:52,517+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=0a88f0dc-eccf-4d24-<wbr>9ac2-22e079a1480a (api:46)<br>29468:2018-03-21 10:43:52,517+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-<wbr>b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000487937', 'lastCheck': '3.0', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000455956', 'lastCheck': '2.9', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=0a88f0dc-eccf-4d24-<wbr>9ac2-22e079a1480a (api:52)<br>29469:2018-03-21 10:43:52,518+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=a7a44ff7-a137-41b6-<wbr>9ad5-2c0b32ab79ec (api:46)<br>29470:2018-03-21 10:43:52,518+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=a7a44ff7-a137-41b6-<wbr>9ad5-2c0b32ab79ec (api:52)<br>29471:2018-03-21 10:43:52,524+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '10': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '13': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '15': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '1': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': '1.13', 'cpuIdle': '97.74'}, '0': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '5': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '4': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.60'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '99.67'}, '6': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '8': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15143'}, '0': {'memPercent': 5, 'memFree': '15349'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000487937', 'lastCheck': '3.0', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000455956', 'lastCheck': '2.9', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'enp3s0f0', 'tx': '1091312739', 'txDropped': '0', 'rx': '11914899051', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1443'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'ovirtmgmt', 'tx': '1048663107', 'txDropped': '0', 'rx': '11615236520', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'lo', 'tx': '58090295494', 'txDropped': '0', 'rx': '58090295494', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621825.13588, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621825.13588, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621825.13588, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76391.35', 'cpuLoad': '0.15', 'cpuSys': '0.20', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.13', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1443', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:43:52 GMT', 'cpuUser': '0.14', 'memFree': 31336, 'cpuIdle': '99.66', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.73'}} from=::ffff:192.168.0.4,49914 (api:52)<br>29472:2018-03-21 10:43:52,526+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)<br>29473:2018-03-21 10:43:58,686+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=d6d29cf6-c083-4639-<wbr>91a2-f062eae4e629 (api:46)<br>29474:2018-03-21 10:43:58,690+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=d6d29cf6-c083-4639-<wbr>91a2-f062eae4e629 (api:52)<br>29475:2018-03-21 10:43:58,690+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)<br>29476:2018-03-21 10:43:58,697+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=9dbd91df-dd9c-469c-<wbr>8308-bfbfc228937c (api:46)<br>29477:2018-03-21 10:43:58,703+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8:Active,8a2c304b-<wbr>c8ae-438b-af54-fc8797ea149f:<wbr>Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=9dbd91df-dd9c-469c-<wbr>8308-bfbfc228937c (api:52)<br>29478:2018-03-21 10:43:58,704+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)<br><br><br><br><b><font color="#942192">grep -n 10:46 vdsm.log | tail -1000</font></b><br>1017:2018-03-21 06:10:46,882+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)<br>1018:2018-03-21 06:10:46,883+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=6e045b00-88b2-469b-<wbr>80d6-af38303f8c32 (api:46)<br>1019:2018-03-21 06:10:46,883+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-<wbr>b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000332785', 'lastCheck': '7.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000203914', 'lastCheck': '5.6', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=6e045b00-88b2-469b-<wbr>80d6-af38303f8c32 (api:52)<br>1020:2018-03-21 06:10:46,884+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=475a5b40-4a43-42ee-<wbr>9524-2bdd220aed83 (api:46)<br>1021:2018-03-21 06:10:46,884+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=475a5b40-4a43-42ee-<wbr>9524-2bdd220aed83 (api:52)<br>1022:2018-03-21 06:10:46,889+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '10': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '12': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '15': {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.40'}, '14': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '1': {'cpuUser': '3.00', 'nodeIndex': 1, 'cpuSys': '1.40', 'cpuIdle': '95.60'}, '0': {'cpuUser': '0.73', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.07'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '2': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '4': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '7': {'cpuUser': '0.80', 'nodeIndex': 1, 'cpuSys': '0.80', 'cpuIdle': '98.40'}, '6': {'cpuUser': '5.46', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '94.27'}, '9': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '8': {'cpuUser': '1.27', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '98.26'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15128'}, '0': {'memPercent': 5, 'memFree': '15366'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000332785', 'lastCheck': '7.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000203914', 'lastCheck': '5.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'enp3s0f0', 'tx': '1082264809', 'txDropped': '0', 'rx': '11907188594', 'rxErrors': '0', 'speed': '100', 'rxDropped': '289'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'ovirtmgmt', 'tx': '1039869554', 'txDropped': '0', 'rx': '11608697383', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'lo', 'tx': '45587320171', 'txDropped': '0', 'rx': '45587320171', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '53620', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521605437.203776, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521605437.203776, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521605437.203776, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '60005.72', 'cpuLoad': '0.06', 'cpuSys': '0.38', 'diskStats': {'/var/log': {'free': '7357'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31084, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '289', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T04:10:46 GMT', 'cpuUser': '0.83', 'memFree': 31340, 'cpuIdle': '98.79', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.80'}} from=::ffff:192.168.0.4,49914 (api:52)<br>1023:2018-03-21 06:10:46,891+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)<br>13522:2018-03-21 08:10:46,827+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br>13523:2018-03-21 08:10:46,828+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br>13524:2018-03-21 08:10:46,828+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br>29734:2018-03-21 10:46:00,338+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=af099ce6-0f44-461b-<wbr>9e8e-99de63b3884f (api:46)<br>29735:2018-03-21 10:46:00,339+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-<wbr>b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470245', 'lastCheck': '0.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000476935', 'lastCheck': '0.7', 'valid': True}} from=internal, task_id=af099ce6-0f44-461b-<wbr>9e8e-99de63b3884f (api:52)<br>29736:2018-03-21 10:46:00,339+0200 INFO (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=4a5709d9-e322-44eb-<wbr>a8a4-bbe3f5e5e1bd (api:46)<br>29737:2018-03-21 10:46:00,340+0200 INFO (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=4a5709d9-e322-44eb-<wbr>a8a4-bbe3f5e5e1bd (api:52)<br>29738:2018-03-21 10:46:04,637+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=5b9300f7-f8c4-4f63-<wbr>a49c-45ca8119d3b9 (api:46)<br>29739:2018-03-21 10:46:04,642+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=5b9300f7-f8c4-4f63-<wbr>a49c-45ca8119d3b9 (api:52)<br>29740:2018-03-21 10:46:04,642+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)<br>29741:2018-03-21 10:46:04,648+0200 INFO (jsonrpc/1) [vdsm.api] START getStoragePoolInfo(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=f4db5bfa-e131-4f9a-<wbr>baaa-e7555c996ab0 (api:46)<br>29742:2018-03-21 10:46:04,655+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8:Active,8a2c304b-<wbr>c8ae-438b-af54-fc8797ea149f:<wbr>Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=f4db5bfa-e131-4f9a-<wbr>baaa-e7555c996ab0 (api:52)<br>29743:2018-03-21 10:46:04,656+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:573)<br>29744:2018-03-21 10:46:05,451+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br>29745:2018-03-21 10:46:05,452+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br>29746:2018-03-21 10:46:05,452+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br>29747:2018-03-21 10:46:06,903+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br>29748:2018-03-21 10:46:06,904+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br>29749:2018-03-21 10:46:06,904+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br>29750:2018-03-21 10:46:08,791+0200 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/6 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'id': u'461f65a9-3a81-4f3f-a46d-<wbr>c5ed12520524', u'connection': u'node10.starlett.lv:/vmdata/<wbr>nfs/exports', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'protocol_version': u'auto', u'password': '********', u'port': u''}], u'storagepoolID': u'00000000-0000-0000-0000-<wbr>000000000000', u'domainType': 1}, 'jsonrpc': '2.0', 'method': u'StoragePool.<wbr>connectStorageServer', 'id': u'71143eef-aac6-4996-9783-<wbr>3e0e3da180c3'} at 0x3604fd0> timeout=60, duration=120 at 0x3604e10> task#=3762 at 0x3541650>, traceback:<br>29801:2018-03-21 10:46:12,852+0200 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)<br>29802:2018-03-21 10:46:12,853+0200 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=391fdab9-d3c2-4075-<wbr>9c34-177a07a21ec3 (api:46)<br>29803:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-<wbr>b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000492277', 'lastCheck': '3.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': '3.2', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=391fdab9-d3c2-4075-<wbr>9c34-177a07a21ec3 (api:52)<br>29804:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=7e53adbc-3974-4f51-<wbr>bfc8-8fbd3ca6b749 (api:46)<br>29805:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=7e53adbc-3974-4f51-<wbr>bfc8-8fbd3ca6b749 (api:52)<br>29806:2018-03-21 10:46:12,861+0200 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '10': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '12': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '15': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '14': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '1': {'cpuUser': '1.00', 'nodeIndex': 1, 'cpuSys': '1.33', 'cpuIdle': '97.67'}, '0': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '2': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.47'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '7': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.53'}, '6': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '8': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15142'}, '0': {'memPercent': 5, 'memFree': '15348'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000492277', 'lastCheck': '3.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': '3.2', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'enp3s0f0', 'tx': '1091387790', 'txDropped': '0', 'rx': '11914959714', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1451'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'ovirtmgmt', 'tx': '1048735814', 'txDropped': '0', 'rx': '11615289173', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'lo', 'tx': '58192068637', 'txDropped': '0', 'rx': '58192068637', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621960.326184, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621960.326184, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621960.326184, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76531.69', 'cpuLoad': '0.17', 'cpuSys': '0.32', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.00', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31081, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1451', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:46:12 GMT', 'cpuUser': '0.21', 'memFree': 31337, 'cpuIdle': '99.46', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.80'}} from=::ffff:192.168.0.4,49914 (api:52)<br>29807:2018-03-21 10:46:12,863+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)<br>29808:2018-03-21 10:46:13,829+0200 ERROR (jsonrpc/6) [storage.HSM] Could not connect to storageServer (hsm:2407)<br>29828:2018-03-21 10:46:13,829+0200 INFO (jsonrpc/6) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 477, 'id': u'461f65a9-3a81-4f3f-a46d-<wbr>c5ed12520524'}]} from=::ffff:192.168.0.4,49914, flow_id=1435fc81, task_id=0a828d2c-d9f4-4f83-<wbr>a9e9-7393159d5323 (api:52)<br>29829:2018-03-21 10:46:13,830+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.<wbr>connectStorageServer succeeded in 125.04 seconds (__init__:573)<br>29830:2018-03-21 10:46:14,767+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=49cd2ae6-7943-47c9-<wbr>ad86-9e0b7e58bca3 (api:46)<br>29831:2018-03-21 10:46:14,772+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=49cd2ae6-7943-47c9-<wbr>ad86-9e0b7e58bca3 (api:52)<br>29832:2018-03-21 10:46:14,773+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)<br>29833:2018-03-21 10:46:14,815+0200 INFO (jsonrpc/2) [vdsm.api] START getStoragePoolInfo(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=19909616-efb4-490d-<wbr>982c-66eda9ca4381 (api:46)<br>29834:2018-03-21 10:46:14,822+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8:Active,8a2c304b-<wbr>c8ae-438b-af54-fc8797ea149f:<wbr>Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=19909616-efb4-490d-<wbr>982c-66eda9ca4381 (api:52)<br>29835:2018-03-21 10:46:14,823+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)<br>29836:2018-03-21 10:46:15,360+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=da74ea9f-164b-450d-<wbr>b836-6818caa3fdc5 (api:46)<br>29837:2018-03-21 10:46:15,360+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-<wbr>b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000492277', 'lastCheck': '5.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': '5.7', 'valid': True}} from=internal, task_id=da74ea9f-164b-450d-<wbr>b836-6818caa3fdc5 (api:52)<br>29838:2018-03-21 10:46:15,361+0200 INFO (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=06245e2b-e8ca-41c4-<wbr>90bf-4294d0c699b8 (api:46)<br>29839:2018-03-21 10:46:15,361+0200 INFO (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=06245e2b-e8ca-41c4-<wbr>90bf-4294d0c699b8 (api:52)<br>29840:2018-03-21 10:46:20,456+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br>29841:2018-03-21 10:46:20,457+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br>29842:2018-03-21 10:46:20,457+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573)<br>29843:2018-03-21 10:46:21,925+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br>29844:2018-03-21 10:46:21,926+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br>29845:2018-03-21 10:46:21,926+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br>29846:2018-03-21 10:46:24,930+0200 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=d964535d-e5eb-42bc-<wbr>b324-432a05c364da (api:46)<br>29847:2018-03-21 10:46:24,935+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=d964535d-e5eb-42bc-<wbr>b324-432a05c364da (api:52)<br>29848:2018-03-21 10:46:24,935+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:573)<br>29849:2018-03-21 10:46:24,987+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=167dcd23-04b5-4de3-<wbr>9c81-7d25ba56407a (api:46)<br>29850:2018-03-21 10:46:25,022+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8:Active,8a2c304b-<wbr>c8ae-438b-af54-fc8797ea149f:<wbr>Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=167dcd23-04b5-4de3-<wbr>9c81-7d25ba56407a (api:52)<br>29851:2018-03-21 10:46:25,023+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.03 seconds (__init__:573)<br>29852:2018-03-21 10:46:28,010+0200 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)<br>29853:2018-03-21 10:46:28,011+0200 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=011616ce-53c9-4cfc-<wbr>8b29-247b1be03409 (api:46)<br>29854:2018-03-21 10:46:28,011+0200 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-<wbr>b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000218955', 'lastCheck': '8.4', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000351741', 'lastCheck': '8.4', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=011616ce-53c9-4cfc-<wbr>8b29-247b1be03409 (api:52)<br>29855:2018-03-21 10:46:28,012+0200 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=85b42a8d-1bf1-495d-<wbr>ad32-a1f9710a6468 (api:46)<br>29856:2018-03-21 10:46:28,012+0200 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=85b42a8d-1bf1-495d-<wbr>ad32-a1f9710a6468 (api:52)<br>29857:2018-03-21 10:46:28,018+0200 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '10': {'cpuUser': '1.73', 'nodeIndex': 0, 'cpuSys': '0.53', 'cpuIdle': '97.74'}, '13': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '12': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '15': {'cpuUser': '0.67', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.06'}, '14': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.47'}, '1': {'cpuUser': '1.26', 'nodeIndex': 1, 'cpuSys': '1.33', 'cpuIdle': '97.41'}, '0': {'cpuUser': '0.40', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.53'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.73'}, '5': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '4': {'cpuUser': '13.78', 'nodeIndex': 0, 'cpuSys': '0.80', 'cpuIdle': '85.42'}, '7': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.67', 'cpuIdle': '98.86'}, '6': {'cpuUser': '1.07', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '98.66'}, '9': {'cpuUser': '1.20', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '98.53'}, '8': {'cpuUser': '0.80', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '98.87'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15136'}, '0': {'memPercent': 5, 'memFree': '15357'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000218955', 'lastCheck': '8.4', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000351741', 'lastCheck': '8.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'enp3s0f0', 'tx': '1091397599', 'txDropped': '0', 'rx': '11914966591', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1454'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'ovirtmgmt', 'tx': '1048745347', 'txDropped': '0', 'rx': '11615294897', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'lo', 'tx': '58205583384', 'txDropped': '0', 'rx': '58205583384', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621975.346633, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621975.346633, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621975.346633, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76546.85', 'cpuLoad': '0.18', 'cpuSys': '0.39', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.20', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31031, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1454', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:46:28 GMT', 'cpuUser': '1.38', 'memFree': 31287, 'cpuIdle': '98.23', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.87'}} from=::ffff:192.168.0.4,49914 (api:52)<br>29858:2018-03-21 10:46:28,020+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)<br>29859:2018-03-21 10:46:29,667+0200 INFO (itmap/0) [IOProcessClient] Starting client ioprocess-575 (__init__:308)<br>29860:2018-03-21 10:46:29,700+0200 INFO (itmap/1) [IOProcessClient] Starting client ioprocess-576 (__init__:308)<br>29861:2018-03-21 10:46:29,714+0200 INFO (ioprocess/1711202) [IOProcess] Starting ioprocess (__init__:437)<br>29862:2018-03-21 10:46:29,725+0200 INFO (ioprocess/1711208) [IOProcess] Starting ioprocess (__init__:437)<br>29863:2018-03-21 10:46:30,375+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=bdacf1c0-3fcb-43b1-<wbr>82e2-a041a26d0d7f (api:46)<br>29864:2018-03-21 10:46:30,376+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-<wbr>b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00031111', 'lastCheck': '0.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00045594', 'lastCheck': '0.6', 'valid': True}} from=internal, task_id=bdacf1c0-3fcb-43b1-<wbr>82e2-a041a26d0d7f (api:52)<br>29865:2018-03-21 10:46:30,376+0200 INFO (periodic/0) [vdsm.api] START multipath_health() from=internal, task_id=3d3cad70-e5c8-49b2-<wbr>9f5d-92a249cc102d (api:46)<br>29866:2018-03-21 10:46:30,377+0200 INFO (periodic/0) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=3d3cad70-e5c8-49b2-<wbr>9f5d-92a249cc102d (api:52)<br>29867:2018-03-21 10:46:35,108+0200 INFO (jsonrpc/6) [vdsm.api] START getSpmStatus(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=304d7e7c-03fc-44d6-<wbr>be32-d083ada09b30 (api:46)<br>29868:2018-03-21 10:46:35,114+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=304d7e7c-03fc-44d6-<wbr>be32-d083ada09b30 (api:52)<br>29869:2018-03-21 10:46:35,114+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:573)<br>29870:2018-03-21 10:46:35,170+0200 INFO (jsonrpc/4) [vdsm.api] START getStoragePoolInfo(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=76736a93-d4c4-4b2f-<wbr>9848-7b8731fe2b67 (api:46)<br>29871:2018-03-21 10:46:35,176+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8:Active,8a2c304b-<wbr>c8ae-438b-af54-fc8797ea149f:<wbr>Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=76736a93-d4c4-4b2f-<wbr>9848-7b8731fe2b67 (api:52)<br>29872:2018-03-21 10:46:35,177+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)<br>29873:2018-03-21 10:46:35,462+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br>29874:2018-03-21 10:46:35,462+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br>29875:2018-03-21 10:46:35,463+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br>29876:2018-03-21 10:46:36,948+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br>29877:2018-03-21 10:46:36,948+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br>29878:2018-03-21 10:46:36,949+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br>29879:2018-03-21 10:46:43,179+0200 INFO (jsonrpc/1) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)<br>29880:2018-03-21 10:46:43,180+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=83bbc95c-6d59-4eec-<wbr>82b1-e8993be28759 (api:46)<br>29881:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-<wbr>b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000486101', 'lastCheck': '3.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': '3.4', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=83bbc95c-6d59-4eec-<wbr>82b1-e8993be28759 (api:52)<br>29882:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=9b781324-b50b-49a9-<wbr>bf31-5eca835dc75c (api:46)<br>29883:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=9b781324-b50b-49a9-<wbr>bf31-5eca835dc75c (api:52)<br>29884:2018-03-21 10:46:43,188+0200 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.40', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.33'}, '10': {'cpuUser': '0.60', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '98.93'}, '13': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.73'}, '12': {'cpuUser': '1.46', 'nodeIndex': 0, 'cpuSys': '0.60', 'cpuIdle': '97.94'}, '15': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '99.47'}, '14': {'cpuUser': '0.40', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '99.13'}, '1': {'cpuUser': '1.33', 'nodeIndex': 1, 'cpuSys': '1.93', 'cpuIdle': '96.74'}, '0': {'cpuUser': '0.60', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.13'}, '3': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.40'}, '2': {'cpuUser': '6.99', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '92.61'}, '5': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '4': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.46'}, '7': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.60', 'cpuIdle': '98.93'}, '6': {'cpuUser': '0.53', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.20'}, '9': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '99.20'}, '8': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15131'}, '0': {'memPercent': 5, 'memFree': '15357'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000486101', 'lastCheck': '3.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': '3.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'enp3s0f0', 'tx': '1091404888', 'txDropped': '0', 'rx': '11914970667', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1455'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'ovirtmgmt', 'tx': '1048752406', 'txDropped': '0', 'rx': '11615298347', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'lo', 'tx': '58215969455', 'txDropped': '0', 'rx': '58215969455', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621990.363726, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621990.363726, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621990.363726, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '206', 'ksmPages': 100, 'elapsedTime': '76562.02', 'cpuLoad': '0.18', 'cpuSys': '0.45', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.27', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1455', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:46:43 GMT', 'cpuUser': '0.87', 'memFree': 31336, 'cpuIdle': '98.68', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '1.20'}} from=::ffff:192.168.0.4,49914 (api:52)<br>29885:2018-03-21 10:46:43,190+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)<br>29886:2018-03-21 10:46:45,288+0200 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=ffe57349-2532-4b64-<wbr>b32c-8a558273d8e5 (api:46)<br>29887:2018-03-21 10:46:45,292+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=ffe57349-2532-4b64-<wbr>b32c-8a558273d8e5 (api:52)<br>29888:2018-03-21 10:46:45,293+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:573)<br>29889:2018-03-21 10:46:45,298+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=b2ea28f0-abf0-4258-<wbr>8152-6035d93aa5dc (api:46)<br>29890:2018-03-21 10:46:45,305+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8:Active,8a2c304b-<wbr>c8ae-438b-af54-fc8797ea149f:<wbr>Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=b2ea28f0-abf0-4258-<wbr>8152-6035d93aa5dc (api:52)<br>29891:2018-03-21 10:46:45,306+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:573)<br>29892:2018-03-21 10:46:45,395+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=4010bf72-5c86-4daf-<wbr>a8f9-12ec229a86e4 (api:46)<br>29893:2018-03-21 10:46:45,395+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-<wbr>b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000486101', 'lastCheck': '5.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': '5.7', 'valid': True}} from=internal, task_id=4010bf72-5c86-4daf-<wbr>a8f9-12ec229a86e4 (api:52)<br>29894:2018-03-21 10:46:45,396+0200 INFO (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=37a1132f-452b-4805-<wbr>9cc0-b8e74e8d6b02 (api:46)<br>29895:2018-03-21 10:46:45,396+0200 INFO (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=37a1132f-452b-4805-<wbr>9cc0-b8e74e8d6b02 (api:52)<br>29896:2018-03-21 10:46:50,467+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)<br>29897:2018-03-21 10:46:50,468+0200 INFO (jsonrpc/0) [throttled] Current getAllVmStats: {} (throttledlog:103)<br>29898:2018-03-21 10:46:50,468+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)<br>29899:2018-03-21 10:46:50,468+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573)<br>29900:2018-03-21 10:46:51,967+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,36114 (api:46)<br>29901:2018-03-21 10:46:51,968+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)<br>29902:2018-03-21 10:46:51,968+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br>29903:2018-03-21 10:46:55,648+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=a975635f-59d7-4f40-<wbr>b9a7-a5ea7150368e (api:46)<br>29904:2018-03-21 10:46:55,653+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=a975635f-59d7-4f40-<wbr>b9a7-a5ea7150368e (api:52)<br>29905:2018-03-21 10:46:55,653+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:573)<br>29906:2018-03-21 10:46:55,659+0200 INFO (jsonrpc/2) [vdsm.api] START getStoragePoolInfo(spUUID=u'<wbr>80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=97d4ae6c-3719-4149-<wbr>85fd-28e924ad8883 (api:46)<br>29907:2018-03-21 10:46:55,666+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8:Active,8a2c304b-<wbr>c8ae-438b-af54-fc8797ea149f:<wbr>Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/<wbr>node11.starlett.lv:_vmraid_<wbr>nfs_iso/8a2c304b-c8ae-438b-<wbr>af54-fc8797ea149f/images/<wbr>11111111-1111-1111-1111-<wbr>111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=97d4ae6c-3719-4149-<wbr>85fd-28e924ad8883 (api:52)<br>29908:2018-03-21 10:46:55,667+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:573)<br>29909:2018-03-21 10:46:59,473+0200 INFO (jsonrpc/7) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)<br>29910:2018-03-21 10:46:59,474+0200 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=177b551b-9bd3-45a4-<wbr>84d0-5244630c08e5 (api:46)<br>29911:2018-03-21 10:46:59,474+0200 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-<wbr>b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470906', 'lastCheck': '9.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000334928', 'lastCheck': '9.7', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=177b551b-9bd3-45a4-<wbr>84d0-5244630c08e5 (api:52)<br>29912:2018-03-21 10:46:59,475+0200 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=30753d5c-9bfd-4b0f-<wbr>ac83-e32278d82b63 (api:46)<br>29913:2018-03-21 10:46:59,475+0200 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=30753d5c-9bfd-4b0f-<wbr>ac83-e32278d82b63 (api:52)<br>29914:2018-03-21 10:46:59,482+0200 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '10': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '15': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '1': {'cpuUser': '1.07', 'nodeIndex': 1, 'cpuSys': '1.07', 'cpuIdle': '97.86'}, '0': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '4': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.26'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.47', 'cpuIdle': '99.53'}, '6': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.73'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '8': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15125'}, '0': {'memPercent': 5, 'memFree': '15364'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-<wbr>85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470906', 'lastCheck': '9.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-<wbr>fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000334928', 'lastCheck': '9.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'enp3s0f0', 'tx': '1091412332', 'txDropped': '0', 'rx': '11914976557', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1456'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'ovirtmgmt', 'tx': '1048761326', 'txDropped': '0', 'rx': '11615304242', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'lo', 'tx': '58228460774', 'txDropped': '0', 'rx': '58228460774', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521622005.383289, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521622005.383289, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521622005.383289, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '208', 'ksmPages': 100, 'elapsedTime': '76578.31', 'cpuLoad': '0.17', 'cpuSys': '0.20', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.13', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31078, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1456', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:46:59 GMT', 'cpuUser': '0.15', 'memFree': 31334, 'cpuIdle': '99.65', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.60'}} from=::ffff:192.168.0.4,49914 (api:52)<br>29915:2018-03-21 10:46:59,484+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)</div><div><br><div><br><blockquote type="cite"><div>On 21 Mar 2018, at 11:33, Fred Rolland <<a href="mailto:frolland@redhat.com" target="_blank">frolland@redhat.com</a>> wrote:</div><br class="m_1118283051208492744Apple-interchange-newline"><div><div dir="ltr"><div>Can you provide the vdsm logs from the host.<br></div>It looks the vdsm failed to connect to the server.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 21, 2018 at 11:21 AM, Andrei Verovski <span dir="ltr"><<a href="mailto:andreil1@starlett.lv" target="_blank">andreil1@starlett.lv</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space">Hi,<div><br></div><div>I have 2-host oVirt setup with 2 Data Centers, one with local storage domain (DC #1) for VMs + Export domain on NFS, another with all NFS shared (DC #2).</div><div>Trying to export VMs from DC #1 to DC #2.</div><div>VMs are exported to DC #1 export domain (NFS), then domain put into maintenance mode and detached from DC #1.</div><div><br></div><div>Unfortunately, attaching it to DC #2 failed. Logs attached. Tried to run this command twice.</div><div>Workaround are possible in order to accomplish this task, yet it would be better to do in a way as it was designed.<br>
Thanks.<br><br><br><div>2018-03-21 10:43:24,024+02 ERROR [org.ovirt.engine.core.dal.dbb<wbr>roker.auditloghandling.AuditLo<wbr>gDirector] (EE-ManagedThreadFactory-engin<wbr>e-Thread-128904) [<font color="#942192">570ec5d9-fff5-4656-afbd-90b32<wbr>07a616e</font>] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_<wbr>POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz)<br>2018-03-21 10:46:16,512+02 ERROR [org.ovirt.engine.core.dal.dbb<wbr>roker.auditloghandling.AuditLo<wbr>gDirector] (EE-ManagedThreadFactory-engin<wbr>e-Thread-128956) [1435fc81] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED<wbr>(188), Failed to connect Host node11 to the Storage Domains node10-NFS-EXPORTS.<br>2018-03-21 10:46:16,651+02 ERROR [org.ovirt.engine.core.dal.dbb<wbr>roker.auditloghandling.AuditLo<wbr>gDirector] (EE-ManagedThreadFactory-engin<wbr>e-Thread-128955) [<font color="#942192">a81ffa4a-5a58-41a0-888a-f0edc<wbr>321609b</font>] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_<wbr>POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz)<br><br><br><font color="#aa7942"><b>tail -n 1000 engine.log | grep 570ec5d9-fff5-4656-afbd-90b320<wbr>7a616e</b></font><br>2018-03-21 10:41:14,643+02 INFO [org.ovirt.engine.core.bll.sto<wbr>rage.domain.AttachStorageDomai<wbr>nToPoolCommand] (default task-2) [570ec5d9-fff5-4656-afbd-90b32<wbr>07a616e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[<wbr>1d7208ce-d3a1-4406-9638-fe7051<wbr>562994=STORAGE]', sharedLocks=''}'<br>2018-03-21 10:41:16,129+02 INFO [org.ovirt.engine.core.bll.sto<wbr>rage.domain.AttachStorageDomai<wbr>nToPoolCommand] (EE-ManagedThreadFactory-engin<wbr>e-Thread-128904) [570ec5d9-fff5-4656-afbd-90b32<wbr>07a616e] Running command: AttachStorageDomainToPoolComma<wbr>nd internal: false. Entities affected : ID: 1d7208ce-d3a1-4406-9638-fe7051<wbr>562994 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: 80cc922f-8dea-4fed-b951-1060ba<wbr>116ad5 Type: StoragePoolAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN<br>2018-03-21 10:43:23,564+02 ERROR [org.ovirt.engine.core.bll.sto<wbr>rage.domain.AttachStorageDomai<wbr>nToPoolCommand] (EE-ManagedThreadFactory-engin<wbr>e-Thread-128904) [570ec5d9-fff5-4656-afbd-90b32<wbr>07a616e] Cannot connect storage connection server, aborting attach storage domain operation.<br>2018-03-21 10:43:23,567+02 INFO [<a href="http://org.ovirt.engine.core.bll.Com">org.ovirt.engine.core.bll.Com</a><wbr>mandCompensator] (EE-ManagedThreadFactory-engin<wbr>e-Thread-128904) [570ec5d9-fff5-4656-afbd-90b32<wbr>07a616e] Command [id=921ca7cd-4f93-46aa-8de2-91<wbr>b13b8f96cb]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.b<wbr>usinessentities.StoragePoolIso<wbr>Map; snapshot: StoragePoolIsoMapId:{storagePo<wbr>olId='80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', storageId='1d7208ce-d3a1-4406-<wbr>9638-fe7051562994'}.<br>2018-03-21 10:43:24,024+02 ERROR [org.ovirt.engine.core.dal.dbb<wbr>roker.auditloghandling.AuditLo<wbr>gDirector] (EE-ManagedThreadFactory-engin<wbr>e-Thread-128904) [570ec5d9-fff5-4656-afbd-90b32<wbr>07a616e] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_<wbr>POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz)<br>2018-03-21 10:43:24,114+02 INFO [org.ovirt.engine.core.bll.sto<wbr>rage.domain.AttachStorageDomai<wbr>nToPoolCommand] (EE-ManagedThreadFactory-engin<wbr>e-Thread-128904) [570ec5d9-fff5-4656-afbd-90b32<wbr>07a616e] Lock freed to object 'EngineLock:{exclusiveLocks='[<wbr>1d7208ce-d3a1-4406-9638-fe7051<wbr>562994=STORAGE]', sharedLocks=''}'<br><br><br><br><font color="#aa7942"><b>[root@node00 ovirt-engine]# tail -n 1000 engine.log | grep a81ffa4a-5a58-41a0-888a-f0edc3<wbr>21609b</b></font><br>2018-03-21 10:44:11,025+02 INFO [org.ovirt.engine.core.bll.sto<wbr>rage.domain.AttachStorageDomai<wbr>nToPoolCommand] (default task-16) [a81ffa4a-5a58-41a0-888a-f0edc<wbr>321609b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[<wbr>1d7208ce-d3a1-4406-9638-fe7051<wbr>562994=STORAGE]', sharedLocks=''}'<br>2018-03-21 10:44:11,236+02 INFO [org.ovirt.engine.core.bll.sto<wbr>rage.domain.AttachStorageDomai<wbr>nToPoolCommand] (EE-ManagedThreadFactory-engin<wbr>e-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc<wbr>321609b] Running command: AttachStorageDomainToPoolComma<wbr>nd internal: false. Entities affected : ID: 1d7208ce-d3a1-4406-9638-fe7051<wbr>562994 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: 80cc922f-8dea-4fed-b951-1060ba<wbr>116ad5 Type: StoragePoolAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN<br>2018-03-21 10:46:16,567+02 ERROR [org.ovirt.engine.core.bll.sto<wbr>rage.domain.AttachStorageDomai<wbr>nToPoolCommand] (EE-ManagedThreadFactory-engin<wbr>e-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc<wbr>321609b] Cannot connect storage connection server, aborting attach storage domain operation.<br>2018-03-21 10:46:16,568+02 INFO [<a href="http://org.ovirt.engine.core.bll.Com">org.ovirt.engine.core.bll.Com</a><wbr>mandCompensator] (EE-ManagedThreadFactory-engin<wbr>e-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc<wbr>321609b] Command [id=b5c25100-1a8a-4db0-9509-99<wbr>cfa60995b1]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.b<wbr>usinessentities.StoragePoolIso<wbr>Map; snapshot: StoragePoolIsoMapId:{storagePo<wbr>olId='80cc922f-8dea-4fed-b951-<wbr>1060ba116ad5', storageId='1d7208ce-d3a1-4406-<wbr>9638-fe7051562994'}.<br>2018-03-21 10:46:16,651+02 ERROR [org.ovirt.engine.core.dal.dbb<wbr>roker.auditloghandling.AuditLo<wbr>gDirector] (EE-ManagedThreadFactory-engin<wbr>e-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc<wbr>321609b] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_<wbr>POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz)<br>2018-03-21 10:46:16,681+02 INFO [org.ovirt.engine.core.bll.sto<wbr>rage.domain.AttachStorageDomai<wbr>nToPoolCommand] (EE-ManagedThreadFactory-engin<wbr>e-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc<wbr>321609b] Lock freed to object 'EngineLock:{exclusiveLocks='[<wbr>1d7208ce-d3a1-4406-9638-fe7051<wbr>562994=STORAGE]', sharedLocks='’}'</div><div><br></div></div></div><br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
<br></blockquote></div><br></div>
</div></blockquote></div><br></div></div></blockquote></div><br></div>