node11 fails to mount the export domain from node 10.

You can try manually to see if you have access from node 11.
Look at this page to debug the NFS connection:
https://www.ovirt.org/documentation/how-to/troubleshooting/troubleshooting-nfs-storage-issues/

On Wed, Mar 21, 2018 at 12:04 PM, Andrei Verovski <andreil1@starlett.lv> wrote:
Hi,


Errors occurred at 10.43 AM and 10.46 AM
node00.starlett.lv - 192.168.0.4 - oVirt host engine (separate PC, not hosted)
node10.starlett.lv - 192.168.0.5 - host #1 of DC #1, export domain from which was detached from DC #1
node11.starlett.lv - 192.168.0.6 - host #1 of DC #2, 

Logs from DC#2 node11, to which I’m trying to attach export domain located at NFS share node10.

grep 570ec5d9-fff5-4656-afbd-90b3207a616e
within vdsm.log returned nothing, so I did


grep -n 10:43 vdsm.log | tail -1000
1011:2018-03-21 06:10:43,077+0200 INFO  (jsonrpc/3) [api.host] START getAllVmStats() from=::1,36114 (api:46)
1012:2018-03-21 06:10:43,077+0200 INFO  (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)
1013:2018-03-21 06:10:43,077+0200 INFO  (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573)
1014:2018-03-21 06:10:43,868+0200 INFO  (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)
1015:2018-03-21 06:10:43,868+0200 INFO  (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)
1016:2018-03-21 06:10:43,868+0200 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)
19729:2018-03-21 09:10:43,641+0200 INFO  (jsonrpc/5) [api.host] START getAllVmStats() from=::1,36114 (api:46)
19730:2018-03-21 09:10:43,641+0200 INFO  (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)
19731:2018-03-21 09:10:43,642+0200 INFO  (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)
29302:2018-03-21 10:43:00,085+0200 INFO  (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=7b690c8a-5470-44da-a8f3-7e7e9b018e88 (api:46)
29303:2018-03-21 10:43:00,085+0200 INFO  (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000477038', 'lastCheck': '0.5', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': '0.4', 'valid': True}} from=internal, task_id=7b690c8a-5470-44da-a8f3-7e7e9b018e88 (api:52)
29304:2018-03-21 10:43:00,086+0200 INFO  (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=9a625d43-a03c-429c-856f-9aa8e5ff65b5 (api:46)
29305:2018-03-21 10:43:00,086+0200 INFO  (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=9a625d43-a03c-429c-856f-9aa8e5ff65b5 (api:52)
29306:2018-03-21 10:43:05,382+0200 INFO  (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)
29307:2018-03-21 10:43:05,383+0200 INFO  (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)
29308:2018-03-21 10:43:05,383+0200 INFO  (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)
29309:2018-03-21 10:43:06,248+0200 INFO  (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)
29310:2018-03-21 10:43:06,249+0200 INFO  (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=f80f9bb8-2e84-4f45-ab58-09a88e808cf3 (api:46)
29311:2018-03-21 10:43:06,250+0200 INFO  (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000477038', 'lastCheck': '6.7', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': '6.6', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=f80f9bb8-2e84-4f45-ab58-09a88e808cf3 (api:52)
29312:2018-03-21 10:43:06,250+0200 INFO  (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=ddd7064d-cd12-43bf-a177-b2a93cc9035e (api:46)
29313:2018-03-21 10:43:06,250+0200 INFO  (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=ddd7064d-cd12-43bf-a177-b2a93cc9035e (api:52)
29314:2018-03-21 10:43:06,256+0200 INFO  (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '10': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.47'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '12': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '15': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.47', 'cpuIdle': '99.33'}, '14': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '1': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': '1.60', 'cpuIdle': '97.27'}, '0': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.26'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '2': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.54'}, '5': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.74'}, '4': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '7': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.60', 'cpuIdle': '99.27'}, '6': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '8': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15142'}, '0': {'memPercent': 5, 'memFree': '15349'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000477038', 'lastCheck': '6.7', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': '6.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'enp3s0f0', 'tx': '1091287978', 'txDropped': '0', 'rx': '11914868563', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1439'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'ovirtmgmt', 'tx': '1048639112', 'txDropped': '0', 'rx': '11615209092', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'lo', 'tx': '58057075945', 'txDropped': '0', 'rx': '58057075945', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621780.072418, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621780.072418, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621780.072418, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76345.09', 'cpuLoad': '0.16', 'cpuSys': '0.36', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31081, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1439', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:43:06 GMT', 'cpuUser': '0.21', 'memFree': 31337, 'cpuIdle': '99.43', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.93'}} from=::ffff:192.168.0.4,49914 (api:52)
29315:2018-03-21 10:43:06,258+0200 INFO  (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)
29316:2018-03-21 10:43:06,632+0200 INFO  (jsonrpc/0) [api.host] START getAllVmStats() from=::1,36114 (api:46)
29317:2018-03-21 10:43:06,632+0200 INFO  (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)
29318:2018-03-21 10:43:06,633+0200 INFO  (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)
29319:2018-03-21 10:43:07,415+0200 INFO  (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=276c13c7-61ec-4782-a0d9-5cd0f0b0a729 (api:46)
29320:2018-03-21 10:43:07,420+0200 INFO  (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=276c13c7-61ec-4782-a0d9-5cd0f0b0a729 (api:52)
29321:2018-03-21 10:43:07,420+0200 INFO  (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)
29322:2018-03-21 10:43:07,427+0200 INFO  (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=344b18df-a428-429a-b49f-381f901d1618 (api:46)
29323:2018-03-21 10:43:07,434+0200 INFO  (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=344b18df-a428-429a-b49f-381f901d1618 (api:52)
29324:2018-03-21 10:43:07,435+0200 INFO  (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)
29325:2018-03-21 10:43:13,839+0200 WARN  (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/1 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524', u'connection': u'node10.starlett.lv:/vmdata/nfs/exports', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'protocol_version': u'auto', u'password': '********', u'port': u''}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': u'5f98dfd5-998f-44a6-8594-e5f40144b0ed'} at 0x4076150> timeout=60, duration=120 at 0x402fe50> task#=3754 at 0x353e690>, traceback:
29376:2018-03-21 10:43:15,104+0200 INFO  (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=672a883e-1d60-4981-9eee-bdcac2ec30c1 (api:46)
29377:2018-03-21 10:43:15,105+0200 INFO  (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000488777', 'lastCheck': '5.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000481962', 'lastCheck': '5.4', 'valid': True}} from=internal, task_id=672a883e-1d60-4981-9eee-bdcac2ec30c1 (api:52)
29378:2018-03-21 10:43:15,105+0200 INFO  (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=55f1b797-da2a-4b07-affe-5b9e41a74307 (api:46)
29379:2018-03-21 10:43:15,106+0200 INFO  (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=55f1b797-da2a-4b07-affe-5b9e41a74307 (api:52)
29380:2018-03-21 10:43:17,645+0200 INFO  (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=52a6ddeb-3c20-4f3f-b27d-94d19580b473 (api:46)
29381:2018-03-21 10:43:17,649+0200 INFO  (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=52a6ddeb-3c20-4f3f-b27d-94d19580b473 (api:52)
29382:2018-03-21 10:43:17,650+0200 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)
29383:2018-03-21 10:43:17,656+0200 INFO  (jsonrpc/2) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=1a103883-6a6a-4d61-ade6-96559591ddb0 (api:46)
29384:2018-03-21 10:43:17,663+0200 INFO  (jsonrpc/2) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=1a103883-6a6a-4d61-ade6-96559591ddb0 (api:52)
29385:2018-03-21 10:43:17,663+0200 INFO  (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)
29386:2018-03-21 10:43:18,882+0200 ERROR (jsonrpc/1) [storage.HSM] Could not connect to storageServer (hsm:2407)
29406:2018-03-21 10:43:18,883+0200 INFO  (jsonrpc/1) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 477, 'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524'}]} from=::ffff:192.168.0.4,49914, flow_id=4a53b512, task_id=e598cbe0-cde8-4c73-b526-4398df05e67f (api:52)
29407:2018-03-21 10:43:18,883+0200 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 125.04 seconds (__init__:573)
29408:2018-03-21 10:43:20,387+0200 INFO  (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)
29409:2018-03-21 10:43:20,388+0200 INFO  (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)
29410:2018-03-21 10:43:20,388+0200 INFO  (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573)
29411:2018-03-21 10:43:21,497+0200 INFO  (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)
29412:2018-03-21 10:43:21,498+0200 INFO  (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=a7469c1f-3d6d-42e7-8798-5c5335da4a9b (api:46)
29413:2018-03-21 10:43:21,498+0200 INFO  (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000445096', 'lastCheck': '1.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000296586', 'lastCheck': '1.8', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=a7469c1f-3d6d-42e7-8798-5c5335da4a9b (api:52)
29414:2018-03-21 10:43:21,499+0200 INFO  (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=9fa5dae3-e2bd-4186-9a1d-afb0aa6e2511 (api:46)
29415:2018-03-21 10:43:21,499+0200 INFO  (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=9fa5dae3-e2bd-4186-9a1d-afb0aa6e2511 (api:52)
29416:2018-03-21 10:43:21,505+0200 INFO  (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '10': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '12': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '15': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.60'}, '1': {'cpuUser': '1.00', 'nodeIndex': 1, 'cpuSys': '1.00', 'cpuIdle': '98.00'}, '0': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '3': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '2': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.66'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '7': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '6': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '8': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15142'}, '0': {'memPercent': 5, 'memFree': '15351'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000445096', 'lastCheck': '1.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000296586', 'lastCheck': '1.8', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'enp3s0f0', 'tx': '1091295628', 'txDropped': '0', 'rx': '11914875210', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1442'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'ovirtmgmt', 'tx': '1048646480', 'txDropped': '0', 'rx': '11615214750', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'lo', 'tx': '58066397668', 'txDropped': '0', 'rx': '58066397668', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621795.091263, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621795.091263, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621795.091263, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76360.34', 'cpuLoad': '0.15', 'cpuSys': '0.18', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31031, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1442', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:43:21 GMT', 'cpuUser': '0.15', 'memFree': 31287, 'cpuIdle': '99.67', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.67'}} from=::ffff:192.168.0.4,49914 (api:52)
29417:2018-03-21 10:43:21,508+0200 INFO  (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.02 seconds (__init__:573)
29418:2018-03-21 10:43:21,651+0200 INFO  (jsonrpc/0) [api.host] START getAllVmStats() from=::1,36114 (api:46)
29419:2018-03-21 10:43:21,651+0200 INFO  (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)
29420:2018-03-21 10:43:21,652+0200 INFO  (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)
29421:2018-03-21 10:43:27,764+0200 INFO  (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=46d19cab-8903-4c52-9bb2-4dd8f370997e (api:46)
29422:2018-03-21 10:43:27,769+0200 INFO  (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=46d19cab-8903-4c52-9bb2-4dd8f370997e (api:52)
29423:2018-03-21 10:43:27,770+0200 INFO  (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)
29424:2018-03-21 10:43:27,815+0200 INFO  (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=290acc73-8f25-4fd7-9263-7bd05d46430a (api:46)
29425:2018-03-21 10:43:27,822+0200 INFO  (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=290acc73-8f25-4fd7-9263-7bd05d46430a (api:52)
29426:2018-03-21 10:43:27,823+0200 INFO  (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)
29427:2018-03-21 10:43:30,121+0200 INFO  (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=0abdebcf-2b0b-4d33-b552-c0f2ed3e567b (api:46)
29428:2018-03-21 10:43:30,122+0200 INFO  (periodic/3) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000402077', 'lastCheck': '0.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': '0.5', 'valid': True}} from=internal, task_id=0abdebcf-2b0b-4d33-b552-c0f2ed3e567b (api:52)
29429:2018-03-21 10:43:30,122+0200 INFO  (periodic/3) [vdsm.api] START multipath_health() from=internal, task_id=acde7e7d-ab4a-4811-934f-4d312da33208 (api:46)
29430:2018-03-21 10:43:30,123+0200 INFO  (periodic/3) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=acde7e7d-ab4a-4811-934f-4d312da33208 (api:52)
29431:2018-03-21 10:43:35,393+0200 INFO  (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)
29432:2018-03-21 10:43:35,394+0200 INFO  (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)
29433:2018-03-21 10:43:35,394+0200 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)
29434:2018-03-21 10:43:36,671+0200 INFO  (jsonrpc/2) [api.host] START getAllVmStats() from=::1,36114 (api:46)
29435:2018-03-21 10:43:36,671+0200 INFO  (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)
29436:2018-03-21 10:43:36,672+0200 INFO  (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)
29437:2018-03-21 10:43:37,356+0200 INFO  (jsonrpc/1) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)
29438:2018-03-21 10:43:37,357+0200 INFO  (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=bc14e253-d3d4-4cc3-9ae1-c0e6ac6e37f7 (api:46)
29439:2018-03-21 10:43:37,357+0200 INFO  (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000402077', 'lastCheck': '7.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': '7.7', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=bc14e253-d3d4-4cc3-9ae1-c0e6ac6e37f7 (api:52)
29440:2018-03-21 10:43:37,358+0200 INFO  (jsonrpc/1) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=23c698b8-d5bd-4746-b738-97b10f3e4bfb (api:46)
29441:2018-03-21 10:43:37,358+0200 INFO  (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=23c698b8-d5bd-4746-b738-97b10f3e4bfb (api:52)
29442:2018-03-21 10:43:37,365+0200 INFO  (jsonrpc/1) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '10': {'cpuUser': '3.06', 'nodeIndex': 0, 'cpuSys': '0.80', 'cpuIdle': '96.14'}, '13': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '12': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '15': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '14': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.33'}, '1': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': '1.33', 'cpuIdle': '97.54'}, '0': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.20'}, '3': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '2': {'cpuUser': '0.67', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '98.93'}, '5': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.74'}, '4': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '7': {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': '0.73', 'cpuIdle': '98.94'}, '6': {'cpuUser': '5.39', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '94.14'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '8': {'cpuUser': '0.53', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.07'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15143'}, '0': {'memPercent': 5, 'memFree': '15348'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000402077', 'lastCheck': '7.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': '7.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'enp3s0f0', 'tx': '1091305295', 'txDropped': '0', 'rx': '11914880953', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1442'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'ovirtmgmt', 'tx': '1048655897', 'txDropped': '0', 'rx': '11615220341', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'lo', 'tx': '58078891247', 'txDropped': '0', 'rx': '58078891247', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621810.108511, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621810.108511, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621810.108511, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76376.19', 'cpuLoad': '0.16', 'cpuSys': '0.39', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.00', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1442', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:43:37 GMT', 'cpuUser': '0.79', 'memFree': 31336, 'cpuIdle': '98.82', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.73'}} from=::ffff:192.168.0.4,49914 (api:52)
29443:2018-03-21 10:43:37,367+0200 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)
29444:2018-03-21 10:43:38,124+0200 INFO  (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=9d2a82dd-c76c-4eb5-a1e1-b49bf004384e (api:46)
29445:2018-03-21 10:43:38,128+0200 INFO  (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=9d2a82dd-c76c-4eb5-a1e1-b49bf004384e (api:52)
29446:2018-03-21 10:43:38,129+0200 INFO  (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)
29447:2018-03-21 10:43:38,171+0200 INFO  (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=62d44b0b-20b2-47ae-bfb8-d4717e301b69 (api:46)
29448:2018-03-21 10:43:38,177+0200 INFO  (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=62d44b0b-20b2-47ae-bfb8-d4717e301b69 (api:52)
29449:2018-03-21 10:43:38,178+0200 INFO  (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)
29450:2018-03-21 10:43:45,147+0200 INFO  (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=49c33285-a222-464d-8984-4e75a5c6354a (api:46)
29451:2018-03-21 10:43:45,148+0200 INFO  (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000454102', 'lastCheck': '5.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000347381', 'lastCheck': '5.5', 'valid': True}} from=internal, task_id=49c33285-a222-464d-8984-4e75a5c6354a (api:52)
29452:2018-03-21 10:43:45,148+0200 INFO  (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=eed6b564-76b1-4f9b-80a2-1f0f6d9bbe32 (api:46)
29453:2018-03-21 10:43:45,149+0200 INFO  (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=eed6b564-76b1-4f9b-80a2-1f0f6d9bbe32 (api:52)
29454:2018-03-21 10:43:48,405+0200 INFO  (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=84651a37-7d13-41a9-9dcf-2169a9c46fc5 (api:46)
29455:2018-03-21 10:43:48,410+0200 INFO  (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=84651a37-7d13-41a9-9dcf-2169a9c46fc5 (api:52)
29456:2018-03-21 10:43:48,411+0200 INFO  (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)
29457:2018-03-21 10:43:48,417+0200 INFO  (jsonrpc/4) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=4ebd1d1a-11ac-41ee-b459-7e45f008776b (api:46)
29458:2018-03-21 10:43:48,423+0200 INFO  (jsonrpc/4) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=4ebd1d1a-11ac-41ee-b459-7e45f008776b (api:52)
29459:2018-03-21 10:43:48,424+0200 INFO  (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)
29460:2018-03-21 10:43:50,399+0200 INFO  (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)
29461:2018-03-21 10:43:50,400+0200 INFO  (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)
29462:2018-03-21 10:43:50,400+0200 INFO  (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)
29463:2018-03-21 10:43:51,694+0200 INFO  (jsonrpc/7) [api.host] START getAllVmStats() from=::1,36114 (api:46)
29464:2018-03-21 10:43:51,694+0200 INFO  (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)
29465:2018-03-21 10:43:51,695+0200 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)
29466:2018-03-21 10:43:52,516+0200 INFO  (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)
29467:2018-03-21 10:43:52,517+0200 INFO  (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=0a88f0dc-eccf-4d24-9ac2-22e079a1480a (api:46)
29468:2018-03-21 10:43:52,517+0200 INFO  (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000487937', 'lastCheck': '3.0', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000455956', 'lastCheck': '2.9', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=0a88f0dc-eccf-4d24-9ac2-22e079a1480a (api:52)
29469:2018-03-21 10:43:52,518+0200 INFO  (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=a7a44ff7-a137-41b6-9ad5-2c0b32ab79ec (api:46)
29470:2018-03-21 10:43:52,518+0200 INFO  (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=a7a44ff7-a137-41b6-9ad5-2c0b32ab79ec (api:52)
29471:2018-03-21 10:43:52,524+0200 INFO  (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '10': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '13': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '15': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '1': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': '1.13', 'cpuIdle': '97.74'}, '0': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '5': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '4': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.60'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '99.67'}, '6': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '8': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15143'}, '0': {'memPercent': 5, 'memFree': '15349'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000487937', 'lastCheck': '3.0', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000455956', 'lastCheck': '2.9', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'enp3s0f0', 'tx': '1091312739', 'txDropped': '0', 'rx': '11914899051', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1443'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'ovirtmgmt', 'tx': '1048663107', 'txDropped': '0', 'rx': '11615236520', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'lo', 'tx': '58090295494', 'txDropped': '0', 'rx': '58090295494', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621825.13588, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621825.13588, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621825.13588, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76391.35', 'cpuLoad': '0.15', 'cpuSys': '0.20', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.13', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1443', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:43:52 GMT', 'cpuUser': '0.14', 'memFree': 31336, 'cpuIdle': '99.66', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.73'}} from=::ffff:192.168.0.4,49914 (api:52)
29472:2018-03-21 10:43:52,526+0200 INFO  (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)
29473:2018-03-21 10:43:58,686+0200 INFO  (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=d6d29cf6-c083-4639-91a2-f062eae4e629 (api:46)
29474:2018-03-21 10:43:58,690+0200 INFO  (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=d6d29cf6-c083-4639-91a2-f062eae4e629 (api:52)
29475:2018-03-21 10:43:58,690+0200 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)
29476:2018-03-21 10:43:58,697+0200 INFO  (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=9dbd91df-dd9c-469c-8308-bfbfc228937c (api:46)
29477:2018-03-21 10:43:58,703+0200 INFO  (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=9dbd91df-dd9c-469c-8308-bfbfc228937c (api:52)
29478:2018-03-21 10:43:58,704+0200 INFO  (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)



grep -n 10:46 vdsm.log | tail -1000
1017:2018-03-21 06:10:46,882+0200 INFO  (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)
1018:2018-03-21 06:10:46,883+0200 INFO  (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=6e045b00-88b2-469b-80d6-af38303f8c32 (api:46)
1019:2018-03-21 06:10:46,883+0200 INFO  (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000332785', 'lastCheck': '7.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000203914', 'lastCheck': '5.6', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=6e045b00-88b2-469b-80d6-af38303f8c32 (api:52)
1020:2018-03-21 06:10:46,884+0200 INFO  (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=475a5b40-4a43-42ee-9524-2bdd220aed83 (api:46)
1021:2018-03-21 06:10:46,884+0200 INFO  (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=475a5b40-4a43-42ee-9524-2bdd220aed83 (api:52)
1022:2018-03-21 06:10:46,889+0200 INFO  (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '10': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '12': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '15': {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.40'}, '14': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '1': {'cpuUser': '3.00', 'nodeIndex': 1, 'cpuSys': '1.40', 'cpuIdle': '95.60'}, '0': {'cpuUser': '0.73', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.07'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '2': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '4': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '7': {'cpuUser': '0.80', 'nodeIndex': 1, 'cpuSys': '0.80', 'cpuIdle': '98.40'}, '6': {'cpuUser': '5.46', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '94.27'}, '9': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '8': {'cpuUser': '1.27', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '98.26'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15128'}, '0': {'memPercent': 5, 'memFree': '15366'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000332785', 'lastCheck': '7.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000203914', 'lastCheck': '5.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'enp3s0f0', 'tx': '1082264809', 'txDropped': '0', 'rx': '11907188594', 'rxErrors': '0', 'speed': '100', 'rxDropped': '289'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'ovirtmgmt', 'tx': '1039869554', 'txDropped': '0', 'rx': '11608697383', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'lo', 'tx': '45587320171', 'txDropped': '0', 'rx': '45587320171', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '53620', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521605437.203776, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521605437.203776, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521605437.203776, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '60005.72', 'cpuLoad': '0.06', 'cpuSys': '0.38', 'diskStats': {'/var/log': {'free': '7357'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31084, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '289', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T04:10:46 GMT', 'cpuUser': '0.83', 'memFree': 31340, 'cpuIdle': '98.79', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.80'}} from=::ffff:192.168.0.4,49914 (api:52)
1023:2018-03-21 06:10:46,891+0200 INFO  (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)
13522:2018-03-21 08:10:46,827+0200 INFO  (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)
13523:2018-03-21 08:10:46,828+0200 INFO  (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)
13524:2018-03-21 08:10:46,828+0200 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)
29734:2018-03-21 10:46:00,338+0200 INFO  (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=af099ce6-0f44-461b-9e8e-99de63b3884f (api:46)
29735:2018-03-21 10:46:00,339+0200 INFO  (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470245', 'lastCheck': '0.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000476935', 'lastCheck': '0.7', 'valid': True}} from=internal, task_id=af099ce6-0f44-461b-9e8e-99de63b3884f (api:52)
29736:2018-03-21 10:46:00,339+0200 INFO  (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=4a5709d9-e322-44eb-a8a4-bbe3f5e5e1bd (api:46)
29737:2018-03-21 10:46:00,340+0200 INFO  (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=4a5709d9-e322-44eb-a8a4-bbe3f5e5e1bd (api:52)
29738:2018-03-21 10:46:04,637+0200 INFO  (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=5b9300f7-f8c4-4f63-a49c-45ca8119d3b9 (api:46)
29739:2018-03-21 10:46:04,642+0200 INFO  (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=5b9300f7-f8c4-4f63-a49c-45ca8119d3b9 (api:52)
29740:2018-03-21 10:46:04,642+0200 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)
29741:2018-03-21 10:46:04,648+0200 INFO  (jsonrpc/1) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=f4db5bfa-e131-4f9a-baaa-e7555c996ab0 (api:46)
29742:2018-03-21 10:46:04,655+0200 INFO  (jsonrpc/1) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=f4db5bfa-e131-4f9a-baaa-e7555c996ab0 (api:52)
29743:2018-03-21 10:46:04,656+0200 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:573)
29744:2018-03-21 10:46:05,451+0200 INFO  (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)
29745:2018-03-21 10:46:05,452+0200 INFO  (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)
29746:2018-03-21 10:46:05,452+0200 INFO  (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)
29747:2018-03-21 10:46:06,903+0200 INFO  (jsonrpc/3) [api.host] START getAllVmStats() from=::1,36114 (api:46)
29748:2018-03-21 10:46:06,904+0200 INFO  (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)
29749:2018-03-21 10:46:06,904+0200 INFO  (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)
29750:2018-03-21 10:46:08,791+0200 WARN  (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/6 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524', u'connection': u'node10.starlett.lv:/vmdata/nfs/exports', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'protocol_version': u'auto', u'password': '********', u'port': u''}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': u'71143eef-aac6-4996-9783-3e0e3da180c3'} at 0x3604fd0> timeout=60, duration=120 at 0x3604e10> task#=3762 at 0x3541650>, traceback:
29801:2018-03-21 10:46:12,852+0200 INFO  (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)
29802:2018-03-21 10:46:12,853+0200 INFO  (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=391fdab9-d3c2-4075-9c34-177a07a21ec3 (api:46)
29803:2018-03-21 10:46:12,854+0200 INFO  (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000492277', 'lastCheck': '3.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': '3.2', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=391fdab9-d3c2-4075-9c34-177a07a21ec3 (api:52)
29804:2018-03-21 10:46:12,854+0200 INFO  (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=7e53adbc-3974-4f51-bfc8-8fbd3ca6b749 (api:46)
29805:2018-03-21 10:46:12,854+0200 INFO  (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=7e53adbc-3974-4f51-bfc8-8fbd3ca6b749 (api:52)
29806:2018-03-21 10:46:12,861+0200 INFO  (jsonrpc/0) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '10': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '12': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '15': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '14': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '1': {'cpuUser': '1.00', 'nodeIndex': 1, 'cpuSys': '1.33', 'cpuIdle': '97.67'}, '0': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '2': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.47'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '7': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.53'}, '6': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '8': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15142'}, '0': {'memPercent': 5, 'memFree': '15348'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000492277', 'lastCheck': '3.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': '3.2', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'enp3s0f0', 'tx': '1091387790', 'txDropped': '0', 'rx': '11914959714', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1451'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'ovirtmgmt', 'tx': '1048735814', 'txDropped': '0', 'rx': '11615289173', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'lo', 'tx': '58192068637', 'txDropped': '0', 'rx': '58192068637', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621960.326184, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621960.326184, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621960.326184, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76531.69', 'cpuLoad': '0.17', 'cpuSys': '0.32', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.00', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31081, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1451', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:46:12 GMT', 'cpuUser': '0.21', 'memFree': 31337, 'cpuIdle': '99.46', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.80'}} from=::ffff:192.168.0.4,49914 (api:52)
29807:2018-03-21 10:46:12,863+0200 INFO  (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)
29808:2018-03-21 10:46:13,829+0200 ERROR (jsonrpc/6) [storage.HSM] Could not connect to storageServer (hsm:2407)
29828:2018-03-21 10:46:13,829+0200 INFO  (jsonrpc/6) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 477, 'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524'}]} from=::ffff:192.168.0.4,49914, flow_id=1435fc81, task_id=0a828d2c-d9f4-4f83-a9e9-7393159d5323 (api:52)
29829:2018-03-21 10:46:13,830+0200 INFO  (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 125.04 seconds (__init__:573)
29830:2018-03-21 10:46:14,767+0200 INFO  (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=49cd2ae6-7943-47c9-ad86-9e0b7e58bca3 (api:46)
29831:2018-03-21 10:46:14,772+0200 INFO  (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=49cd2ae6-7943-47c9-ad86-9e0b7e58bca3 (api:52)
29832:2018-03-21 10:46:14,773+0200 INFO  (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573)
29833:2018-03-21 10:46:14,815+0200 INFO  (jsonrpc/2) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=19909616-efb4-490d-982c-66eda9ca4381 (api:46)
29834:2018-03-21 10:46:14,822+0200 INFO  (jsonrpc/2) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=19909616-efb4-490d-982c-66eda9ca4381 (api:52)
29835:2018-03-21 10:46:14,823+0200 INFO  (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)
29836:2018-03-21 10:46:15,360+0200 INFO  (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=da74ea9f-164b-450d-b836-6818caa3fdc5 (api:46)
29837:2018-03-21 10:46:15,360+0200 INFO  (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000492277', 'lastCheck': '5.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': '5.7', 'valid': True}} from=internal, task_id=da74ea9f-164b-450d-b836-6818caa3fdc5 (api:52)
29838:2018-03-21 10:46:15,361+0200 INFO  (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=06245e2b-e8ca-41c4-90bf-4294d0c699b8 (api:46)
29839:2018-03-21 10:46:15,361+0200 INFO  (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=06245e2b-e8ca-41c4-90bf-4294d0c699b8 (api:52)
29840:2018-03-21 10:46:20,456+0200 INFO  (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)
29841:2018-03-21 10:46:20,457+0200 INFO  (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)
29842:2018-03-21 10:46:20,457+0200 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573)
29843:2018-03-21 10:46:21,925+0200 INFO  (jsonrpc/1) [api.host] START getAllVmStats() from=::1,36114 (api:46)
29844:2018-03-21 10:46:21,926+0200 INFO  (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)
29845:2018-03-21 10:46:21,926+0200 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)
29846:2018-03-21 10:46:24,930+0200 INFO  (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=d964535d-e5eb-42bc-b324-432a05c364da (api:46)
29847:2018-03-21 10:46:24,935+0200 INFO  (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=d964535d-e5eb-42bc-b324-432a05c364da (api:52)
29848:2018-03-21 10:46:24,935+0200 INFO  (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:573)
29849:2018-03-21 10:46:24,987+0200 INFO  (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=167dcd23-04b5-4de3-9c81-7d25ba56407a (api:46)
29850:2018-03-21 10:46:25,022+0200 INFO  (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=167dcd23-04b5-4de3-9c81-7d25ba56407a (api:52)
29851:2018-03-21 10:46:25,023+0200 INFO  (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.03 seconds (__init__:573)
29852:2018-03-21 10:46:28,010+0200 INFO  (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)
29853:2018-03-21 10:46:28,011+0200 INFO  (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=011616ce-53c9-4cfc-8b29-247b1be03409 (api:46)
29854:2018-03-21 10:46:28,011+0200 INFO  (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000218955', 'lastCheck': '8.4', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000351741', 'lastCheck': '8.4', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=011616ce-53c9-4cfc-8b29-247b1be03409 (api:52)
29855:2018-03-21 10:46:28,012+0200 INFO  (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=85b42a8d-1bf1-495d-ad32-a1f9710a6468 (api:46)
29856:2018-03-21 10:46:28,012+0200 INFO  (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=85b42a8d-1bf1-495d-ad32-a1f9710a6468 (api:52)
29857:2018-03-21 10:46:28,018+0200 INFO  (jsonrpc/0) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '10': {'cpuUser': '1.73', 'nodeIndex': 0, 'cpuSys': '0.53', 'cpuIdle': '97.74'}, '13': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '12': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '15': {'cpuUser': '0.67', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.06'}, '14': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.47'}, '1': {'cpuUser': '1.26', 'nodeIndex': 1, 'cpuSys': '1.33', 'cpuIdle': '97.41'}, '0': {'cpuUser': '0.40', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.53'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.73'}, '5': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '4': {'cpuUser': '13.78', 'nodeIndex': 0, 'cpuSys': '0.80', 'cpuIdle': '85.42'}, '7': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.67', 'cpuIdle': '98.86'}, '6': {'cpuUser': '1.07', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '98.66'}, '9': {'cpuUser': '1.20', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '98.53'}, '8': {'cpuUser': '0.80', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '98.87'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15136'}, '0': {'memPercent': 5, 'memFree': '15357'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000218955', 'lastCheck': '8.4', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000351741', 'lastCheck': '8.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'enp3s0f0', 'tx': '1091397599', 'txDropped': '0', 'rx': '11914966591', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1454'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'ovirtmgmt', 'tx': '1048745347', 'txDropped': '0', 'rx': '11615294897', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'lo', 'tx': '58205583384', 'txDropped': '0', 'rx': '58205583384', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621975.346633, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621975.346633, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621975.346633, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76546.85', 'cpuLoad': '0.18', 'cpuSys': '0.39', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.20', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31031, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1454', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:46:28 GMT', 'cpuUser': '1.38', 'memFree': 31287, 'cpuIdle': '98.23', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.87'}} from=::ffff:192.168.0.4,49914 (api:52)
29858:2018-03-21 10:46:28,020+0200 INFO  (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)
29859:2018-03-21 10:46:29,667+0200 INFO  (itmap/0) [IOProcessClient] Starting client ioprocess-575 (__init__:308)
29860:2018-03-21 10:46:29,700+0200 INFO  (itmap/1) [IOProcessClient] Starting client ioprocess-576 (__init__:308)
29861:2018-03-21 10:46:29,714+0200 INFO  (ioprocess/1711202) [IOProcess] Starting ioprocess (__init__:437)
29862:2018-03-21 10:46:29,725+0200 INFO  (ioprocess/1711208) [IOProcess] Starting ioprocess (__init__:437)
29863:2018-03-21 10:46:30,375+0200 INFO  (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=bdacf1c0-3fcb-43b1-82e2-a041a26d0d7f (api:46)
29864:2018-03-21 10:46:30,376+0200 INFO  (periodic/0) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00031111', 'lastCheck': '0.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00045594', 'lastCheck': '0.6', 'valid': True}} from=internal, task_id=bdacf1c0-3fcb-43b1-82e2-a041a26d0d7f (api:52)
29865:2018-03-21 10:46:30,376+0200 INFO  (periodic/0) [vdsm.api] START multipath_health() from=internal, task_id=3d3cad70-e5c8-49b2-9f5d-92a249cc102d (api:46)
29866:2018-03-21 10:46:30,377+0200 INFO  (periodic/0) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=3d3cad70-e5c8-49b2-9f5d-92a249cc102d (api:52)
29867:2018-03-21 10:46:35,108+0200 INFO  (jsonrpc/6) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=304d7e7c-03fc-44d6-be32-d083ada09b30 (api:46)
29868:2018-03-21 10:46:35,114+0200 INFO  (jsonrpc/6) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=304d7e7c-03fc-44d6-be32-d083ada09b30 (api:52)
29869:2018-03-21 10:46:35,114+0200 INFO  (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:573)
29870:2018-03-21 10:46:35,170+0200 INFO  (jsonrpc/4) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=76736a93-d4c4-4b2f-9848-7b8731fe2b67 (api:46)
29871:2018-03-21 10:46:35,176+0200 INFO  (jsonrpc/4) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=76736a93-d4c4-4b2f-9848-7b8731fe2b67 (api:52)
29872:2018-03-21 10:46:35,177+0200 INFO  (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)
29873:2018-03-21 10:46:35,462+0200 INFO  (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)
29874:2018-03-21 10:46:35,462+0200 INFO  (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)
29875:2018-03-21 10:46:35,463+0200 INFO  (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)
29876:2018-03-21 10:46:36,948+0200 INFO  (jsonrpc/7) [api.host] START getAllVmStats() from=::1,36114 (api:46)
29877:2018-03-21 10:46:36,948+0200 INFO  (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)
29878:2018-03-21 10:46:36,949+0200 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)
29879:2018-03-21 10:46:43,179+0200 INFO  (jsonrpc/1) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)
29880:2018-03-21 10:46:43,180+0200 INFO  (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=83bbc95c-6d59-4eec-82b1-e8993be28759 (api:46)
29881:2018-03-21 10:46:43,181+0200 INFO  (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000486101', 'lastCheck': '3.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': '3.4', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=83bbc95c-6d59-4eec-82b1-e8993be28759 (api:52)
29882:2018-03-21 10:46:43,181+0200 INFO  (jsonrpc/1) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=9b781324-b50b-49a9-bf31-5eca835dc75c (api:46)
29883:2018-03-21 10:46:43,181+0200 INFO  (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=9b781324-b50b-49a9-bf31-5eca835dc75c (api:52)
29884:2018-03-21 10:46:43,188+0200 INFO  (jsonrpc/1) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.40', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.33'}, '10': {'cpuUser': '0.60', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '98.93'}, '13': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.73'}, '12': {'cpuUser': '1.46', 'nodeIndex': 0, 'cpuSys': '0.60', 'cpuIdle': '97.94'}, '15': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '99.47'}, '14': {'cpuUser': '0.40', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '99.13'}, '1': {'cpuUser': '1.33', 'nodeIndex': 1, 'cpuSys': '1.93', 'cpuIdle': '96.74'}, '0': {'cpuUser': '0.60', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.13'}, '3': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.40'}, '2': {'cpuUser': '6.99', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '92.61'}, '5': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '4': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.46'}, '7': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.60', 'cpuIdle': '98.93'}, '6': {'cpuUser': '0.53', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.20'}, '9': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '99.20'}, '8': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15131'}, '0': {'memPercent': 5, 'memFree': '15357'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000486101', 'lastCheck': '3.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': '3.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'enp3s0f0', 'tx': '1091404888', 'txDropped': '0', 'rx': '11914970667', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1455'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'ovirtmgmt', 'tx': '1048752406', 'txDropped': '0', 'rx': '11615298347', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'lo', 'tx': '58215969455', 'txDropped': '0', 'rx': '58215969455', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621990.363726, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621990.363726, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621990.363726, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '206', 'ksmPages': 100, 'elapsedTime': '76562.02', 'cpuLoad': '0.18', 'cpuSys': '0.45', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.27', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1455', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:46:43 GMT', 'cpuUser': '0.87', 'memFree': 31336, 'cpuIdle': '98.68', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '1.20'}} from=::ffff:192.168.0.4,49914 (api:52)
29885:2018-03-21 10:46:43,190+0200 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)
29886:2018-03-21 10:46:45,288+0200 INFO  (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=ffe57349-2532-4b64-b32c-8a558273d8e5 (api:46)
29887:2018-03-21 10:46:45,292+0200 INFO  (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=ffe57349-2532-4b64-b32c-8a558273d8e5 (api:52)
29888:2018-03-21 10:46:45,293+0200 INFO  (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:573)
29889:2018-03-21 10:46:45,298+0200 INFO  (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=b2ea28f0-abf0-4258-8152-6035d93aa5dc (api:46)
29890:2018-03-21 10:46:45,305+0200 INFO  (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=b2ea28f0-abf0-4258-8152-6035d93aa5dc (api:52)
29891:2018-03-21 10:46:45,306+0200 INFO  (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:573)
29892:2018-03-21 10:46:45,395+0200 INFO  (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=4010bf72-5c86-4daf-a8f9-12ec229a86e4 (api:46)
29893:2018-03-21 10:46:45,395+0200 INFO  (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000486101', 'lastCheck': '5.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': '5.7', 'valid': True}} from=internal, task_id=4010bf72-5c86-4daf-a8f9-12ec229a86e4 (api:52)
29894:2018-03-21 10:46:45,396+0200 INFO  (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=37a1132f-452b-4805-9cc0-b8e74e8d6b02 (api:46)
29895:2018-03-21 10:46:45,396+0200 INFO  (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=37a1132f-452b-4805-9cc0-b8e74e8d6b02 (api:52)
29896:2018-03-21 10:46:50,467+0200 INFO  (jsonrpc/0) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46)
29897:2018-03-21 10:46:50,468+0200 INFO  (jsonrpc/0) [throttled] Current getAllVmStats: {} (throttledlog:103)
29898:2018-03-21 10:46:50,468+0200 INFO  (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52)
29899:2018-03-21 10:46:50,468+0200 INFO  (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573)
29900:2018-03-21 10:46:51,967+0200 INFO  (jsonrpc/6) [api.host] START getAllVmStats() from=::1,36114 (api:46)
29901:2018-03-21 10:46:51,968+0200 INFO  (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52)
29902:2018-03-21 10:46:51,968+0200 INFO  (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)
29903:2018-03-21 10:46:55,648+0200 INFO  (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=a975635f-59d7-4f40-b9a7-a5ea7150368e (api:46)
29904:2018-03-21 10:46:55,653+0200 INFO  (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=a975635f-59d7-4f40-b9a7-a5ea7150368e (api:52)
29905:2018-03-21 10:46:55,653+0200 INFO  (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:573)
29906:2018-03-21 10:46:55,659+0200 INFO  (jsonrpc/2) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=97d4ae6c-3719-4149-85fd-28e924ad8883 (api:46)
29907:2018-03-21 10:46:55,666+0200 INFO  (jsonrpc/2) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=97d4ae6c-3719-4149-85fd-28e924ad8883 (api:52)
29908:2018-03-21 10:46:55,667+0200 INFO  (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:573)
29909:2018-03-21 10:46:59,473+0200 INFO  (jsonrpc/7) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46)
29910:2018-03-21 10:46:59,474+0200 INFO  (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=177b551b-9bd3-45a4-84d0-5244630c08e5 (api:46)
29911:2018-03-21 10:46:59,474+0200 INFO  (jsonrpc/7) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470906', 'lastCheck': '9.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000334928', 'lastCheck': '9.7', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=177b551b-9bd3-45a4-84d0-5244630c08e5 (api:52)
29912:2018-03-21 10:46:59,475+0200 INFO  (jsonrpc/7) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=30753d5c-9bfd-4b0f-ac83-e32278d82b63 (api:46)
29913:2018-03-21 10:46:59,475+0200 INFO  (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=30753d5c-9bfd-4b0f-ac83-e32278d82b63 (api:52)
29914:2018-03-21 10:46:59,482+0200 INFO  (jsonrpc/7) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '10': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '15': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '1': {'cpuUser': '1.07', 'nodeIndex': 1, 'cpuSys': '1.07', 'cpuIdle': '97.86'}, '0': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '4': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.26'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.47', 'cpuIdle': '99.53'}, '6': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.73'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '8': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15125'}, '0': {'memPercent': 5, 'memFree': '15364'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470906', 'lastCheck': '9.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000334928', 'lastCheck': '9.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'enp3s0f0', 'tx': '1091412332', 'txDropped': '0', 'rx': '11914976557', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1456'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'ovirtmgmt', 'tx': '1048761326', 'txDropped': '0', 'rx': '11615304242', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'lo', 'tx': '58228460774', 'txDropped': '0', 'rx': '58228460774', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521622005.383289, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521622005.383289, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521622005.383289, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '208', 'ksmPages': 100, 'elapsedTime': '76578.31', 'cpuLoad': '0.17', 'cpuSys': '0.20', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.13', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31078, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1456', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:46:59 GMT', 'cpuUser': '0.15', 'memFree': 31334, 'cpuIdle': '99.65', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.60'}} from=::ffff:192.168.0.4,49914 (api:52)
29915:2018-03-21 10:46:59,484+0200 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)


On 21 Mar 2018, at 11:33, Fred Rolland <frolland@redhat.com> wrote:

Can you provide the vdsm logs from the host.
It looks the vdsm failed to connect to the server.

On Wed, Mar 21, 2018 at 11:21 AM, Andrei Verovski <andreil1@starlett.lv> wrote:
Hi,

I have 2-host oVirt setup with 2 Data Centers, one with local storage domain (DC #1) for VMs + Export domain on NFS, another with all NFS shared (DC #2).
Trying to export VMs from DC #1 to DC #2.
VMs are exported to DC #1 export domain (NFS), then domain put into maintenance mode and detached from DC #1.

Unfortunately, attaching it to DC #2 failed. Logs attached. Tried to run this command twice.
Workaround are possible in order to accomplish this task, yet it would be better to do in a way as it was designed.
Thanks.


2018-03-21 10:43:24,024+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-128904) [570ec5d9-fff5-4656-afbd-90b3207a616e] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz)
2018-03-21 10:46:16,512+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-128956) [1435fc81] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Failed to connect Host node11 to the Storage Domains node10-NFS-EXPORTS.
2018-03-21 10:46:16,651+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc321609b] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz)


tail -n 1000 engine.log | grep 570ec5d9-fff5-4656-afbd-90b3207a616e
2018-03-21 10:41:14,643+02 INFO  [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-2) [570ec5d9-fff5-4656-afbd-90b3207a616e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[1d7208ce-d3a1-4406-9638-fe7051562994=STORAGE]', sharedLocks=''}'
2018-03-21 10:41:16,129+02 INFO  [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (EE-ManagedThreadFactory-engine-Thread-128904) [570ec5d9-fff5-4656-afbd-90b3207a616e] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected :  ID: 1d7208ce-d3a1-4406-9638-fe7051562994 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN,  ID: 80cc922f-8dea-4fed-b951-1060ba116ad5 Type: StoragePoolAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN
2018-03-21 10:43:23,564+02 ERROR [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (EE-ManagedThreadFactory-engine-Thread-128904) [570ec5d9-fff5-4656-afbd-90b3207a616e] Cannot connect storage connection server, aborting attach storage domain operation.
2018-03-21 10:43:23,567+02 INFO  [org.ovirt.engine.core.bll.CommandCompensator] (EE-ManagedThreadFactory-engine-Thread-128904) [570ec5d9-fff5-4656-afbd-90b3207a616e] Command [id=921ca7cd-4f93-46aa-8de2-91b13b8f96cb]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: StoragePoolIsoMapId:{storagePoolId='80cc922f-8dea-4fed-b951-1060ba116ad5', storageId='1d7208ce-d3a1-4406-9638-fe7051562994'}.
2018-03-21 10:43:24,024+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-128904) [570ec5d9-fff5-4656-afbd-90b3207a616e] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz)
2018-03-21 10:43:24,114+02 INFO  [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (EE-ManagedThreadFactory-engine-Thread-128904) [570ec5d9-fff5-4656-afbd-90b3207a616e] Lock freed to object 'EngineLock:{exclusiveLocks='[1d7208ce-d3a1-4406-9638-fe7051562994=STORAGE]', sharedLocks=''}'



[root@node00 ovirt-engine]# tail -n 1000 engine.log | grep a81ffa4a-5a58-41a0-888a-f0edc321609b
2018-03-21 10:44:11,025+02 INFO  [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-16) [a81ffa4a-5a58-41a0-888a-f0edc321609b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[1d7208ce-d3a1-4406-9638-fe7051562994=STORAGE]', sharedLocks=''}'
2018-03-21 10:44:11,236+02 INFO  [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (EE-ManagedThreadFactory-engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc321609b] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected :  ID: 1d7208ce-d3a1-4406-9638-fe7051562994 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN,  ID: 80cc922f-8dea-4fed-b951-1060ba116ad5 Type: StoragePoolAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN
2018-03-21 10:46:16,567+02 ERROR [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (EE-ManagedThreadFactory-engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc321609b] Cannot connect storage connection server, aborting attach storage domain operation.
2018-03-21 10:46:16,568+02 INFO  [org.ovirt.engine.core.bll.CommandCompensator] (EE-ManagedThreadFactory-engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc321609b] Command [id=b5c25100-1a8a-4db0-9509-99cfa60995b1]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: StoragePoolIsoMapId:{storagePoolId='80cc922f-8dea-4fed-b951-1060ba116ad5', storageId='1d7208ce-d3a1-4406-9638-fe7051562994'}.
2018-03-21 10:46:16,651+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc321609b] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz)
2018-03-21 10:46:16,681+02 INFO  [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (EE-ManagedThreadFactory-engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc321609b] Lock freed to object 'EngineLock:{exclusiveLocks='[1d7208ce-d3a1-4406-9638-fe7051562994=STORAGE]', sharedLocks='’}'


_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users