--Apple-Mail=_C5052CF0-37B9-4927-AE48-DF656F09C5B6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi, Errors occurred at 10.43 AM and 10.46 AM node00.starlett.lv - 192.168.0.4 - oVirt host engine (separate PC, not = hosted) node10.starlett.lv - 192.168.0.5 - host #1 of DC #1, export domain from = which was detached from DC #1 node11.starlett.lv - 192.168.0.6 - host #1 of DC #2,=20 Logs from DC#2 node11, to which I=E2=80=99m trying to attach export = domain located at NFS share node10.
grep 570ec5d9-fff5-4656-afbd-90b3207a616e within vdsm.log returned nothing, so I did
On 21 Mar 2018, at 11:33, Fred Rolland <frolland@redhat.com> wrote: =20 Can you provide the vdsm logs from the host. It looks the vdsm failed to connect to the server. =20 On Wed, Mar 21, 2018 at 11:21 AM, Andrei Verovski = <andreil1@starlett.lv <mailto:andreil1@starlett.lv>> wrote: Hi, =20 I have 2-host oVirt setup with 2 Data Centers, one with local storage = domain (DC #1) for VMs + Export domain on NFS, another with all NFS = shared (DC #2). Trying to export VMs from DC #1 to DC #2. VMs are exported to DC #1 export domain (NFS), then domain put into =
=20 Unfortunately, attaching it to DC #2 failed. Logs attached. Tried to = run this command twice. Workaround are possible in order to accomplish this task, yet it would = be better to do in a way as it was designed. Thanks. =20 =20 2018-03-21 10:43:24,024+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage = Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: = admin@internal-authz) 2018-03-21 10:46:16,512+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128956) [1435fc81] EVENT_ID: = VDS_STORAGES_CONNECTION_FAILED(188), Failed to connect Host node11 to =
grep -n 10:43 vdsm.log | tail -1000 1011:2018-03-21 06:10:43,077+0200 INFO (jsonrpc/3) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 1012:2018-03-21 06:10:43,077+0200 INFO (jsonrpc/3) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 1013:2018-03-21 06:10:43,077+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 = seconds (__init__:573) 1014:2018-03-21 06:10:43,868+0200 INFO (jsonrpc/7) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 1015:2018-03-21 06:10:43,868+0200 INFO (jsonrpc/7) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 1016:2018-03-21 06:10:43,868+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 19729:2018-03-21 09:10:43,641+0200 INFO (jsonrpc/5) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 19730:2018-03-21 09:10:43,641+0200 INFO (jsonrpc/5) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 19731:2018-03-21 09:10:43,642+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29302:2018-03-21 10:43:00,085+0200 INFO (periodic/2) [vdsm.api] START = repoStats(domains=3D()) from=3Dinternal, = task_id=3D7b690c8a-5470-44da-a8f3-7e7e9b018e88 (api:46) 29303:2018-03-21 10:43:00,085+0200 INFO (periodic/2) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000477038', = 'lastCheck': '0.5', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': = '0.4', 'valid': True}} from=3Dinternal, = task_id=3D7b690c8a-5470-44da-a8f3-7e7e9b018e88 (api:52) 29304:2018-03-21 10:43:00,086+0200 INFO (periodic/2) [vdsm.api] START = multipath_health() from=3Dinternal, = task_id=3D9a625d43-a03c-429c-856f-9aa8e5ff65b5 (api:46) 29305:2018-03-21 10:43:00,086+0200 INFO (periodic/2) [vdsm.api] FINISH = multipath_health return=3D{} from=3Dinternal, = task_id=3D9a625d43-a03c-429c-856f-9aa8e5ff65b5 (api:52) 29306:2018-03-21 10:43:05,382+0200 INFO (jsonrpc/5) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29307:2018-03-21 10:43:05,383+0200 INFO (jsonrpc/5) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 29308:2018-03-21 10:43:05,383+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29309:2018-03-21 10:43:06,248+0200 INFO (jsonrpc/3) [api.host] START = getStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29310:2018-03-21 10:43:06,249+0200 INFO (jsonrpc/3) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3Df80f9bb8-2e84-4f45-ab58-09a88e808cf3 (api:46) 29311:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000477038', = 'lastCheck': '6.7', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': = '6.6', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3Df80f9bb8-2e84-4f45-ab58-09a88e808cf3 (api:52) 29312:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) [vdsm.api] START = multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3Dddd7064d-cd12-43bf-a177-b2a93cc9035e (api:46) 29313:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) [vdsm.api] FINISH = multipath_health return=3D{} from=3D::ffff:192.168.0.4,49914, = task_id=3Dddd7064d-cd12-43bf-a177-b2a93cc9035e (api:52) 29314:2018-03-21 10:43:06,256+0200 INFO (jsonrpc/3) [api.host] FINISH = getStats return=3D{'status': {'message': 'Done', 'code': 0}, 'info': = {'cpuStatistics': {'11': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': = '0.27', 'cpuIdle': '99.60'}, '10': {'cpuUser': '0.13', 'nodeIndex': 0, = 'cpuSys': '0.40', 'cpuIdle': '99.47'}, '13': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '12': {'cpuUser': = '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '15': = {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.47', 'cpuIdle': = '99.33'}, '14': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.27', = 'cpuIdle': '99.60'}, '1': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': = '1.60', 'cpuIdle': '97.27'}, '0': {'cpuUser': '0.47', 'nodeIndex': 0, = 'cpuSys': '0.27', 'cpuIdle': '99.26'}, '3': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '2': {'cpuUser': = '0.33', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.54'}, '5': = {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.74'}, '4': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', = 'cpuIdle': '99.67'}, '7': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': = '0.60', 'cpuIdle': '99.27'}, '6': {'cpuUser': '0.13', 'nodeIndex': 0, = 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '9': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '8': {'cpuUser': = '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}}, = 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15142'}, '0': = {'memPercent': 5, 'memFree': '15349'}}, 'memShared': 0, 'thpState': = 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', = 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000477038', = 'lastCheck': '6.7', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': = '6.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': = {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621780.072418, 'name': 'enp3s0f0', 'tx': '1091287978', 'txDropped': = '0', 'rx': '11914868563', 'rxErrors': '0', 'speed': '100', 'rxDropped': = '1439'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621780.072418, 'name': 'ovirtmgmt', 'tx': '1048639112', 'txDropped': = '0', 'rx': '11615209092', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621780.072418, 'name': 'lo', 'tx': '58057075945', 'txDropped': '0', = 'rx': '58057075945', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621780.072418, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621780.072418, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621780.072418, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621780.072418, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, = 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, = 'elapsedTime': '76345.09', 'cpuLoad': '0.16', 'cpuSys': '0.36', = 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': = '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31081, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1439', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, = 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, = 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:43:06 GMT', 'cpuUser': '0.21', = 'memFree': 31337, 'cpuIdle': '99.43', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.93'}} from=3D::ffff:192.168.0.4,49914 (api:52) 29315:2018-03-21 10:43:06,258+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573) 29316:2018-03-21 10:43:06,632+0200 INFO (jsonrpc/0) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 29317:2018-03-21 10:43:06,632+0200 INFO (jsonrpc/0) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 29318:2018-03-21 10:43:06,633+0200 INFO (jsonrpc/0) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29319:2018-03-21 10:43:07,415+0200 INFO (jsonrpc/4) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D276c13c7-61ec-4782-a0d9-5cd0f0b0a729 (api:46) 29320:2018-03-21 10:43:07,420+0200 INFO (jsonrpc/4) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D276c13c7-61ec-4782-a0d9-5cd0f0b0a729 (api:52) 29321:2018-03-21 10:43:07,420+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573) 29322:2018-03-21 10:43:07,427+0200 INFO (jsonrpc/6) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D344b18df-a428-429a-b49f-381f901d1618 (api:46) 29323:2018-03-21 10:43:07,434+0200 INFO (jsonrpc/6) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D344b18df-a428-429a-b49f-381f901d1618 (api:52) 29324:2018-03-21 10:43:07,435+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573) 29325:2018-03-21 10:43:13,839+0200 WARN (vdsm.Scheduler) [Executor] = Worker blocked: <Worker name=3Djsonrpc/1 running <Task <JsonRpcTask = {'params': {u'connectionParams': [{u'id': = u'461f65a9-3a81-4f3f-a46d-c5ed12520524', u'connection': = u'node10.starlett.lv:/vmdata/nfs/exports', u'iqn': u'', u'user': u'', = u'tpgt': u'1', u'protocol_version': u'auto', u'password': '********', = u'port': u''}], u'storagepoolID': = u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': = '2.0', 'method': u'StoragePool.connectStorageServer', 'id': = u'5f98dfd5-998f-44a6-8594-e5f40144b0ed'} at 0x4076150> timeout=3D60, = duration=3D120 at 0x402fe50> task#=3D3754 at 0x353e690>, traceback: 29376:2018-03-21 10:43:15,104+0200 INFO (periodic/2) [vdsm.api] START = repoStats(domains=3D()) from=3Dinternal, = task_id=3D672a883e-1d60-4981-9eee-bdcac2ec30c1 (api:46) 29377:2018-03-21 10:43:15,105+0200 INFO (periodic/2) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000488777', = 'lastCheck': '5.6', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000481962', 'lastCheck': = '5.4', 'valid': True}} from=3Dinternal, = task_id=3D672a883e-1d60-4981-9eee-bdcac2ec30c1 (api:52) 29378:2018-03-21 10:43:15,105+0200 INFO (periodic/2) [vdsm.api] START = multipath_health() from=3Dinternal, = task_id=3D55f1b797-da2a-4b07-affe-5b9e41a74307 (api:46) 29379:2018-03-21 10:43:15,106+0200 INFO (periodic/2) [vdsm.api] FINISH = multipath_health return=3D{} from=3Dinternal, = task_id=3D55f1b797-da2a-4b07-affe-5b9e41a74307 (api:52) 29380:2018-03-21 10:43:17,645+0200 INFO (jsonrpc/7) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D52a6ddeb-3c20-4f3f-b27d-94d19580b473 (api:46) 29381:2018-03-21 10:43:17,649+0200 INFO (jsonrpc/7) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D52a6ddeb-3c20-4f3f-b27d-94d19580b473 (api:52) 29382:2018-03-21 10:43:17,650+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573) 29383:2018-03-21 10:43:17,656+0200 INFO (jsonrpc/2) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D1a103883-6a6a-4d61-ade6-96559591ddb0 (api:46) 29384:2018-03-21 10:43:17,663+0200 INFO (jsonrpc/2) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D1a103883-6a6a-4d61-ade6-96559591ddb0 (api:52) 29385:2018-03-21 10:43:17,663+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573) 29386:2018-03-21 10:43:18,882+0200 ERROR (jsonrpc/1) [storage.HSM] Could = not connect to storageServer (hsm:2407) 29406:2018-03-21 10:43:18,883+0200 INFO (jsonrpc/1) [vdsm.api] FINISH = connectStorageServer return=3D{'statuslist': [{'status': 477, 'id': = u'461f65a9-3a81-4f3f-a46d-c5ed12520524'}]} = from=3D::ffff:192.168.0.4,49914, flow_id=3D4a53b512, = task_id=3De598cbe0-cde8-4c73-b526-4398df05e67f (api:52) 29407:2018-03-21 10:43:18,883+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer = succeeded in 125.04 seconds (__init__:573) 29408:2018-03-21 10:43:20,387+0200 INFO (jsonrpc/5) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29409:2018-03-21 10:43:20,388+0200 INFO (jsonrpc/5) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 29410:2018-03-21 10:43:20,388+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 = seconds (__init__:573) 29411:2018-03-21 10:43:21,497+0200 INFO (jsonrpc/3) [api.host] START = getStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29412:2018-03-21 10:43:21,498+0200 INFO (jsonrpc/3) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3Da7469c1f-3d6d-42e7-8798-5c5335da4a9b (api:46) 29413:2018-03-21 10:43:21,498+0200 INFO (jsonrpc/3) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000445096', = 'lastCheck': '1.9', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000296586', 'lastCheck': = '1.8', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3Da7469c1f-3d6d-42e7-8798-5c5335da4a9b (api:52) 29414:2018-03-21 10:43:21,499+0200 INFO (jsonrpc/3) [vdsm.api] START = multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D9fa5dae3-e2bd-4186-9a1d-afb0aa6e2511 (api:46) 29415:2018-03-21 10:43:21,499+0200 INFO (jsonrpc/3) [vdsm.api] FINISH = multipath_health return=3D{} from=3D::ffff:192.168.0.4,49914, = task_id=3D9fa5dae3-e2bd-4186-9a1d-afb0aa6e2511 (api:52) 29416:2018-03-21 10:43:21,505+0200 INFO (jsonrpc/3) [api.host] FINISH = getStats return=3D{'status': {'message': 'Done', 'code': 0}, 'info': = {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.07', 'cpuIdle': '99.86'}, '10': {'cpuUser': '0.07', 'nodeIndex': 0, = 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '13': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '12': {'cpuUser': = '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '15': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.87'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.33', = 'cpuIdle': '99.60'}, '1': {'cpuUser': '1.00', 'nodeIndex': 1, 'cpuSys': = '1.00', 'cpuIdle': '98.00'}, '0': {'cpuUser': '0.07', 'nodeIndex': 0, = 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '3': {'cpuUser': '0.00', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '2': {'cpuUser': = '0.27', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.66'}, '5': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.86'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', = 'cpuIdle': '99.86'}, '7': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.27', 'cpuIdle': '99.66'}, '6': {'cpuUser': '0.07', 'nodeIndex': 0, = 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '9': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '8': {'cpuUser': = '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}}, = 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15142'}, '0': = {'memPercent': 5, 'memFree': '15351'}}, 'memShared': 0, 'thpState': = 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', = 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000445096', = 'lastCheck': '1.9', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000296586', 'lastCheck': = '1.8', 'valid': True}}, 'incomingVmMigrations': 0, 'network': = {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621795.091263, 'name': 'enp3s0f0', 'tx': '1091295628', 'txDropped': = '0', 'rx': '11914875210', 'rxErrors': '0', 'speed': '100', 'rxDropped': = '1442'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621795.091263, 'name': 'ovirtmgmt', 'tx': '1048646480', 'txDropped': = '0', 'rx': '11615214750', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621795.091263, 'name': 'lo', 'tx': '58066397668', 'txDropped': '0', = 'rx': '58066397668', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621795.091263, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621795.091263, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621795.091263, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621795.091263, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, = 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, = 'elapsedTime': '76360.34', 'cpuLoad': '0.15', 'cpuSys': '0.18', = 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': = '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31031, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1442', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, = 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, = 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:43:21 GMT', 'cpuUser': '0.15', = 'memFree': 31287, 'cpuIdle': '99.67', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.67'}} from=3D::ffff:192.168.0.4,49914 (api:52) 29417:2018-03-21 10:43:21,508+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.02 seconds = (__init__:573) 29418:2018-03-21 10:43:21,651+0200 INFO (jsonrpc/0) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 29419:2018-03-21 10:43:21,651+0200 INFO (jsonrpc/0) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 29420:2018-03-21 10:43:21,652+0200 INFO (jsonrpc/0) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29421:2018-03-21 10:43:27,764+0200 INFO (jsonrpc/4) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D46d19cab-8903-4c52-9bb2-4dd8f370997e (api:46) 29422:2018-03-21 10:43:27,769+0200 INFO (jsonrpc/4) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D46d19cab-8903-4c52-9bb2-4dd8f370997e (api:52) 29423:2018-03-21 10:43:27,770+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573) 29424:2018-03-21 10:43:27,815+0200 INFO (jsonrpc/6) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D290acc73-8f25-4fd7-9263-7bd05d46430a (api:46) 29425:2018-03-21 10:43:27,822+0200 INFO (jsonrpc/6) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D290acc73-8f25-4fd7-9263-7bd05d46430a (api:52) 29426:2018-03-21 10:43:27,823+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573) 29427:2018-03-21 10:43:30,121+0200 INFO (periodic/3) [vdsm.api] START = repoStats(domains=3D()) from=3Dinternal, = task_id=3D0abdebcf-2b0b-4d33-b552-c0f2ed3e567b (api:46) 29428:2018-03-21 10:43:30,122+0200 INFO (periodic/3) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000402077', = 'lastCheck': '0.6', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': = '0.5', 'valid': True}} from=3Dinternal, = task_id=3D0abdebcf-2b0b-4d33-b552-c0f2ed3e567b (api:52) 29429:2018-03-21 10:43:30,122+0200 INFO (periodic/3) [vdsm.api] START = multipath_health() from=3Dinternal, = task_id=3Dacde7e7d-ab4a-4811-934f-4d312da33208 (api:46) 29430:2018-03-21 10:43:30,123+0200 INFO (periodic/3) [vdsm.api] FINISH = multipath_health return=3D{} from=3Dinternal, = task_id=3Dacde7e7d-ab4a-4811-934f-4d312da33208 (api:52) 29431:2018-03-21 10:43:35,393+0200 INFO (jsonrpc/7) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29432:2018-03-21 10:43:35,394+0200 INFO (jsonrpc/7) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 29433:2018-03-21 10:43:35,394+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29434:2018-03-21 10:43:36,671+0200 INFO (jsonrpc/2) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 29435:2018-03-21 10:43:36,671+0200 INFO (jsonrpc/2) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 29436:2018-03-21 10:43:36,672+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29437:2018-03-21 10:43:37,356+0200 INFO (jsonrpc/1) [api.host] START = getStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29438:2018-03-21 10:43:37,357+0200 INFO (jsonrpc/1) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3Dbc14e253-d3d4-4cc3-9ae1-c0e6ac6e37f7 (api:46) 29439:2018-03-21 10:43:37,357+0200 INFO (jsonrpc/1) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000402077', = 'lastCheck': '7.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': = '7.7', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3Dbc14e253-d3d4-4cc3-9ae1-c0e6ac6e37f7 (api:52) 29440:2018-03-21 10:43:37,358+0200 INFO (jsonrpc/1) [vdsm.api] START = multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D23c698b8-d5bd-4746-b738-97b10f3e4bfb (api:46) 29441:2018-03-21 10:43:37,358+0200 INFO (jsonrpc/1) [vdsm.api] FINISH = multipath_health return=3D{} from=3D::ffff:192.168.0.4,49914, = task_id=3D23c698b8-d5bd-4746-b738-97b10f3e4bfb (api:52) 29442:2018-03-21 10:43:37,365+0200 INFO (jsonrpc/1) [api.host] FINISH = getStats return=3D{'status': {'message': 'Done', 'code': 0}, 'info': = {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.07', 'cpuIdle': '99.86'}, '10': {'cpuUser': '3.06', 'nodeIndex': 0, = 'cpuSys': '0.80', 'cpuIdle': '96.14'}, '13': {'cpuUser': '0.00', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '12': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '15': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.87'}, '14': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.40', = 'cpuIdle': '99.33'}, '1': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': = '1.33', 'cpuIdle': '97.54'}, '0': {'cpuUser': '0.47', 'nodeIndex': 0, = 'cpuSys': '0.33', 'cpuIdle': '99.20'}, '3': {'cpuUser': '0.00', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '2': {'cpuUser': = '0.67', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '98.93'}, '5': = {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.74'}, '4': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.40', = 'cpuIdle': '99.13'}, '7': {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': = '0.73', 'cpuIdle': '98.94'}, '6': {'cpuUser': '5.39', 'nodeIndex': 0, = 'cpuSys': '0.47', 'cpuIdle': '94.14'}, '9': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '8': {'cpuUser': = '0.53', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.07'}}, = 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15143'}, '0': = {'memPercent': 5, 'memFree': '15348'}}, 'memShared': 0, 'thpState': = 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', = 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000402077', = 'lastCheck': '7.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': = '7.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': = {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621810.108511, 'name': 'enp3s0f0', 'tx': '1091305295', 'txDropped': = '0', 'rx': '11914880953', 'rxErrors': '0', 'speed': '100', 'rxDropped': = '1442'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621810.108511, 'name': 'ovirtmgmt', 'tx': '1048655897', 'txDropped': = '0', 'rx': '11615220341', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621810.108511, 'name': 'lo', 'tx': '58078891247', 'txDropped': '0', = 'rx': '58078891247', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621810.108511, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621810.108511, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621810.108511, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621810.108511, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, = 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, = 'elapsedTime': '76376.19', 'cpuLoad': '0.16', 'cpuSys': '0.39', = 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': = '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.00', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1442', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, = 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, = 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:43:37 GMT', 'cpuUser': '0.79', = 'memFree': 31336, 'cpuIdle': '98.82', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.73'}} from=3D::ffff:192.168.0.4,49914 (api:52) 29443:2018-03-21 10:43:37,367+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573) 29444:2018-03-21 10:43:38,124+0200 INFO (jsonrpc/5) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D9d2a82dd-c76c-4eb5-a1e1-b49bf004384e (api:46) 29445:2018-03-21 10:43:38,128+0200 INFO (jsonrpc/5) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D9d2a82dd-c76c-4eb5-a1e1-b49bf004384e (api:52) 29446:2018-03-21 10:43:38,129+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573) 29447:2018-03-21 10:43:38,171+0200 INFO (jsonrpc/3) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D62d44b0b-20b2-47ae-bfb8-d4717e301b69 (api:46) 29448:2018-03-21 10:43:38,177+0200 INFO (jsonrpc/3) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D62d44b0b-20b2-47ae-bfb8-d4717e301b69 (api:52) 29449:2018-03-21 10:43:38,178+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573) 29450:2018-03-21 10:43:45,147+0200 INFO (periodic/2) [vdsm.api] START = repoStats(domains=3D()) from=3Dinternal, = task_id=3D49c33285-a222-464d-8984-4e75a5c6354a (api:46) 29451:2018-03-21 10:43:45,148+0200 INFO (periodic/2) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000454102', = 'lastCheck': '5.6', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000347381', 'lastCheck': = '5.5', 'valid': True}} from=3Dinternal, = task_id=3D49c33285-a222-464d-8984-4e75a5c6354a (api:52) 29452:2018-03-21 10:43:45,148+0200 INFO (periodic/2) [vdsm.api] START = multipath_health() from=3Dinternal, = task_id=3Deed6b564-76b1-4f9b-80a2-1f0f6d9bbe32 (api:46) 29453:2018-03-21 10:43:45,149+0200 INFO (periodic/2) [vdsm.api] FINISH = multipath_health return=3D{} from=3Dinternal, = task_id=3Deed6b564-76b1-4f9b-80a2-1f0f6d9bbe32 (api:52) 29454:2018-03-21 10:43:48,405+0200 INFO (jsonrpc/0) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D84651a37-7d13-41a9-9dcf-2169a9c46fc5 (api:46) 29455:2018-03-21 10:43:48,410+0200 INFO (jsonrpc/0) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D84651a37-7d13-41a9-9dcf-2169a9c46fc5 (api:52) 29456:2018-03-21 10:43:48,411+0200 INFO (jsonrpc/0) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573) 29457:2018-03-21 10:43:48,417+0200 INFO (jsonrpc/4) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D4ebd1d1a-11ac-41ee-b459-7e45f008776b (api:46) 29458:2018-03-21 10:43:48,423+0200 INFO (jsonrpc/4) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D4ebd1d1a-11ac-41ee-b459-7e45f008776b (api:52) 29459:2018-03-21 10:43:48,424+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573) 29460:2018-03-21 10:43:50,399+0200 INFO (jsonrpc/6) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29461:2018-03-21 10:43:50,400+0200 INFO (jsonrpc/6) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 29462:2018-03-21 10:43:50,400+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29463:2018-03-21 10:43:51,694+0200 INFO (jsonrpc/7) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 29464:2018-03-21 10:43:51,694+0200 INFO (jsonrpc/7) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 29465:2018-03-21 10:43:51,695+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29466:2018-03-21 10:43:52,516+0200 INFO (jsonrpc/2) [api.host] START = getStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29467:2018-03-21 10:43:52,517+0200 INFO (jsonrpc/2) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D0a88f0dc-eccf-4d24-9ac2-22e079a1480a (api:46) 29468:2018-03-21 10:43:52,517+0200 INFO (jsonrpc/2) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000487937', = 'lastCheck': '3.0', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000455956', 'lastCheck': = '2.9', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D0a88f0dc-eccf-4d24-9ac2-22e079a1480a (api:52) 29469:2018-03-21 10:43:52,518+0200 INFO (jsonrpc/2) [vdsm.api] START = multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3Da7a44ff7-a137-41b6-9ad5-2c0b32ab79ec (api:46) 29470:2018-03-21 10:43:52,518+0200 INFO (jsonrpc/2) [vdsm.api] FINISH = multipath_health return=3D{} from=3D::ffff:192.168.0.4,49914, = task_id=3Da7a44ff7-a137-41b6-9ad5-2c0b32ab79ec (api:52) 29471:2018-03-21 10:43:52,524+0200 INFO (jsonrpc/2) [api.host] FINISH = getStats return=3D{'status': {'message': 'Done', 'code': 0}, 'info': = {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.80'}, '10': {'cpuUser': '0.00', 'nodeIndex': 0, = 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '13': {'cpuUser': '0.00', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '12': {'cpuUser': = '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '15': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.86'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', = 'cpuIdle': '99.87'}, '1': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': = '1.13', 'cpuIdle': '97.74'}, '0': {'cpuUser': '0.33', 'nodeIndex': 0, = 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '3': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '2': {'cpuUser': = '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '5': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.87'}, '4': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.13', = 'cpuIdle': '99.60'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': = '0.33', 'cpuIdle': '99.67'}, '6': {'cpuUser': '0.00', 'nodeIndex': 0, = 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '9': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '8': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}}, = 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15143'}, '0': = {'memPercent': 5, 'memFree': '15349'}}, 'memShared': 0, 'thpState': = 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', = 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000487937', = 'lastCheck': '3.0', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000455956', 'lastCheck': = '2.9', 'valid': True}}, 'incomingVmMigrations': 0, 'network': = {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621825.13588, 'name': 'enp3s0f0', 'tx': '1091312739', 'txDropped': = '0', 'rx': '11914899051', 'rxErrors': '0', 'speed': '100', 'rxDropped': = '1443'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621825.13588, 'name': 'ovirtmgmt', 'tx': '1048663107', 'txDropped': = '0', 'rx': '11615236520', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621825.13588, 'name': 'lo', 'tx': '58090295494', 'txDropped': '0', = 'rx': '58090295494', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621825.13588, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': = '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621825.13588, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621825.13588, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': = '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': = {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621825.13588, = 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': = '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', = 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76391.35', = 'cpuLoad': '0.15', 'cpuSys': '0.20', 'diskStats': {'/var/log': {'free': = '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, = 'cpuUserVdsmd': '1.13', 'netConfigDirty': 'False', 'memCommitted': 0, = 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, = 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': = False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': = False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': = '1443', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': = 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:43:52 GMT', 'cpuUser': '0.14', = 'memFree': 31336, 'cpuIdle': '99.66', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.73'}} from=3D::ffff:192.168.0.4,49914 (api:52) 29472:2018-03-21 10:43:52,526+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573) 29473:2018-03-21 10:43:58,686+0200 INFO (jsonrpc/1) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3Dd6d29cf6-c083-4639-91a2-f062eae4e629 (api:46) 29474:2018-03-21 10:43:58,690+0200 INFO (jsonrpc/1) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3Dd6d29cf6-c083-4639-91a2-f062eae4e629 (api:52) 29475:2018-03-21 10:43:58,690+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573) 29476:2018-03-21 10:43:58,697+0200 INFO (jsonrpc/5) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D9dbd91df-dd9c-469c-8308-bfbfc228937c (api:46) 29477:2018-03-21 10:43:58,703+0200 INFO (jsonrpc/5) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D9dbd91df-dd9c-469c-8308-bfbfc228937c (api:52) 29478:2018-03-21 10:43:58,704+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573) grep -n 10:46 vdsm.log | tail -1000 1017:2018-03-21 06:10:46,882+0200 INFO (jsonrpc/2) [api.host] START = getStats() from=3D::ffff:192.168.0.4,49914 (api:46) 1018:2018-03-21 06:10:46,883+0200 INFO (jsonrpc/2) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D6e045b00-88b2-469b-80d6-af38303f8c32 (api:46) 1019:2018-03-21 06:10:46,883+0200 INFO (jsonrpc/2) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000332785', = 'lastCheck': '7.3', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000203914', 'lastCheck': = '5.6', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D6e045b00-88b2-469b-80d6-af38303f8c32 (api:52) 1020:2018-03-21 06:10:46,884+0200 INFO (jsonrpc/2) [vdsm.api] START = multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D475a5b40-4a43-42ee-9524-2bdd220aed83 (api:46) 1021:2018-03-21 06:10:46,884+0200 INFO (jsonrpc/2) [vdsm.api] FINISH = multipath_health return=3D{} from=3D::ffff:192.168.0.4,49914, = task_id=3D475a5b40-4a43-42ee-9524-2bdd220aed83 (api:52) 1022:2018-03-21 06:10:46,889+0200 INFO (jsonrpc/2) [api.host] FINISH = getStats return=3D{'status': {'message': 'Done', 'code': 0}, 'info': = {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.20', 'cpuIdle': '99.73'}, '10': {'cpuUser': '0.13', 'nodeIndex': 0, = 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '13': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '12': {'cpuUser': = '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '15': = {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '99.40'}, '14': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.40', = 'cpuIdle': '99.13'}, '1': {'cpuUser': '3.00', 'nodeIndex': 1, 'cpuSys': = '1.40', 'cpuIdle': '95.60'}, '0': {'cpuUser': '0.73', 'nodeIndex': 0, = 'cpuSys': '0.20', 'cpuIdle': '99.07'}, '3': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '2': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '5': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': = '99.73'}, '4': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', = 'cpuIdle': '99.54'}, '7': {'cpuUser': '0.80', 'nodeIndex': 1, 'cpuSys': = '0.80', 'cpuIdle': '98.40'}, '6': {'cpuUser': '5.46', 'nodeIndex': 0, = 'cpuSys': '0.27', 'cpuIdle': '94.27'}, '9': {'cpuUser': '0.20', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '8': {'cpuUser': = '1.27', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '98.26'}}, = 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15128'}, '0': = {'memPercent': 5, 'memFree': '15366'}}, 'memShared': 0, 'thpState': = 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', = 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000332785', = 'lastCheck': '7.3', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000203914', 'lastCheck': = '5.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': = {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521605437.203776, 'name': 'enp3s0f0', 'tx': '1082264809', 'txDropped': = '0', 'rx': '11907188594', 'rxErrors': '0', 'speed': '100', 'rxDropped': = '289'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521605437.203776, 'name': 'ovirtmgmt', 'tx': '1039869554', 'txDropped': = '0', 'rx': '11608697383', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521605437.203776, 'name': 'lo', 'tx': '45587320171', 'txDropped': '0', = 'rx': '45587320171', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521605437.203776, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '53620', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521605437.203776, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521605437.203776, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521605437.203776, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, = 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, = 'elapsedTime': '60005.72', 'cpuLoad': '0.06', 'cpuSys': '0.38', = 'diskStats': {'/var/log': {'free': '7357'}, '/var/run/vdsm/': {'free': = '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31084, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '289', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, = 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, = 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T04:10:46 GMT', 'cpuUser': '0.83', = 'memFree': 31340, 'cpuIdle': '98.79', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.80'}} from=3D::ffff:192.168.0.4,49914 (api:52) 1023:2018-03-21 06:10:46,891+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573) 13522:2018-03-21 08:10:46,827+0200 INFO (jsonrpc/7) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 13523:2018-03-21 08:10:46,828+0200 INFO (jsonrpc/7) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 13524:2018-03-21 08:10:46,828+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29734:2018-03-21 10:46:00,338+0200 INFO (periodic/2) [vdsm.api] START = repoStats(domains=3D()) from=3Dinternal, = task_id=3Daf099ce6-0f44-461b-9e8e-99de63b3884f (api:46) 29735:2018-03-21 10:46:00,339+0200 INFO (periodic/2) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470245', = 'lastCheck': '0.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000476935', 'lastCheck': = '0.7', 'valid': True}} from=3Dinternal, = task_id=3Daf099ce6-0f44-461b-9e8e-99de63b3884f (api:52) 29736:2018-03-21 10:46:00,339+0200 INFO (periodic/2) [vdsm.api] START = multipath_health() from=3Dinternal, = task_id=3D4a5709d9-e322-44eb-a8a4-bbe3f5e5e1bd (api:46) 29737:2018-03-21 10:46:00,340+0200 INFO (periodic/2) [vdsm.api] FINISH = multipath_health return=3D{} from=3Dinternal, = task_id=3D4a5709d9-e322-44eb-a8a4-bbe3f5e5e1bd (api:52) 29738:2018-03-21 10:46:04,637+0200 INFO (jsonrpc/7) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D5b9300f7-f8c4-4f63-a49c-45ca8119d3b9 (api:46) 29739:2018-03-21 10:46:04,642+0200 INFO (jsonrpc/7) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D5b9300f7-f8c4-4f63-a49c-45ca8119d3b9 (api:52) 29740:2018-03-21 10:46:04,642+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573) 29741:2018-03-21 10:46:04,648+0200 INFO (jsonrpc/1) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3Df4db5bfa-e131-4f9a-baaa-e7555c996ab0 (api:46) 29742:2018-03-21 10:46:04,655+0200 INFO (jsonrpc/1) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3Df4db5bfa-e131-4f9a-baaa-e7555c996ab0 (api:52) 29743:2018-03-21 10:46:04,656+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 = seconds (__init__:573) 29744:2018-03-21 10:46:05,451+0200 INFO (jsonrpc/5) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29745:2018-03-21 10:46:05,452+0200 INFO (jsonrpc/5) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 29746:2018-03-21 10:46:05,452+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29747:2018-03-21 10:46:06,903+0200 INFO (jsonrpc/3) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 29748:2018-03-21 10:46:06,904+0200 INFO (jsonrpc/3) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 29749:2018-03-21 10:46:06,904+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29750:2018-03-21 10:46:08,791+0200 WARN (vdsm.Scheduler) [Executor] = Worker blocked: <Worker name=3Djsonrpc/6 running <Task <JsonRpcTask = {'params': {u'connectionParams': [{u'id': = u'461f65a9-3a81-4f3f-a46d-c5ed12520524', u'connection': = u'node10.starlett.lv:/vmdata/nfs/exports', u'iqn': u'', u'user': u'', = u'tpgt': u'1', u'protocol_version': u'auto', u'password': '********', = u'port': u''}], u'storagepoolID': = u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': = '2.0', 'method': u'StoragePool.connectStorageServer', 'id': = u'71143eef-aac6-4996-9783-3e0e3da180c3'} at 0x3604fd0> timeout=3D60, = duration=3D120 at 0x3604e10> task#=3D3762 at 0x3541650>, traceback: 29801:2018-03-21 10:46:12,852+0200 INFO (jsonrpc/0) [api.host] START = getStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29802:2018-03-21 10:46:12,853+0200 INFO (jsonrpc/0) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D391fdab9-d3c2-4075-9c34-177a07a21ec3 (api:46) 29803:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000492277', = 'lastCheck': '3.3', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': = '3.2', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D391fdab9-d3c2-4075-9c34-177a07a21ec3 (api:52) 29804:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) [vdsm.api] START = multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D7e53adbc-3974-4f51-bfc8-8fbd3ca6b749 (api:46) 29805:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) [vdsm.api] FINISH = multipath_health return=3D{} from=3D::ffff:192.168.0.4,49914, = task_id=3D7e53adbc-3974-4f51-bfc8-8fbd3ca6b749 (api:52) 29806:2018-03-21 10:46:12,861+0200 INFO (jsonrpc/0) [api.host] FINISH = getStats return=3D{'status': {'message': 'Done', 'code': 0}, 'info': = {'cpuStatistics': {'11': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': = '0.40', 'cpuIdle': '99.13'}, '10': {'cpuUser': '0.13', 'nodeIndex': 0, = 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '13': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '12': {'cpuUser': = '0.20', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '15': = {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '99.60'}, '14': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', = 'cpuIdle': '99.54'}, '1': {'cpuUser': '1.00', 'nodeIndex': 1, 'cpuSys': = '1.33', 'cpuIdle': '97.67'}, '0': {'cpuUser': '0.13', 'nodeIndex': 0, = 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '3': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '2': {'cpuUser': = '0.20', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.47'}, '5': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.80'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', = 'cpuIdle': '99.73'}, '7': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.40', 'cpuIdle': '99.53'}, '6': {'cpuUser': '0.47', 'nodeIndex': 0, = 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '9': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '8': {'cpuUser': = '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}}, = 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15142'}, '0': = {'memPercent': 5, 'memFree': '15348'}}, 'memShared': 0, 'thpState': = 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', = 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000492277', = 'lastCheck': '3.3', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': = '3.2', 'valid': True}}, 'incomingVmMigrations': 0, 'network': = {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621960.326184, 'name': 'enp3s0f0', 'tx': '1091387790', 'txDropped': = '0', 'rx': '11914959714', 'rxErrors': '0', 'speed': '100', 'rxDropped': = '1451'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621960.326184, 'name': 'ovirtmgmt', 'tx': '1048735814', 'txDropped': = '0', 'rx': '11615289173', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621960.326184, 'name': 'lo', 'tx': '58192068637', 'txDropped': '0', = 'rx': '58192068637', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621960.326184, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621960.326184, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621960.326184, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621960.326184, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, = 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, = 'elapsedTime': '76531.69', 'cpuLoad': '0.17', 'cpuSys': '0.32', = 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': = '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.00', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31081, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1451', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, = 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, = 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:46:12 GMT', 'cpuUser': '0.21', = 'memFree': 31337, 'cpuIdle': '99.46', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.80'}} from=3D::ffff:192.168.0.4,49914 (api:52) 29807:2018-03-21 10:46:12,863+0200 INFO (jsonrpc/0) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573) 29808:2018-03-21 10:46:13,829+0200 ERROR (jsonrpc/6) [storage.HSM] Could = not connect to storageServer (hsm:2407) 29828:2018-03-21 10:46:13,829+0200 INFO (jsonrpc/6) [vdsm.api] FINISH = connectStorageServer return=3D{'statuslist': [{'status': 477, 'id': = u'461f65a9-3a81-4f3f-a46d-c5ed12520524'}]} = from=3D::ffff:192.168.0.4,49914, flow_id=3D1435fc81, = task_id=3D0a828d2c-d9f4-4f83-a9e9-7393159d5323 (api:52) 29829:2018-03-21 10:46:13,830+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer = succeeded in 125.04 seconds (__init__:573) 29830:2018-03-21 10:46:14,767+0200 INFO (jsonrpc/4) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D49cd2ae6-7943-47c9-ad86-9e0b7e58bca3 (api:46) 29831:2018-03-21 10:46:14,772+0200 INFO (jsonrpc/4) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D49cd2ae6-7943-47c9-ad86-9e0b7e58bca3 (api:52) 29832:2018-03-21 10:46:14,773+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573) 29833:2018-03-21 10:46:14,815+0200 INFO (jsonrpc/2) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D19909616-efb4-490d-982c-66eda9ca4381 (api:46) 29834:2018-03-21 10:46:14,822+0200 INFO (jsonrpc/2) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D19909616-efb4-490d-982c-66eda9ca4381 (api:52) 29835:2018-03-21 10:46:14,823+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573) 29836:2018-03-21 10:46:15,360+0200 INFO (periodic/2) [vdsm.api] START = repoStats(domains=3D()) from=3Dinternal, = task_id=3Dda74ea9f-164b-450d-b836-6818caa3fdc5 (api:46) 29837:2018-03-21 10:46:15,360+0200 INFO (periodic/2) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000492277', = 'lastCheck': '5.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': = '5.7', 'valid': True}} from=3Dinternal, = task_id=3Dda74ea9f-164b-450d-b836-6818caa3fdc5 (api:52) 29838:2018-03-21 10:46:15,361+0200 INFO (periodic/2) [vdsm.api] START = multipath_health() from=3Dinternal, = task_id=3D06245e2b-e8ca-41c4-90bf-4294d0c699b8 (api:46) 29839:2018-03-21 10:46:15,361+0200 INFO (periodic/2) [vdsm.api] FINISH = multipath_health return=3D{} from=3Dinternal, = task_id=3D06245e2b-e8ca-41c4-90bf-4294d0c699b8 (api:52) 29840:2018-03-21 10:46:20,456+0200 INFO (jsonrpc/7) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29841:2018-03-21 10:46:20,457+0200 INFO (jsonrpc/7) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 29842:2018-03-21 10:46:20,457+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 = seconds (__init__:573) 29843:2018-03-21 10:46:21,925+0200 INFO (jsonrpc/1) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 29844:2018-03-21 10:46:21,926+0200 INFO (jsonrpc/1) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 29845:2018-03-21 10:46:21,926+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29846:2018-03-21 10:46:24,930+0200 INFO (jsonrpc/5) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3Dd964535d-e5eb-42bc-b324-432a05c364da (api:46) 29847:2018-03-21 10:46:24,935+0200 INFO (jsonrpc/5) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3Dd964535d-e5eb-42bc-b324-432a05c364da (api:52) 29848:2018-03-21 10:46:24,935+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.00 seconds (__init__:573) 29849:2018-03-21 10:46:24,987+0200 INFO (jsonrpc/3) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D167dcd23-04b5-4de3-9c81-7d25ba56407a (api:46) 29850:2018-03-21 10:46:25,022+0200 INFO (jsonrpc/3) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D167dcd23-04b5-4de3-9c81-7d25ba56407a (api:52) 29851:2018-03-21 10:46:25,023+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.03 = seconds (__init__:573) 29852:2018-03-21 10:46:28,010+0200 INFO (jsonrpc/0) [api.host] START = getStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29853:2018-03-21 10:46:28,011+0200 INFO (jsonrpc/0) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D011616ce-53c9-4cfc-8b29-247b1be03409 (api:46) 29854:2018-03-21 10:46:28,011+0200 INFO (jsonrpc/0) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000218955', = 'lastCheck': '8.4', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000351741', 'lastCheck': = '8.4', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D011616ce-53c9-4cfc-8b29-247b1be03409 (api:52) 29855:2018-03-21 10:46:28,012+0200 INFO (jsonrpc/0) [vdsm.api] START = multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D85b42a8d-1bf1-495d-ad32-a1f9710a6468 (api:46) 29856:2018-03-21 10:46:28,012+0200 INFO (jsonrpc/0) [vdsm.api] FINISH = multipath_health return=3D{} from=3D::ffff:192.168.0.4,49914, = task_id=3D85b42a8d-1bf1-495d-ad32-a1f9710a6468 (api:52) 29857:2018-03-21 10:46:28,018+0200 INFO (jsonrpc/0) [api.host] FINISH = getStats return=3D{'status': {'message': 'Done', 'code': 0}, 'info': = {'cpuStatistics': {'11': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': = '0.20', 'cpuIdle': '99.67'}, '10': {'cpuUser': '1.73', 'nodeIndex': 0, = 'cpuSys': '0.53', 'cpuIdle': '97.74'}, '13': {'cpuUser': '0.20', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '12': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '15': = {'cpuUser': '0.67', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '99.06'}, '14': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.33', = 'cpuIdle': '99.47'}, '1': {'cpuUser': '1.26', 'nodeIndex': 1, 'cpuSys': = '1.33', 'cpuIdle': '97.41'}, '0': {'cpuUser': '0.40', 'nodeIndex': 0, = 'cpuSys': '0.07', 'cpuIdle': '99.53'}, '3': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '2': {'cpuUser': = '0.00', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.73'}, '5': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.87'}, '4': {'cpuUser': '13.78', 'nodeIndex': 0, 'cpuSys': '0.80', = 'cpuIdle': '85.42'}, '7': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': = '0.67', 'cpuIdle': '98.86'}, '6': {'cpuUser': '1.07', 'nodeIndex': 0, = 'cpuSys': '0.27', 'cpuIdle': '98.66'}, '9': {'cpuUser': '1.20', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '98.53'}, '8': {'cpuUser': = '0.80', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '98.87'}}, = 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15136'}, '0': = {'memPercent': 5, 'memFree': '15357'}}, 'memShared': 0, 'thpState': = 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', = 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000218955', = 'lastCheck': '8.4', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000351741', 'lastCheck': = '8.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': = {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621975.346633, 'name': 'enp3s0f0', 'tx': '1091397599', 'txDropped': = '0', 'rx': '11914966591', 'rxErrors': '0', 'speed': '100', 'rxDropped': = '1454'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621975.346633, 'name': 'ovirtmgmt', 'tx': '1048745347', 'txDropped': = '0', 'rx': '11615294897', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621975.346633, 'name': 'lo', 'tx': '58205583384', 'txDropped': '0', = 'rx': '58205583384', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621975.346633, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621975.346633, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621975.346633, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621975.346633, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, = 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, = 'elapsedTime': '76546.85', 'cpuLoad': '0.18', 'cpuSys': '0.39', = 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': = '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.20', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31031, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1454', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, = 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, = 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:46:28 GMT', 'cpuUser': '1.38', = 'memFree': 31287, 'cpuIdle': '98.23', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.87'}} from=3D::ffff:192.168.0.4,49914 (api:52) 29858:2018-03-21 10:46:28,020+0200 INFO (jsonrpc/0) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573) 29859:2018-03-21 10:46:29,667+0200 INFO (itmap/0) [IOProcessClient] = Starting client ioprocess-575 (__init__:308) 29860:2018-03-21 10:46:29,700+0200 INFO (itmap/1) [IOProcessClient] = Starting client ioprocess-576 (__init__:308) 29861:2018-03-21 10:46:29,714+0200 INFO (ioprocess/1711202) [IOProcess] = Starting ioprocess (__init__:437) 29862:2018-03-21 10:46:29,725+0200 INFO (ioprocess/1711208) [IOProcess] = Starting ioprocess (__init__:437) 29863:2018-03-21 10:46:30,375+0200 INFO (periodic/0) [vdsm.api] START = repoStats(domains=3D()) from=3Dinternal, = task_id=3Dbdacf1c0-3fcb-43b1-82e2-a041a26d0d7f (api:46) 29864:2018-03-21 10:46:30,376+0200 INFO (periodic/0) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00031111', = 'lastCheck': '0.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.00045594', 'lastCheck': = '0.6', 'valid': True}} from=3Dinternal, = task_id=3Dbdacf1c0-3fcb-43b1-82e2-a041a26d0d7f (api:52) 29865:2018-03-21 10:46:30,376+0200 INFO (periodic/0) [vdsm.api] START = multipath_health() from=3Dinternal, = task_id=3D3d3cad70-e5c8-49b2-9f5d-92a249cc102d (api:46) 29866:2018-03-21 10:46:30,377+0200 INFO (periodic/0) [vdsm.api] FINISH = multipath_health return=3D{} from=3Dinternal, = task_id=3D3d3cad70-e5c8-49b2-9f5d-92a249cc102d (api:52) 29867:2018-03-21 10:46:35,108+0200 INFO (jsonrpc/6) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D304d7e7c-03fc-44d6-be32-d083ada09b30 (api:46) 29868:2018-03-21 10:46:35,114+0200 INFO (jsonrpc/6) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D304d7e7c-03fc-44d6-be32-d083ada09b30 (api:52) 29869:2018-03-21 10:46:35,114+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.00 seconds (__init__:573) 29870:2018-03-21 10:46:35,170+0200 INFO (jsonrpc/4) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D76736a93-d4c4-4b2f-9848-7b8731fe2b67 (api:46) 29871:2018-03-21 10:46:35,176+0200 INFO (jsonrpc/4) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D76736a93-d4c4-4b2f-9848-7b8731fe2b67 (api:52) 29872:2018-03-21 10:46:35,177+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573) 29873:2018-03-21 10:46:35,462+0200 INFO (jsonrpc/2) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29874:2018-03-21 10:46:35,462+0200 INFO (jsonrpc/2) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 29875:2018-03-21 10:46:35,463+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29876:2018-03-21 10:46:36,948+0200 INFO (jsonrpc/7) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 29877:2018-03-21 10:46:36,948+0200 INFO (jsonrpc/7) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 29878:2018-03-21 10:46:36,949+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29879:2018-03-21 10:46:43,179+0200 INFO (jsonrpc/1) [api.host] START = getStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29880:2018-03-21 10:46:43,180+0200 INFO (jsonrpc/1) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D83bbc95c-6d59-4eec-82b1-e8993be28759 (api:46) 29881:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000486101', = 'lastCheck': '3.6', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': = '3.4', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D83bbc95c-6d59-4eec-82b1-e8993be28759 (api:52) 29882:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) [vdsm.api] START = multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D9b781324-b50b-49a9-bf31-5eca835dc75c (api:46) 29883:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) [vdsm.api] FINISH = multipath_health return=3D{} from=3D::ffff:192.168.0.4,49914, = task_id=3D9b781324-b50b-49a9-bf31-5eca835dc75c (api:52) 29884:2018-03-21 10:46:43,188+0200 INFO (jsonrpc/1) [api.host] FINISH = getStats return=3D{'status': {'message': 'Done', 'code': 0}, 'info': = {'cpuStatistics': {'11': {'cpuUser': '0.40', 'nodeIndex': 1, 'cpuSys': = '0.27', 'cpuIdle': '99.33'}, '10': {'cpuUser': '0.60', 'nodeIndex': 0, = 'cpuSys': '0.47', 'cpuIdle': '98.93'}, '13': {'cpuUser': '0.00', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.73'}, '12': {'cpuUser': = '1.46', 'nodeIndex': 0, 'cpuSys': '0.60', 'cpuIdle': '97.94'}, '15': = {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': = '99.47'}, '14': {'cpuUser': '0.40', 'nodeIndex': 0, 'cpuSys': '0.47', = 'cpuIdle': '99.13'}, '1': {'cpuUser': '1.33', 'nodeIndex': 1, 'cpuSys': = '1.93', 'cpuIdle': '96.74'}, '0': {'cpuUser': '0.60', 'nodeIndex': 0, = 'cpuSys': '0.27', 'cpuIdle': '99.13'}, '3': {'cpuUser': '0.20', = 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.40'}, '2': {'cpuUser': = '6.99', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '92.61'}, '5': = {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': = '99.67'}, '4': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.27', = 'cpuIdle': '99.46'}, '7': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': = '0.60', 'cpuIdle': '98.93'}, '6': {'cpuUser': '0.53', 'nodeIndex': 0, = 'cpuSys': '0.27', 'cpuIdle': '99.20'}, '9': {'cpuUser': '0.47', = 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '99.20'}, '8': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}}, = 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15131'}, '0': = {'memPercent': 5, 'memFree': '15357'}}, 'memShared': 0, 'thpState': = 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', = 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000486101', = 'lastCheck': '3.6', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': = '3.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': = {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621990.363726, 'name': 'enp3s0f0', 'tx': '1091404888', 'txDropped': = '0', 'rx': '11914970667', 'rxErrors': '0', 'speed': '100', 'rxDropped': = '1455'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621990.363726, 'name': 'ovirtmgmt', 'tx': '1048752406', 'txDropped': = '0', 'rx': '11615298347', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621990.363726, 'name': 'lo', 'tx': '58215969455', 'txDropped': '0', = 'rx': '58215969455', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621990.363726, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621990.363726, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621990.363726, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621990.363726, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, = 'txDropped': '0', 'anonHugePages': '206', 'ksmPages': 100, = 'elapsedTime': '76562.02', 'cpuLoad': '0.18', 'cpuSys': '0.45', = 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': = '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.27', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1455', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, = 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, = 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:46:43 GMT', 'cpuUser': '0.87', = 'memFree': 31336, 'cpuIdle': '98.68', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '1.20'}} from=3D::ffff:192.168.0.4,49914 (api:52) 29885:2018-03-21 10:46:43,190+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573) 29886:2018-03-21 10:46:45,288+0200 INFO (jsonrpc/5) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3Dffe57349-2532-4b64-b32c-8a558273d8e5 (api:46) 29887:2018-03-21 10:46:45,292+0200 INFO (jsonrpc/5) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3Dffe57349-2532-4b64-b32c-8a558273d8e5 (api:52) 29888:2018-03-21 10:46:45,293+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.00 seconds (__init__:573) 29889:2018-03-21 10:46:45,298+0200 INFO (jsonrpc/3) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3Db2ea28f0-abf0-4258-8152-6035d93aa5dc (api:46) 29890:2018-03-21 10:46:45,305+0200 INFO (jsonrpc/3) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3Db2ea28f0-abf0-4258-8152-6035d93aa5dc (api:52) 29891:2018-03-21 10:46:45,306+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 = seconds (__init__:573) 29892:2018-03-21 10:46:45,395+0200 INFO (periodic/2) [vdsm.api] START = repoStats(domains=3D()) from=3Dinternal, = task_id=3D4010bf72-5c86-4daf-a8f9-12ec229a86e4 (api:46) 29893:2018-03-21 10:46:45,395+0200 INFO (periodic/2) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000486101', = 'lastCheck': '5.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': = '5.7', 'valid': True}} from=3Dinternal, = task_id=3D4010bf72-5c86-4daf-a8f9-12ec229a86e4 (api:52) 29894:2018-03-21 10:46:45,396+0200 INFO (periodic/2) [vdsm.api] START = multipath_health() from=3Dinternal, = task_id=3D37a1132f-452b-4805-9cc0-b8e74e8d6b02 (api:46) 29895:2018-03-21 10:46:45,396+0200 INFO (periodic/2) [vdsm.api] FINISH = multipath_health return=3D{} from=3Dinternal, = task_id=3D37a1132f-452b-4805-9cc0-b8e74e8d6b02 (api:52) 29896:2018-03-21 10:46:50,467+0200 INFO (jsonrpc/0) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29897:2018-03-21 10:46:50,468+0200 INFO (jsonrpc/0) [throttled] Current = getAllVmStats: {} (throttledlog:103) 29898:2018-03-21 10:46:50,468+0200 INFO (jsonrpc/0) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 29899:2018-03-21 10:46:50,468+0200 INFO (jsonrpc/0) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 = seconds (__init__:573) 29900:2018-03-21 10:46:51,967+0200 INFO (jsonrpc/6) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 29901:2018-03-21 10:46:51,968+0200 INFO (jsonrpc/6) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 29902:2018-03-21 10:46:51,968+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29903:2018-03-21 10:46:55,648+0200 INFO (jsonrpc/4) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3Da975635f-59d7-4f40-b9a7-a5ea7150368e (api:46) 29904:2018-03-21 10:46:55,653+0200 INFO (jsonrpc/4) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3Da975635f-59d7-4f40-b9a7-a5ea7150368e (api:52) 29905:2018-03-21 10:46:55,653+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.00 seconds (__init__:573) 29906:2018-03-21 10:46:55,659+0200 INFO (jsonrpc/2) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D97d4ae6c-3719-4149-85fd-28e924ad8883 (api:46) 29907:2018-03-21 10:46:55,666+0200 INFO (jsonrpc/2) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D97d4ae6c-3719-4149-85fd-28e924ad8883 (api:52) 29908:2018-03-21 10:46:55,667+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 = seconds (__init__:573) 29909:2018-03-21 10:46:59,473+0200 INFO (jsonrpc/7) [api.host] START = getStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29910:2018-03-21 10:46:59,474+0200 INFO (jsonrpc/7) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D177b551b-9bd3-45a4-84d0-5244630c08e5 (api:46) 29911:2018-03-21 10:46:59,474+0200 INFO (jsonrpc/7) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470906', = 'lastCheck': '9.9', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000334928', 'lastCheck': = '9.7', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D177b551b-9bd3-45a4-84d0-5244630c08e5 (api:52) 29912:2018-03-21 10:46:59,475+0200 INFO (jsonrpc/7) [vdsm.api] START = multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D30753d5c-9bfd-4b0f-ac83-e32278d82b63 (api:46) 29913:2018-03-21 10:46:59,475+0200 INFO (jsonrpc/7) [vdsm.api] FINISH = multipath_health return=3D{} from=3D::ffff:192.168.0.4,49914, = task_id=3D30753d5c-9bfd-4b0f-ac83-e32278d82b63 (api:52) 29914:2018-03-21 10:46:59,482+0200 INFO (jsonrpc/7) [api.host] FINISH = getStats return=3D{'status': {'message': 'Done', 'code': 0}, 'info': = {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.80'}, '10': {'cpuUser': '0.07', 'nodeIndex': 0, = 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '13': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '12': {'cpuUser': = '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '15': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.87'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', = 'cpuIdle': '99.73'}, '1': {'cpuUser': '1.07', 'nodeIndex': 1, 'cpuSys': = '1.07', 'cpuIdle': '97.86'}, '0': {'cpuUser': '0.07', 'nodeIndex': 0, = 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '3': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '2': {'cpuUser': = '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '5': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.80'}, '4': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.27', = 'cpuIdle': '99.26'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': = '0.47', 'cpuIdle': '99.53'}, '6': {'cpuUser': '0.20', 'nodeIndex': 0, = 'cpuSys': '0.07', 'cpuIdle': '99.73'}, '9': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '8': {'cpuUser': = '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}}, = 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15125'}, '0': = {'memPercent': 5, 'memFree': '15364'}}, 'memShared': 0, 'thpState': = 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', = 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470906', = 'lastCheck': '9.9', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000334928', 'lastCheck': = '9.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': = {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521622005.383289, 'name': 'enp3s0f0', 'tx': '1091412332', 'txDropped': = '0', 'rx': '11914976557', 'rxErrors': '0', 'speed': '100', 'rxDropped': = '1456'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521622005.383289, 'name': 'ovirtmgmt', 'tx': '1048761326', 'txDropped': = '0', 'rx': '11615304242', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521622005.383289, 'name': 'lo', 'tx': '58228460774', 'txDropped': '0', = 'rx': '58228460774', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521622005.383289, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521622005.383289, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521622005.383289, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521622005.383289, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, = 'txDropped': '0', 'anonHugePages': '208', 'ksmPages': 100, = 'elapsedTime': '76578.31', 'cpuLoad': '0.17', 'cpuSys': '0.20', = 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': = '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.13', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31078, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1456', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, = 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, = 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:46:59 GMT', 'cpuUser': '0.15', = 'memFree': 31334, 'cpuIdle': '99.65', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.60'}} from=3D::ffff:192.168.0.4,49914 (api:52) 29915:2018-03-21 10:46:59,484+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573) maintenance mode and detached from DC #1. the Storage Domains node10-NFS-EXPORTS.
2018-03-21 10:46:16,651+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage = Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: = admin@internal-authz) =20 =20 tail -n 1000 engine.log | grep 570ec5d9-fff5-4656-afbd-90b3207a616e 2018-03-21 10:41:14,643+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (default task-2) [570ec5d9-fff5-4656-afbd-90b3207a616e] Lock Acquired = to object = 'EngineLock:{exclusiveLocks=3D'[1d7208ce-d3a1-4406-9638-fe7051562994=3DSTO= RAGE]', sharedLocks=3D''}' 2018-03-21 10:41:16,129+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] Running command: = AttachStorageDomainToPoolCommand internal: false. Entities affected : = ID: 1d7208ce-d3a1-4406-9638-fe7051562994 Type: StorageAction group = MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: = 80cc922f-8dea-4fed-b951-1060ba116ad5 Type: StoragePoolAction group = MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2018-03-21 10:43:23,564+02 ERROR = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] Cannot connect storage connection = server, aborting attach storage domain operation. 2018-03-21 10:43:23,567+02 INFO = [org.ovirt.engine.core.bll.CommandCompensator] = (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] Command = [id=3D921ca7cd-4f93-46aa-8de2-91b13b8f96cb]: Compensating NEW_ENTITY_ID = of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; = snapshot: = StoragePoolIsoMapId:{storagePoolId=3D'80cc922f-8dea-4fed-b951-1060ba116ad5= ', storageId=3D'1d7208ce-d3a1-4406-9638-fe7051562994'}. 2018-03-21 10:43:24,024+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage = Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: = admin@internal-authz) 2018-03-21 10:43:24,114+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] Lock freed to object = 'EngineLock:{exclusiveLocks=3D'[1d7208ce-d3a1-4406-9638-fe7051562994=3DSTO= RAGE]', sharedLocks=3D''}' =20 =20 =20 [root@node00 ovirt-engine]# tail -n 1000 engine.log | grep = a81ffa4a-5a58-41a0-888a-f0edc321609b 2018-03-21 10:44:11,025+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (default task-16) [a81ffa4a-5a58-41a0-888a-f0edc321609b] Lock Acquired = to object = 'EngineLock:{exclusiveLocks=3D'[1d7208ce-d3a1-4406-9638-fe7051562994=3DSTO= RAGE]', sharedLocks=3D''}' 2018-03-21 10:44:11,236+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] Running command: = AttachStorageDomainToPoolCommand internal: false. Entities affected : = ID: 1d7208ce-d3a1-4406-9638-fe7051562994 Type: StorageAction group = MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: = 80cc922f-8dea-4fed-b951-1060ba116ad5 Type: StoragePoolAction group = MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2018-03-21 10:46:16,567+02 ERROR = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] Cannot connect storage connection = server, aborting attach storage domain operation. 2018-03-21 10:46:16,568+02 INFO = [org.ovirt.engine.core.bll.CommandCompensator] = (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] Command = [id=3Db5c25100-1a8a-4db0-9509-99cfa60995b1]: Compensating NEW_ENTITY_ID = of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; = snapshot: = StoragePoolIsoMapId:{storagePoolId=3D'80cc922f-8dea-4fed-b951-1060ba116ad5= ', storageId=3D'1d7208ce-d3a1-4406-9638-fe7051562994'}. 2018-03-21 10:46:16,651+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage = Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: = admin@internal-authz) 2018-03-21 10:46:16,681+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] Lock freed to object = 'EngineLock:{exclusiveLocks=3D'[1d7208ce-d3a1-4406-9638-fe7051562994=3DSTO= RAGE]', sharedLocks=3D'=E2=80=99}' =20 =20 _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users = <http://lists.ovirt.org/mailman/listinfo/users> =20 =20
--Apple-Mail=_C5052CF0-37B9-4927-AE48-DF656F09C5B6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D""><div = class=3D"">Hi,</div><div class=3D""><br class=3D""></div><div = class=3D""><br class=3D""></div><div class=3D"">Errors occurred at 10.43 = AM and 10.46 AM</div><div class=3D"">node00.starlett.lv - 192.168.0.4 - = oVirt host engine (separate PC, not hosted)</div><div = class=3D"">node10.starlett.lv - 192.168.0.5 - host #1 of DC #1, export = domain from which was detached from DC #1</div><div = class=3D"">node11.starlett.lv - 192.168.0.6 - host #1 of DC = #2, </div><div class=3D""><br class=3D""></div>Logs from DC#2 = node11, to which I=E2=80=99m trying to attach export domain located at = NFS share node10.<div class=3D""><br class=3D""></div><div = class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"gmail_extra"><div class=3D"gmail_quote"><blockquote = class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex; = border-left-width: 1px; border-left-style: solid; border-left-color: = rgb(204, 204, 204); padding-left: 1ex;"><div class=3D"" = style=3D"word-wrap: break-word; line-break: after-white-space;"><div = class=3D""><div class=3D""><font color=3D"#aa7942" class=3D""><b = class=3D"">grep 570ec5d9-fff5-4656-afbd-<wbr = class=3D"">90b3207a616e</b></font></div></div></div></blockquote></div></d= iv></blockquote>within vdsm.log returned nothing, so I did</div><div = class=3D""><br class=3D""></div><div class=3D""><br class=3D""></div><div = class=3D""><font color=3D"#942192" class=3D""><b class=3D"">grep -n = 10:43 vdsm.log | tail -1000</b></font><br class=3D"">1011:2018-03-21 = 06:10:43,077+0200 INFO (jsonrpc/3) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46)<br class=3D"">1012:2018-03-21 = 06:10:43,077+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats = return=3D{'status': {'message': 'Done', 'code': 0}, 'statsList': = (suppressed)} from=3D::1,36114 (api:52)<br class=3D"">1013:2018-03-21 = 06:10:43,077+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573)<br = class=3D"">1014:2018-03-21 06:10:43,868+0200 INFO (jsonrpc/7) = [api.host] START getAllVmStats() from=3D::ffff:192.168.0.4,49914 = (api:46)<br class=3D"">1015:2018-03-21 06:10:43,868+0200 INFO = (jsonrpc/7) [api.host] FINISH getAllVmStats return=3D{'status': = {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} = from=3D::ffff:192.168.0.4,49914 (api:52)<br class=3D"">1016:2018-03-21 = 06:10:43,868+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br = class=3D"">19729:2018-03-21 09:10:43,641+0200 INFO (jsonrpc/5) = [api.host] START getAllVmStats() from=3D::1,36114 (api:46)<br = class=3D"">19730:2018-03-21 09:10:43,641+0200 INFO (jsonrpc/5) = [api.host] FINISH getAllVmStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'statsList': (suppressed)} from=3D::1,36114 (api:52)<br = class=3D"">19731:2018-03-21 09:10:43,642+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29302:2018-03-21 10:43:00,085+0200 = INFO (periodic/2) [vdsm.api] START repoStats(domains=3D()) = from=3Dinternal, task_id=3D7b690c8a-5470-44da-a8f3-7e7e9b018e88 = (api:46)<br class=3D"">29303:2018-03-21 10:43:00,085+0200 INFO = (periodic/2) [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000477038', = 'lastCheck': '0.5', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': = '0.4', 'valid': True}} from=3Dinternal, = task_id=3D7b690c8a-5470-44da-a8f3-7e7e9b018e88 (api:52)<br = class=3D"">29304:2018-03-21 10:43:00,086+0200 INFO (periodic/2) = [vdsm.api] START multipath_health() from=3Dinternal, = task_id=3D9a625d43-a03c-429c-856f-9aa8e5ff65b5 (api:46)<br = class=3D"">29305:2018-03-21 10:43:00,086+0200 INFO (periodic/2) = [vdsm.api] FINISH multipath_health return=3D{} from=3Dinternal, = task_id=3D9a625d43-a03c-429c-856f-9aa8e5ff65b5 (api:52)<br = class=3D"">29306:2018-03-21 10:43:05,382+0200 INFO (jsonrpc/5) = [api.host] START getAllVmStats() from=3D::ffff:192.168.0.4,49914 = (api:46)<br class=3D"">29307:2018-03-21 10:43:05,383+0200 INFO = (jsonrpc/5) [api.host] FINISH getAllVmStats return=3D{'status': = {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} = from=3D::ffff:192.168.0.4,49914 (api:52)<br class=3D"">29308:2018-03-21 = 10:43:05,383+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br = class=3D"">29309:2018-03-21 10:43:06,248+0200 INFO (jsonrpc/3) = [api.host] START getStats() from=3D::ffff:192.168.0.4,49914 (api:46)<br = class=3D"">29310:2018-03-21 10:43:06,249+0200 INFO (jsonrpc/3) = [vdsm.api] START repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3Df80f9bb8-2e84-4f45-ab58-09a88e808cf3 (api:46)<br = class=3D"">29311:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000477038', = 'lastCheck': '6.7', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': = '6.6', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3Df80f9bb8-2e84-4f45-ab58-09a88e808cf3 (api:52)<br = class=3D"">29312:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) = [vdsm.api] START multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3Dddd7064d-cd12-43bf-a177-b2a93cc9035e (api:46)<br = class=3D"">29313:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) = [vdsm.api] FINISH multipath_health return=3D{} = from=3D::ffff:192.168.0.4,49914, = task_id=3Dddd7064d-cd12-43bf-a177-b2a93cc9035e (api:52)<br = class=3D"">29314:2018-03-21 10:43:06,256+0200 INFO (jsonrpc/3) = [api.host] FINISH getStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.13', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '10': {'cpuUser': = '0.13', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.47'}, '13': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': = '99.73'}, '12': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', = 'cpuIdle': '99.67'}, '15': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': = '0.47', 'cpuIdle': '99.33'}, '14': {'cpuUser': '0.13', 'nodeIndex': 0, = 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '1': {'cpuUser': '1.13', = 'nodeIndex': 1, 'cpuSys': '1.60', 'cpuIdle': '97.27'}, '0': {'cpuUser': = '0.47', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.26'}, '3': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': = '99.73'}, '2': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.13', = 'cpuIdle': '99.54'}, '5': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.74'}, '4': {'cpuUser': '0.13', 'nodeIndex': 0, = 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '7': {'cpuUser': '0.13', = 'nodeIndex': 1, 'cpuSys': '0.60', 'cpuIdle': '99.27'}, '6': {'cpuUser': = '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '9': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '99.66'}, '8': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', = 'cpuIdle': '99.54'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, = 'memFree': '15142'}, '0': {'memPercent': 5, 'memFree': '15349'}}, = 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, = 'vmCount': 0, 'memUsed': '3', 'storageDomains': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, = 'version': 4, 'acquired': True, 'delay': '0.000477038', 'lastCheck': = '6.7', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.000498252', 'lastCheck': '6.6', 'valid': True}}, = 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'enp3s0f0', = 'tx': '1091287978', 'txDropped': '0', 'rx': '11914868563', 'rxErrors': = '0', 'speed': '100', 'rxDropped': '1439'}, 'ovirtmgmt': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': = 'ovirtmgmt', 'tx': '1048639112', 'txDropped': '0', 'rx': '11615209092', = 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'lo', 'tx': = '58057075945', 'txDropped': '0', 'rx': '58057075945', 'rxErrors': '0', = 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'enp3s0f1', = 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': = '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': = 'down', 'sampleTime': 1521621780.072418, 'name': ';vdsmdummy;', 'tx': = '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621780.072418, 'name': 'enp4s0f0', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621780.072418, 'name': 'enp4s0f1', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', = 'ksmPages': 100, 'elapsedTime': '76345.09', 'cpuLoad': '0.16', 'cpuSys': = '0.36', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': = {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31081, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1439', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:43:06 GMT', 'cpuUser': '0.21', = 'memFree': 31337, 'cpuIdle': '99.43', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.93'}} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29315:2018-03-21 10:43:06,258+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573)<br class=3D"">29316:2018-03-21 10:43:06,632+0200 = INFO (jsonrpc/0) [api.host] START getAllVmStats() from=3D::1,36114 = (api:46)<br class=3D"">29317:2018-03-21 10:43:06,632+0200 INFO = (jsonrpc/0) [api.host] FINISH getAllVmStats return=3D{'status': = {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} = from=3D::1,36114 (api:52)<br class=3D"">29318:2018-03-21 = 10:43:06,633+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br = class=3D"">29319:2018-03-21 10:43:07,415+0200 INFO (jsonrpc/4) = [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D276c13c7-61ec-4782-a0d9-5cd0f0b0a729 (api:46)<br = class=3D"">29320:2018-03-21 10:43:07,420+0200 INFO (jsonrpc/4) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D276c13c7-61ec-4782-a0d9-5cd0f0b0a729 (api:52)<br = class=3D"">29321:2018-03-21 10:43:07,420+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573)<br class=3D"">29322:2018-03-21 = 10:43:07,427+0200 INFO (jsonrpc/6) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D344b18df-a428-429a-b49f-381f901d1618 (api:46)<br = class=3D"">29323:2018-03-21 10:43:07,434+0200 INFO (jsonrpc/6) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D344b18df-a428-429a-b49f-381f901d1618 (api:52)<br = class=3D"">29324:2018-03-21 10:43:07,435+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573)<br class=3D"">29325:2018-03-21 10:43:13,839+0200 = WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker = name=3Djsonrpc/1 running <Task <JsonRpcTask {'params': = {u'connectionParams': [{u'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524', = u'connection': u'node10.starlett.lv:/vmdata/nfs/exports', u'iqn': u'', = u'user': u'', u'tpgt': u'1', u'protocol_version': u'auto', u'password': = '********', u'port': u''}], u'storagepoolID': = u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': = '2.0', 'method': u'StoragePool.connectStorageServer', 'id': = u'5f98dfd5-998f-44a6-8594-e5f40144b0ed'} at 0x4076150> timeout=3D60, = duration=3D120 at 0x402fe50> task#=3D3754 at 0x353e690>, = traceback:<br class=3D"">29376:2018-03-21 10:43:15,104+0200 INFO = (periodic/2) [vdsm.api] START repoStats(domains=3D()) from=3Dinternal, = task_id=3D672a883e-1d60-4981-9eee-bdcac2ec30c1 (api:46)<br = class=3D"">29377:2018-03-21 10:43:15,105+0200 INFO (periodic/2) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000488777', = 'lastCheck': '5.6', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000481962', 'lastCheck': = '5.4', 'valid': True}} from=3Dinternal, = task_id=3D672a883e-1d60-4981-9eee-bdcac2ec30c1 (api:52)<br = class=3D"">29378:2018-03-21 10:43:15,105+0200 INFO (periodic/2) = [vdsm.api] START multipath_health() from=3Dinternal, = task_id=3D55f1b797-da2a-4b07-affe-5b9e41a74307 (api:46)<br = class=3D"">29379:2018-03-21 10:43:15,106+0200 INFO (periodic/2) = [vdsm.api] FINISH multipath_health return=3D{} from=3Dinternal, = task_id=3D55f1b797-da2a-4b07-affe-5b9e41a74307 (api:52)<br = class=3D"">29380:2018-03-21 10:43:17,645+0200 INFO (jsonrpc/7) = [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D52a6ddeb-3c20-4f3f-b27d-94d19580b473 (api:46)<br = class=3D"">29381:2018-03-21 10:43:17,649+0200 INFO (jsonrpc/7) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D52a6ddeb-3c20-4f3f-b27d-94d19580b473 (api:52)<br = class=3D"">29382:2018-03-21 10:43:17,650+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573)<br class=3D"">29383:2018-03-21 = 10:43:17,656+0200 INFO (jsonrpc/2) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D1a103883-6a6a-4d61-ade6-96559591ddb0 (api:46)<br = class=3D"">29384:2018-03-21 10:43:17,663+0200 INFO (jsonrpc/2) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D1a103883-6a6a-4d61-ade6-96559591ddb0 (api:52)<br = class=3D"">29385:2018-03-21 10:43:17,663+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573)<br class=3D"">29386:2018-03-21 10:43:18,882+0200 = ERROR (jsonrpc/1) [storage.HSM] Could not connect to storageServer = (hsm:2407)<br class=3D"">29406:2018-03-21 10:43:18,883+0200 INFO = (jsonrpc/1) [vdsm.api] FINISH connectStorageServer return=3D{'statuslist':= [{'status': 477, 'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524'}]} = from=3D::ffff:192.168.0.4,49914, flow_id=3D4a53b512, = task_id=3De598cbe0-cde8-4c73-b526-4398df05e67f (api:52)<br = class=3D"">29407:2018-03-21 10:43:18,883+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer = succeeded in 125.04 seconds (__init__:573)<br class=3D"">29408:2018-03-21 = 10:43:20,387+0200 INFO (jsonrpc/5) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46)<br = class=3D"">29409:2018-03-21 10:43:20,388+0200 INFO (jsonrpc/5) = [api.host] FINISH getAllVmStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 = (api:52)<br class=3D"">29410:2018-03-21 10:43:20,388+0200 INFO = (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats = succeeded in 0.01 seconds (__init__:573)<br class=3D"">29411:2018-03-21 = 10:43:21,497+0200 INFO (jsonrpc/3) [api.host] START getStats() = from=3D::ffff:192.168.0.4,49914 (api:46)<br class=3D"">29412:2018-03-21 = 10:43:21,498+0200 INFO (jsonrpc/3) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3Da7469c1f-3d6d-42e7-8798-5c5335da4a9b (api:46)<br = class=3D"">29413:2018-03-21 10:43:21,498+0200 INFO (jsonrpc/3) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000445096', = 'lastCheck': '1.9', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000296586', 'lastCheck': = '1.8', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3Da7469c1f-3d6d-42e7-8798-5c5335da4a9b (api:52)<br = class=3D"">29414:2018-03-21 10:43:21,499+0200 INFO (jsonrpc/3) = [vdsm.api] START multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D9fa5dae3-e2bd-4186-9a1d-afb0aa6e2511 (api:46)<br = class=3D"">29415:2018-03-21 10:43:21,499+0200 INFO (jsonrpc/3) = [vdsm.api] FINISH multipath_health return=3D{} = from=3D::ffff:192.168.0.4,49914, = task_id=3D9fa5dae3-e2bd-4186-9a1d-afb0aa6e2511 (api:52)<br = class=3D"">29416:2018-03-21 10:43:21,505+0200 INFO (jsonrpc/3) = [api.host] FINISH getStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '10': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '13': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.86'}, '12': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', = 'cpuIdle': '99.47'}, '15': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.87'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, = 'cpuSys': '0.33', 'cpuIdle': '99.60'}, '1': {'cpuUser': '1.00', = 'nodeIndex': 1, 'cpuSys': '1.00', 'cpuIdle': '98.00'}, '0': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '3': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.93'}, '2': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.07', = 'cpuIdle': '99.66'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.07', 'cpuIdle': '99.86'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, = 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '7': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '6': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '9': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.80'}, '8': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', = 'cpuIdle': '99.93'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, = 'memFree': '15142'}, '0': {'memPercent': 5, 'memFree': '15351'}}, = 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, = 'vmCount': 0, 'memUsed': '3', 'storageDomains': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, = 'version': 4, 'acquired': True, 'delay': '0.000445096', 'lastCheck': = '1.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.000296586', 'lastCheck': '1.8', 'valid': True}}, = 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'enp3s0f0', = 'tx': '1091295628', 'txDropped': '0', 'rx': '11914875210', 'rxErrors': = '0', 'speed': '100', 'rxDropped': '1442'}, 'ovirtmgmt': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': = 'ovirtmgmt', 'tx': '1048646480', 'txDropped': '0', 'rx': '11615214750', = 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'lo', 'tx': = '58066397668', 'txDropped': '0', 'rx': '58066397668', 'rxErrors': '0', = 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'enp3s0f1', = 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': = '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': = 'down', 'sampleTime': 1521621795.091263, 'name': ';vdsmdummy;', 'tx': = '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621795.091263, 'name': 'enp4s0f0', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621795.091263, 'name': 'enp4s0f1', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', = 'ksmPages': 100, 'elapsedTime': '76360.34', 'cpuLoad': '0.15', 'cpuSys': = '0.18', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': = {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31031, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1442', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:43:21 GMT', 'cpuUser': '0.15', = 'memFree': 31287, 'cpuIdle': '99.67', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.67'}} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29417:2018-03-21 10:43:21,508+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.02 seconds = (__init__:573)<br class=3D"">29418:2018-03-21 10:43:21,651+0200 = INFO (jsonrpc/0) [api.host] START getAllVmStats() from=3D::1,36114 = (api:46)<br class=3D"">29419:2018-03-21 10:43:21,651+0200 INFO = (jsonrpc/0) [api.host] FINISH getAllVmStats return=3D{'status': = {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} = from=3D::1,36114 (api:52)<br class=3D"">29420:2018-03-21 = 10:43:21,652+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br = class=3D"">29421:2018-03-21 10:43:27,764+0200 INFO (jsonrpc/4) = [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D46d19cab-8903-4c52-9bb2-4dd8f370997e (api:46)<br = class=3D"">29422:2018-03-21 10:43:27,769+0200 INFO (jsonrpc/4) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D46d19cab-8903-4c52-9bb2-4dd8f370997e (api:52)<br = class=3D"">29423:2018-03-21 10:43:27,770+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573)<br class=3D"">29424:2018-03-21 = 10:43:27,815+0200 INFO (jsonrpc/6) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D290acc73-8f25-4fd7-9263-7bd05d46430a (api:46)<br = class=3D"">29425:2018-03-21 10:43:27,822+0200 INFO (jsonrpc/6) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D290acc73-8f25-4fd7-9263-7bd05d46430a (api:52)<br = class=3D"">29426:2018-03-21 10:43:27,823+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573)<br class=3D"">29427:2018-03-21 10:43:30,121+0200 = INFO (periodic/3) [vdsm.api] START repoStats(domains=3D()) = from=3Dinternal, task_id=3D0abdebcf-2b0b-4d33-b552-c0f2ed3e567b = (api:46)<br class=3D"">29428:2018-03-21 10:43:30,122+0200 INFO = (periodic/3) [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000402077', = 'lastCheck': '0.6', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': = '0.5', 'valid': True}} from=3Dinternal, = task_id=3D0abdebcf-2b0b-4d33-b552-c0f2ed3e567b (api:52)<br = class=3D"">29429:2018-03-21 10:43:30,122+0200 INFO (periodic/3) = [vdsm.api] START multipath_health() from=3Dinternal, = task_id=3Dacde7e7d-ab4a-4811-934f-4d312da33208 (api:46)<br = class=3D"">29430:2018-03-21 10:43:30,123+0200 INFO (periodic/3) = [vdsm.api] FINISH multipath_health return=3D{} from=3Dinternal, = task_id=3Dacde7e7d-ab4a-4811-934f-4d312da33208 (api:52)<br = class=3D"">29431:2018-03-21 10:43:35,393+0200 INFO (jsonrpc/7) = [api.host] START getAllVmStats() from=3D::ffff:192.168.0.4,49914 = (api:46)<br class=3D"">29432:2018-03-21 10:43:35,394+0200 INFO = (jsonrpc/7) [api.host] FINISH getAllVmStats return=3D{'status': = {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} = from=3D::ffff:192.168.0.4,49914 (api:52)<br class=3D"">29433:2018-03-21 = 10:43:35,394+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br = class=3D"">29434:2018-03-21 10:43:36,671+0200 INFO (jsonrpc/2) = [api.host] START getAllVmStats() from=3D::1,36114 (api:46)<br = class=3D"">29435:2018-03-21 10:43:36,671+0200 INFO (jsonrpc/2) = [api.host] FINISH getAllVmStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'statsList': (suppressed)} from=3D::1,36114 (api:52)<br = class=3D"">29436:2018-03-21 10:43:36,672+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29437:2018-03-21 10:43:37,356+0200 = INFO (jsonrpc/1) [api.host] START getStats() = from=3D::ffff:192.168.0.4,49914 (api:46)<br class=3D"">29438:2018-03-21 = 10:43:37,357+0200 INFO (jsonrpc/1) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3Dbc14e253-d3d4-4cc3-9ae1-c0e6ac6e37f7 (api:46)<br = class=3D"">29439:2018-03-21 10:43:37,357+0200 INFO (jsonrpc/1) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000402077', = 'lastCheck': '7.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': = '7.7', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3Dbc14e253-d3d4-4cc3-9ae1-c0e6ac6e37f7 (api:52)<br = class=3D"">29440:2018-03-21 10:43:37,358+0200 INFO (jsonrpc/1) = [vdsm.api] START multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D23c698b8-d5bd-4746-b738-97b10f3e4bfb (api:46)<br = class=3D"">29441:2018-03-21 10:43:37,358+0200 INFO (jsonrpc/1) = [vdsm.api] FINISH multipath_health return=3D{} = from=3D::ffff:192.168.0.4,49914, = task_id=3D23c698b8-d5bd-4746-b738-97b10f3e4bfb (api:52)<br = class=3D"">29442:2018-03-21 10:43:37,365+0200 INFO (jsonrpc/1) = [api.host] FINISH getStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '10': {'cpuUser': = '3.06', 'nodeIndex': 0, 'cpuSys': '0.80', 'cpuIdle': '96.14'}, '13': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.93'}, '12': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', = 'cpuIdle': '99.73'}, '15': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.87'}, '14': {'cpuUser': '0.27', 'nodeIndex': 0, = 'cpuSys': '0.40', 'cpuIdle': '99.33'}, '1': {'cpuUser': '1.13', = 'nodeIndex': 1, 'cpuSys': '1.33', 'cpuIdle': '97.54'}, '0': {'cpuUser': = '0.47', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.20'}, '3': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.93'}, '2': {'cpuUser': '0.67', 'nodeIndex': 0, 'cpuSys': '0.40', = 'cpuIdle': '98.93'}, '5': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.74'}, '4': {'cpuUser': '0.47', 'nodeIndex': 0, = 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '7': {'cpuUser': '0.33', = 'nodeIndex': 1, 'cpuSys': '0.73', 'cpuIdle': '98.94'}, '6': {'cpuUser': = '5.39', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '94.14'}, '9': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '99.66'}, '8': {'cpuUser': '0.53', 'nodeIndex': 0, 'cpuSys': '0.40', = 'cpuIdle': '99.07'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, = 'memFree': '15143'}, '0': {'memPercent': 5, 'memFree': '15348'}}, = 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, = 'vmCount': 0, 'memUsed': '3', 'storageDomains': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, = 'version': 4, 'acquired': True, 'delay': '0.000402077', 'lastCheck': = '7.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.000344445', 'lastCheck': '7.7', 'valid': True}}, = 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'enp3s0f0', = 'tx': '1091305295', 'txDropped': '0', 'rx': '11914880953', 'rxErrors': = '0', 'speed': '100', 'rxDropped': '1442'}, 'ovirtmgmt': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': = 'ovirtmgmt', 'tx': '1048655897', 'txDropped': '0', 'rx': '11615220341', = 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'lo', 'tx': = '58078891247', 'txDropped': '0', 'rx': '58078891247', 'rxErrors': '0', = 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'enp3s0f1', = 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': = '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': = 'down', 'sampleTime': 1521621810.108511, 'name': ';vdsmdummy;', 'tx': = '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621810.108511, 'name': 'enp4s0f0', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621810.108511, 'name': 'enp4s0f1', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', = 'ksmPages': 100, 'elapsedTime': '76376.19', 'cpuLoad': '0.16', 'cpuSys': = '0.39', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': = {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.00', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1442', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:43:37 GMT', 'cpuUser': '0.79', = 'memFree': 31336, 'cpuIdle': '98.82', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.73'}} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29443:2018-03-21 10:43:37,367+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573)<br class=3D"">29444:2018-03-21 10:43:38,124+0200 = INFO (jsonrpc/5) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D9d2a82dd-c76c-4eb5-a1e1-b49bf004384e (api:46)<br = class=3D"">29445:2018-03-21 10:43:38,128+0200 INFO (jsonrpc/5) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D9d2a82dd-c76c-4eb5-a1e1-b49bf004384e (api:52)<br = class=3D"">29446:2018-03-21 10:43:38,129+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573)<br class=3D"">29447:2018-03-21 = 10:43:38,171+0200 INFO (jsonrpc/3) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D62d44b0b-20b2-47ae-bfb8-d4717e301b69 (api:46)<br = class=3D"">29448:2018-03-21 10:43:38,177+0200 INFO (jsonrpc/3) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D62d44b0b-20b2-47ae-bfb8-d4717e301b69 (api:52)<br = class=3D"">29449:2018-03-21 10:43:38,178+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573)<br class=3D"">29450:2018-03-21 10:43:45,147+0200 = INFO (periodic/2) [vdsm.api] START repoStats(domains=3D()) = from=3Dinternal, task_id=3D49c33285-a222-464d-8984-4e75a5c6354a = (api:46)<br class=3D"">29451:2018-03-21 10:43:45,148+0200 INFO = (periodic/2) [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000454102', = 'lastCheck': '5.6', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000347381', 'lastCheck': = '5.5', 'valid': True}} from=3Dinternal, = task_id=3D49c33285-a222-464d-8984-4e75a5c6354a (api:52)<br = class=3D"">29452:2018-03-21 10:43:45,148+0200 INFO (periodic/2) = [vdsm.api] START multipath_health() from=3Dinternal, = task_id=3Deed6b564-76b1-4f9b-80a2-1f0f6d9bbe32 (api:46)<br = class=3D"">29453:2018-03-21 10:43:45,149+0200 INFO (periodic/2) = [vdsm.api] FINISH multipath_health return=3D{} from=3Dinternal, = task_id=3Deed6b564-76b1-4f9b-80a2-1f0f6d9bbe32 (api:52)<br = class=3D"">29454:2018-03-21 10:43:48,405+0200 INFO (jsonrpc/0) = [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D84651a37-7d13-41a9-9dcf-2169a9c46fc5 (api:46)<br = class=3D"">29455:2018-03-21 10:43:48,410+0200 INFO (jsonrpc/0) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D84651a37-7d13-41a9-9dcf-2169a9c46fc5 (api:52)<br = class=3D"">29456:2018-03-21 10:43:48,411+0200 INFO (jsonrpc/0) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573)<br class=3D"">29457:2018-03-21 = 10:43:48,417+0200 INFO (jsonrpc/4) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D4ebd1d1a-11ac-41ee-b459-7e45f008776b (api:46)<br = class=3D"">29458:2018-03-21 10:43:48,423+0200 INFO (jsonrpc/4) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D4ebd1d1a-11ac-41ee-b459-7e45f008776b (api:52)<br = class=3D"">29459:2018-03-21 10:43:48,424+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573)<br class=3D"">29460:2018-03-21 10:43:50,399+0200 = INFO (jsonrpc/6) [api.host] START getAllVmStats() = from=3D::ffff:192.168.0.4,49914 (api:46)<br class=3D"">29461:2018-03-21 = 10:43:50,400+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats = return=3D{'status': {'message': 'Done', 'code': 0}, 'statsList': = (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29462:2018-03-21 10:43:50,400+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29463:2018-03-21 10:43:51,694+0200 = INFO (jsonrpc/7) [api.host] START getAllVmStats() from=3D::1,36114 = (api:46)<br class=3D"">29464:2018-03-21 10:43:51,694+0200 INFO = (jsonrpc/7) [api.host] FINISH getAllVmStats return=3D{'status': = {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} = from=3D::1,36114 (api:52)<br class=3D"">29465:2018-03-21 = 10:43:51,695+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br = class=3D"">29466:2018-03-21 10:43:52,516+0200 INFO (jsonrpc/2) = [api.host] START getStats() from=3D::ffff:192.168.0.4,49914 (api:46)<br = class=3D"">29467:2018-03-21 10:43:52,517+0200 INFO (jsonrpc/2) = [vdsm.api] START repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D0a88f0dc-eccf-4d24-9ac2-22e079a1480a (api:46)<br = class=3D"">29468:2018-03-21 10:43:52,517+0200 INFO (jsonrpc/2) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000487937', = 'lastCheck': '3.0', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000455956', 'lastCheck': = '2.9', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D0a88f0dc-eccf-4d24-9ac2-22e079a1480a (api:52)<br = class=3D"">29469:2018-03-21 10:43:52,518+0200 INFO (jsonrpc/2) = [vdsm.api] START multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3Da7a44ff7-a137-41b6-9ad5-2c0b32ab79ec (api:46)<br = class=3D"">29470:2018-03-21 10:43:52,518+0200 INFO (jsonrpc/2) = [vdsm.api] FINISH multipath_health return=3D{} = from=3D::ffff:192.168.0.4,49914, = task_id=3Da7a44ff7-a137-41b6-9ad5-2c0b32ab79ec (api:52)<br = class=3D"">29471:2018-03-21 10:43:52,524+0200 INFO (jsonrpc/2) = [api.host] FINISH getStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '10': {'cpuUser': = '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '13': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.93'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', = 'cpuIdle': '99.87'}, '15': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.07', 'cpuIdle': '99.86'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, = 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '1': {'cpuUser': '1.13', = 'nodeIndex': 1, 'cpuSys': '1.13', 'cpuIdle': '97.74'}, '0': {'cpuUser': = '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '3': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': = '99.93'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', = 'cpuIdle': '99.87'}, '5': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.87'}, '4': {'cpuUser': '0.27', 'nodeIndex': 0, = 'cpuSys': '0.13', 'cpuIdle': '99.60'}, '7': {'cpuUser': '0.00', = 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '99.67'}, '6': {'cpuUser': = '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '9': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.80'}, '8': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', = 'cpuIdle': '99.80'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, = 'memFree': '15143'}, '0': {'memPercent': 5, 'memFree': '15349'}}, = 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, = 'vmCount': 0, 'memUsed': '3', 'storageDomains': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, = 'version': 4, 'acquired': True, 'delay': '0.000487937', 'lastCheck': = '3.0', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.000455956', 'lastCheck': '2.9', 'valid': True}}, = 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'enp3s0f0', 'tx': = '1091312739', 'txDropped': '0', 'rx': '11914899051', 'rxErrors': '0', = 'speed': '100', 'rxDropped': '1443'}, 'ovirtmgmt': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'ovirtmgmt', = 'tx': '1048663107', 'txDropped': '0', 'rx': '11615236520', 'rxErrors': = '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'lo', 'tx': = '58090295494', 'txDropped': '0', 'rx': '58090295494', 'rxErrors': '0', = 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'enp3s0f1', 'tx': = '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621825.13588, 'name': ';vdsmdummy;', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621825.13588, 'name': 'enp4s0f0', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621825.13588, 'name': 'enp4s0f1', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', = 'ksmPages': 100, 'elapsedTime': '76391.35', 'cpuLoad': '0.15', 'cpuSys': = '0.20', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': = {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.13', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1443', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:43:52 GMT', 'cpuUser': '0.14', = 'memFree': 31336, 'cpuIdle': '99.66', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.73'}} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29472:2018-03-21 10:43:52,526+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573)<br class=3D"">29473:2018-03-21 10:43:58,686+0200 = INFO (jsonrpc/1) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3Dd6d29cf6-c083-4639-91a2-f062eae4e629 (api:46)<br = class=3D"">29474:2018-03-21 10:43:58,690+0200 INFO (jsonrpc/1) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3Dd6d29cf6-c083-4639-91a2-f062eae4e629 (api:52)<br = class=3D"">29475:2018-03-21 10:43:58,690+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573)<br class=3D"">29476:2018-03-21 = 10:43:58,697+0200 INFO (jsonrpc/5) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D9dbd91df-dd9c-469c-8308-bfbfc228937c (api:46)<br = class=3D"">29477:2018-03-21 10:43:58,703+0200 INFO (jsonrpc/5) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D9dbd91df-dd9c-469c-8308-bfbfc228937c (api:52)<br = class=3D"">29478:2018-03-21 10:43:58,704+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573)<br class=3D""><br class=3D""><br class=3D""><br = class=3D""><b class=3D""><font color=3D"#942192" class=3D"">grep -n = 10:46 vdsm.log | tail -1000</font></b><br class=3D"">1017:2018-03-21 = 06:10:46,882+0200 INFO (jsonrpc/2) [api.host] START getStats() = from=3D::ffff:192.168.0.4,49914 (api:46)<br class=3D"">1018:2018-03-21 = 06:10:46,883+0200 INFO (jsonrpc/2) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D6e045b00-88b2-469b-80d6-af38303f8c32 (api:46)<br = class=3D"">1019:2018-03-21 06:10:46,883+0200 INFO (jsonrpc/2) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000332785', = 'lastCheck': '7.3', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000203914', 'lastCheck': = '5.6', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D6e045b00-88b2-469b-80d6-af38303f8c32 (api:52)<br = class=3D"">1020:2018-03-21 06:10:46,884+0200 INFO (jsonrpc/2) = [vdsm.api] START multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D475a5b40-4a43-42ee-9524-2bdd220aed83 (api:46)<br = class=3D"">1021:2018-03-21 06:10:46,884+0200 INFO (jsonrpc/2) = [vdsm.api] FINISH multipath_health return=3D{} = from=3D::ffff:192.168.0.4,49914, = task_id=3D475a5b40-4a43-42ee-9524-2bdd220aed83 (api:52)<br = class=3D"">1022:2018-03-21 06:10:46,889+0200 INFO (jsonrpc/2) = [api.host] FINISH getStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '10': {'cpuUser': = '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '13': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '99.66'}, '12': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', = 'cpuIdle': '99.47'}, '15': {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': = '0.27', 'cpuIdle': '99.40'}, '14': {'cpuUser': '0.47', 'nodeIndex': 0, = 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '1': {'cpuUser': '3.00', = 'nodeIndex': 1, 'cpuSys': '1.40', 'cpuIdle': '95.60'}, '0': {'cpuUser': = '0.73', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.07'}, '3': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.86'}, '2': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.27', = 'cpuIdle': '99.66'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.20', 'cpuIdle': '99.73'}, '4': {'cpuUser': '0.13', 'nodeIndex': 0, = 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '7': {'cpuUser': '0.80', = 'nodeIndex': 1, 'cpuSys': '0.80', 'cpuIdle': '98.40'}, '6': {'cpuUser': = '5.46', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '94.27'}, '9': = {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '99.53'}, '8': {'cpuUser': '1.27', 'nodeIndex': 0, 'cpuSys': '0.47', = 'cpuIdle': '98.26'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, = 'memFree': '15128'}, '0': {'memPercent': 5, 'memFree': '15366'}}, = 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, = 'vmCount': 0, 'memUsed': '3', 'storageDomains': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, = 'version': 4, 'acquired': True, 'delay': '0.000332785', 'lastCheck': = '7.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.000203914', 'lastCheck': '5.6', 'valid': True}}, = 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'enp3s0f0', = 'tx': '1082264809', 'txDropped': '0', 'rx': '11907188594', 'rxErrors': = '0', 'speed': '100', 'rxDropped': '289'}, 'ovirtmgmt': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'ovirtmgmt', = 'tx': '1039869554', 'txDropped': '0', 'rx': '11608697383', 'rxErrors': = '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'lo', 'tx': = '45587320171', 'txDropped': '0', 'rx': '45587320171', 'rxErrors': '0', = 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'enp3s0f1', = 'tx': '0', 'txDropped': '0', 'rx': '53620', 'rxErrors': '0', 'speed': = '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': = 'down', 'sampleTime': 1521605437.203776, 'name': ';vdsmdummy;', 'tx': = '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521605437.203776, 'name': 'enp4s0f0', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521605437.203776, 'name': 'enp4s0f1', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', = 'ksmPages': 100, 'elapsedTime': '60005.72', 'cpuLoad': '0.06', 'cpuSys': = '0.38', 'diskStats': {'/var/log': {'free': '7357'}, '/var/run/vdsm/': = {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31084, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '289', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T04:10:46 GMT', 'cpuUser': '0.83', = 'memFree': 31340, 'cpuIdle': '98.79', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.80'}} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">1023:2018-03-21 06:10:46,891+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573)<br class=3D"">13522:2018-03-21 08:10:46,827+0200 = INFO (jsonrpc/7) [api.host] START getAllVmStats() = from=3D::ffff:192.168.0.4,49914 (api:46)<br class=3D"">13523:2018-03-21 = 08:10:46,828+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats = return=3D{'status': {'message': 'Done', 'code': 0}, 'statsList': = (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">13524:2018-03-21 08:10:46,828+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29734:2018-03-21 10:46:00,338+0200 = INFO (periodic/2) [vdsm.api] START repoStats(domains=3D()) = from=3Dinternal, task_id=3Daf099ce6-0f44-461b-9e8e-99de63b3884f = (api:46)<br class=3D"">29735:2018-03-21 10:46:00,339+0200 INFO = (periodic/2) [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000470245', = 'lastCheck': '0.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000476935', 'lastCheck': = '0.7', 'valid': True}} from=3Dinternal, = task_id=3Daf099ce6-0f44-461b-9e8e-99de63b3884f (api:52)<br = class=3D"">29736:2018-03-21 10:46:00,339+0200 INFO (periodic/2) = [vdsm.api] START multipath_health() from=3Dinternal, = task_id=3D4a5709d9-e322-44eb-a8a4-bbe3f5e5e1bd (api:46)<br = class=3D"">29737:2018-03-21 10:46:00,340+0200 INFO (periodic/2) = [vdsm.api] FINISH multipath_health return=3D{} from=3Dinternal, = task_id=3D4a5709d9-e322-44eb-a8a4-bbe3f5e5e1bd (api:52)<br = class=3D"">29738:2018-03-21 10:46:04,637+0200 INFO (jsonrpc/7) = [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D5b9300f7-f8c4-4f63-a49c-45ca8119d3b9 (api:46)<br = class=3D"">29739:2018-03-21 10:46:04,642+0200 INFO (jsonrpc/7) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D5b9300f7-f8c4-4f63-a49c-45ca8119d3b9 (api:52)<br = class=3D"">29740:2018-03-21 10:46:04,642+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573)<br class=3D"">29741:2018-03-21 = 10:46:04,648+0200 INFO (jsonrpc/1) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3Df4db5bfa-e131-4f9a-baaa-e7555c996ab0 (api:46)<br = class=3D"">29742:2018-03-21 10:46:04,655+0200 INFO (jsonrpc/1) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3Df4db5bfa-e131-4f9a-baaa-e7555c996ab0 (api:52)<br = class=3D"">29743:2018-03-21 10:46:04,656+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29744:2018-03-21 10:46:05,451+0200 = INFO (jsonrpc/5) [api.host] START getAllVmStats() = from=3D::ffff:192.168.0.4,49914 (api:46)<br class=3D"">29745:2018-03-21 = 10:46:05,452+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats = return=3D{'status': {'message': 'Done', 'code': 0}, 'statsList': = (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29746:2018-03-21 10:46:05,452+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29747:2018-03-21 10:46:06,903+0200 = INFO (jsonrpc/3) [api.host] START getAllVmStats() from=3D::1,36114 = (api:46)<br class=3D"">29748:2018-03-21 10:46:06,904+0200 INFO = (jsonrpc/3) [api.host] FINISH getAllVmStats return=3D{'status': = {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} = from=3D::1,36114 (api:52)<br class=3D"">29749:2018-03-21 = 10:46:06,904+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br = class=3D"">29750:2018-03-21 10:46:08,791+0200 WARN = (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=3Djsonrpc/6 = running <Task <JsonRpcTask {'params': {u'connectionParams': = [{u'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524', u'connection': = u'node10.starlett.lv:/vmdata/nfs/exports', u'iqn': u'', u'user': u'', = u'tpgt': u'1', u'protocol_version': u'auto', u'password': '********', = u'port': u''}], u'storagepoolID': = u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': = '2.0', 'method': u'StoragePool.connectStorageServer', 'id': = u'71143eef-aac6-4996-9783-3e0e3da180c3'} at 0x3604fd0> timeout=3D60, = duration=3D120 at 0x3604e10> task#=3D3762 at 0x3541650>, = traceback:<br class=3D"">29801:2018-03-21 10:46:12,852+0200 INFO = (jsonrpc/0) [api.host] START getStats() from=3D::ffff:192.168.0.4,49914 = (api:46)<br class=3D"">29802:2018-03-21 10:46:12,853+0200 INFO = (jsonrpc/0) [vdsm.api] START repoStats(domains=3D()) = from=3D::ffff:192.168.0.4,49914, = task_id=3D391fdab9-d3c2-4075-9c34-177a07a21ec3 (api:46)<br = class=3D"">29803:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000492277', = 'lastCheck': '3.3', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': = '3.2', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D391fdab9-d3c2-4075-9c34-177a07a21ec3 (api:52)<br = class=3D"">29804:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) = [vdsm.api] START multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D7e53adbc-3974-4f51-bfc8-8fbd3ca6b749 (api:46)<br = class=3D"">29805:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) = [vdsm.api] FINISH multipath_health return=3D{} = from=3D::ffff:192.168.0.4,49914, = task_id=3D7e53adbc-3974-4f51-bfc8-8fbd3ca6b749 (api:52)<br = class=3D"">29806:2018-03-21 10:46:12,861+0200 INFO (jsonrpc/0) = [api.host] FINISH getStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.47', = 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '10': {'cpuUser': = '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '13': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': = '99.73'}, '12': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.27', = 'cpuIdle': '99.53'}, '15': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': = '0.27', 'cpuIdle': '99.60'}, '14': {'cpuUser': '0.13', 'nodeIndex': 0, = 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '1': {'cpuUser': '1.00', = 'nodeIndex': 1, 'cpuSys': '1.33', 'cpuIdle': '97.67'}, '0': {'cpuUser': = '0.13', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '3': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': = '99.93'}, '2': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.33', = 'cpuIdle': '99.47'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.80'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, = 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '7': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.53'}, '6': {'cpuUser': = '0.47', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '9': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': = '99.73'}, '8': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', = 'cpuIdle': '99.54'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, = 'memFree': '15142'}, '0': {'memPercent': 5, 'memFree': '15348'}}, = 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, = 'vmCount': 0, 'memUsed': '3', 'storageDomains': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, = 'version': 4, 'acquired': True, 'delay': '0.000492277', 'lastCheck': = '3.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.000479153', 'lastCheck': '3.2', 'valid': True}}, = 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'enp3s0f0', = 'tx': '1091387790', 'txDropped': '0', 'rx': '11914959714', 'rxErrors': = '0', 'speed': '100', 'rxDropped': '1451'}, 'ovirtmgmt': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': = 'ovirtmgmt', 'tx': '1048735814', 'txDropped': '0', 'rx': '11615289173', = 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'lo', 'tx': = '58192068637', 'txDropped': '0', 'rx': '58192068637', 'rxErrors': '0', = 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'enp3s0f1', = 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': = '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': = 'down', 'sampleTime': 1521621960.326184, 'name': ';vdsmdummy;', 'tx': = '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621960.326184, 'name': 'enp4s0f0', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621960.326184, 'name': 'enp4s0f1', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', = 'ksmPages': 100, 'elapsedTime': '76531.69', 'cpuLoad': '0.17', 'cpuSys': = '0.32', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': = {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.00', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31081, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1451', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:46:12 GMT', 'cpuUser': '0.21', = 'memFree': 31337, 'cpuIdle': '99.46', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.80'}} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29807:2018-03-21 10:46:12,863+0200 INFO (jsonrpc/0) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573)<br class=3D"">29808:2018-03-21 10:46:13,829+0200 ERROR = (jsonrpc/6) [storage.HSM] Could not connect to storageServer = (hsm:2407)<br class=3D"">29828:2018-03-21 10:46:13,829+0200 INFO = (jsonrpc/6) [vdsm.api] FINISH connectStorageServer return=3D{'statuslist':= [{'status': 477, 'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524'}]} = from=3D::ffff:192.168.0.4,49914, flow_id=3D1435fc81, = task_id=3D0a828d2c-d9f4-4f83-a9e9-7393159d5323 (api:52)<br = class=3D"">29829:2018-03-21 10:46:13,830+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer = succeeded in 125.04 seconds (__init__:573)<br class=3D"">29830:2018-03-21 = 10:46:14,767+0200 INFO (jsonrpc/4) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D49cd2ae6-7943-47c9-ad86-9e0b7e58bca3 (api:46)<br = class=3D"">29831:2018-03-21 10:46:14,772+0200 INFO (jsonrpc/4) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D49cd2ae6-7943-47c9-ad86-9e0b7e58bca3 (api:52)<br = class=3D"">29832:2018-03-21 10:46:14,773+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573)<br class=3D"">29833:2018-03-21 = 10:46:14,815+0200 INFO (jsonrpc/2) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D19909616-efb4-490d-982c-66eda9ca4381 (api:46)<br = class=3D"">29834:2018-03-21 10:46:14,822+0200 INFO (jsonrpc/2) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D19909616-efb4-490d-982c-66eda9ca4381 (api:52)<br = class=3D"">29835:2018-03-21 10:46:14,823+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573)<br class=3D"">29836:2018-03-21 10:46:15,360+0200 = INFO (periodic/2) [vdsm.api] START repoStats(domains=3D()) = from=3Dinternal, task_id=3Dda74ea9f-164b-450d-b836-6818caa3fdc5 = (api:46)<br class=3D"">29837:2018-03-21 10:46:15,360+0200 INFO = (periodic/2) [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000492277', = 'lastCheck': '5.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': = '5.7', 'valid': True}} from=3Dinternal, = task_id=3Dda74ea9f-164b-450d-b836-6818caa3fdc5 (api:52)<br = class=3D"">29838:2018-03-21 10:46:15,361+0200 INFO (periodic/2) = [vdsm.api] START multipath_health() from=3Dinternal, = task_id=3D06245e2b-e8ca-41c4-90bf-4294d0c699b8 (api:46)<br = class=3D"">29839:2018-03-21 10:46:15,361+0200 INFO (periodic/2) = [vdsm.api] FINISH multipath_health return=3D{} from=3Dinternal, = task_id=3D06245e2b-e8ca-41c4-90bf-4294d0c699b8 (api:52)<br = class=3D"">29840:2018-03-21 10:46:20,456+0200 INFO (jsonrpc/7) = [api.host] START getAllVmStats() from=3D::ffff:192.168.0.4,49914 = (api:46)<br class=3D"">29841:2018-03-21 10:46:20,457+0200 INFO = (jsonrpc/7) [api.host] FINISH getAllVmStats return=3D{'status': = {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} = from=3D::ffff:192.168.0.4,49914 (api:52)<br class=3D"">29842:2018-03-21 = 10:46:20,457+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573)<br = class=3D"">29843:2018-03-21 10:46:21,925+0200 INFO (jsonrpc/1) = [api.host] START getAllVmStats() from=3D::1,36114 (api:46)<br = class=3D"">29844:2018-03-21 10:46:21,926+0200 INFO (jsonrpc/1) = [api.host] FINISH getAllVmStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'statsList': (suppressed)} from=3D::1,36114 (api:52)<br = class=3D"">29845:2018-03-21 10:46:21,926+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29846:2018-03-21 10:46:24,930+0200 = INFO (jsonrpc/5) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3Dd964535d-e5eb-42bc-b324-432a05c364da (api:46)<br = class=3D"">29847:2018-03-21 10:46:24,935+0200 INFO (jsonrpc/5) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3Dd964535d-e5eb-42bc-b324-432a05c364da (api:52)<br = class=3D"">29848:2018-03-21 10:46:24,935+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.00 seconds (__init__:573)<br class=3D"">29849:2018-03-21 = 10:46:24,987+0200 INFO (jsonrpc/3) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D167dcd23-04b5-4de3-9c81-7d25ba56407a (api:46)<br = class=3D"">29850:2018-03-21 10:46:25,022+0200 INFO (jsonrpc/3) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D167dcd23-04b5-4de3-9c81-7d25ba56407a (api:52)<br = class=3D"">29851:2018-03-21 10:46:25,023+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.03 = seconds (__init__:573)<br class=3D"">29852:2018-03-21 10:46:28,010+0200 = INFO (jsonrpc/0) [api.host] START getStats() = from=3D::ffff:192.168.0.4,49914 (api:46)<br class=3D"">29853:2018-03-21 = 10:46:28,011+0200 INFO (jsonrpc/0) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D011616ce-53c9-4cfc-8b29-247b1be03409 (api:46)<br = class=3D"">29854:2018-03-21 10:46:28,011+0200 INFO (jsonrpc/0) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000218955', = 'lastCheck': '8.4', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000351741', 'lastCheck': = '8.4', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D011616ce-53c9-4cfc-8b29-247b1be03409 (api:52)<br = class=3D"">29855:2018-03-21 10:46:28,012+0200 INFO (jsonrpc/0) = [vdsm.api] START multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D85b42a8d-1bf1-495d-ad32-a1f9710a6468 (api:46)<br = class=3D"">29856:2018-03-21 10:46:28,012+0200 INFO (jsonrpc/0) = [vdsm.api] FINISH multipath_health return=3D{} = from=3D::ffff:192.168.0.4,49914, = task_id=3D85b42a8d-1bf1-495d-ad32-a1f9710a6468 (api:52)<br = class=3D"">29857:2018-03-21 10:46:28,018+0200 INFO (jsonrpc/0) = [api.host] FINISH getStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.13', = 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '10': {'cpuUser': = '1.73', 'nodeIndex': 0, 'cpuSys': '0.53', 'cpuIdle': '97.74'}, '13': = {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '99.53'}, '12': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', = 'cpuIdle': '99.73'}, '15': {'cpuUser': '0.67', 'nodeIndex': 1, 'cpuSys': = '0.27', 'cpuIdle': '99.06'}, '14': {'cpuUser': '0.20', 'nodeIndex': 0, = 'cpuSys': '0.33', 'cpuIdle': '99.47'}, '1': {'cpuUser': '1.26', = 'nodeIndex': 1, 'cpuSys': '1.33', 'cpuIdle': '97.41'}, '0': {'cpuUser': = '0.40', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.53'}, '3': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.80'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.27', = 'cpuIdle': '99.73'}, '5': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.87'}, '4': {'cpuUser': '13.78', 'nodeIndex': 0, = 'cpuSys': '0.80', 'cpuIdle': '85.42'}, '7': {'cpuUser': '0.47', = 'nodeIndex': 1, 'cpuSys': '0.67', 'cpuIdle': '98.86'}, '6': {'cpuUser': = '1.07', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '98.66'}, '9': = {'cpuUser': '1.20', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '98.53'}, '8': {'cpuUser': '0.80', 'nodeIndex': 0, 'cpuSys': '0.33', = 'cpuIdle': '98.87'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, = 'memFree': '15136'}, '0': {'memPercent': 5, 'memFree': '15357'}}, = 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, = 'vmCount': 0, 'memUsed': '3', 'storageDomains': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, = 'version': 4, 'acquired': True, 'delay': '0.000218955', 'lastCheck': = '8.4', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.000351741', 'lastCheck': '8.4', 'valid': True}}, = 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'enp3s0f0', = 'tx': '1091397599', 'txDropped': '0', 'rx': '11914966591', 'rxErrors': = '0', 'speed': '100', 'rxDropped': '1454'}, 'ovirtmgmt': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': = 'ovirtmgmt', 'tx': '1048745347', 'txDropped': '0', 'rx': '11615294897', = 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'lo', 'tx': = '58205583384', 'txDropped': '0', 'rx': '58205583384', 'rxErrors': '0', = 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'enp3s0f1', = 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': = '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': = 'down', 'sampleTime': 1521621975.346633, 'name': ';vdsmdummy;', 'tx': = '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621975.346633, 'name': 'enp4s0f0', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621975.346633, 'name': 'enp4s0f1', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', = 'ksmPages': 100, 'elapsedTime': '76546.85', 'cpuLoad': '0.18', 'cpuSys': = '0.39', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': = {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.20', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31031, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1454', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:46:28 GMT', 'cpuUser': '1.38', = 'memFree': 31287, 'cpuIdle': '98.23', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.87'}} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29858:2018-03-21 10:46:28,020+0200 INFO (jsonrpc/0) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573)<br class=3D"">29859:2018-03-21 10:46:29,667+0200 = INFO (itmap/0) [IOProcessClient] Starting client ioprocess-575 = (__init__:308)<br class=3D"">29860:2018-03-21 10:46:29,700+0200 = INFO (itmap/1) [IOProcessClient] Starting client ioprocess-576 = (__init__:308)<br class=3D"">29861:2018-03-21 10:46:29,714+0200 = INFO (ioprocess/1711202) [IOProcess] Starting ioprocess = (__init__:437)<br class=3D"">29862:2018-03-21 10:46:29,725+0200 = INFO (ioprocess/1711208) [IOProcess] Starting ioprocess = (__init__:437)<br class=3D"">29863:2018-03-21 10:46:30,375+0200 = INFO (periodic/0) [vdsm.api] START repoStats(domains=3D()) = from=3Dinternal, task_id=3Dbdacf1c0-3fcb-43b1-82e2-a041a26d0d7f = (api:46)<br class=3D"">29864:2018-03-21 10:46:30,376+0200 INFO = (periodic/0) [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.00031111', = 'lastCheck': '0.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.00045594', 'lastCheck': = '0.6', 'valid': True}} from=3Dinternal, = task_id=3Dbdacf1c0-3fcb-43b1-82e2-a041a26d0d7f (api:52)<br = class=3D"">29865:2018-03-21 10:46:30,376+0200 INFO (periodic/0) = [vdsm.api] START multipath_health() from=3Dinternal, = task_id=3D3d3cad70-e5c8-49b2-9f5d-92a249cc102d (api:46)<br = class=3D"">29866:2018-03-21 10:46:30,377+0200 INFO (periodic/0) = [vdsm.api] FINISH multipath_health return=3D{} from=3Dinternal, = task_id=3D3d3cad70-e5c8-49b2-9f5d-92a249cc102d (api:52)<br = class=3D"">29867:2018-03-21 10:46:35,108+0200 INFO (jsonrpc/6) = [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D304d7e7c-03fc-44d6-be32-d083ada09b30 (api:46)<br = class=3D"">29868:2018-03-21 10:46:35,114+0200 INFO (jsonrpc/6) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D304d7e7c-03fc-44d6-be32-d083ada09b30 (api:52)<br = class=3D"">29869:2018-03-21 10:46:35,114+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.00 seconds (__init__:573)<br class=3D"">29870:2018-03-21 = 10:46:35,170+0200 INFO (jsonrpc/4) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D76736a93-d4c4-4b2f-9848-7b8731fe2b67 (api:46)<br = class=3D"">29871:2018-03-21 10:46:35,176+0200 INFO (jsonrpc/4) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D76736a93-d4c4-4b2f-9848-7b8731fe2b67 (api:52)<br = class=3D"">29872:2018-03-21 10:46:35,177+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573)<br class=3D"">29873:2018-03-21 10:46:35,462+0200 = INFO (jsonrpc/2) [api.host] START getAllVmStats() = from=3D::ffff:192.168.0.4,49914 (api:46)<br class=3D"">29874:2018-03-21 = 10:46:35,462+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats = return=3D{'status': {'message': 'Done', 'code': 0}, 'statsList': = (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29875:2018-03-21 10:46:35,463+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29876:2018-03-21 10:46:36,948+0200 = INFO (jsonrpc/7) [api.host] START getAllVmStats() from=3D::1,36114 = (api:46)<br class=3D"">29877:2018-03-21 10:46:36,948+0200 INFO = (jsonrpc/7) [api.host] FINISH getAllVmStats return=3D{'status': = {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} = from=3D::1,36114 (api:52)<br class=3D"">29878:2018-03-21 = 10:46:36,949+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br = class=3D"">29879:2018-03-21 10:46:43,179+0200 INFO (jsonrpc/1) = [api.host] START getStats() from=3D::ffff:192.168.0.4,49914 (api:46)<br = class=3D"">29880:2018-03-21 10:46:43,180+0200 INFO (jsonrpc/1) = [vdsm.api] START repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D83bbc95c-6d59-4eec-82b1-e8993be28759 (api:46)<br = class=3D"">29881:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000486101', = 'lastCheck': '3.6', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': = '3.4', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D83bbc95c-6d59-4eec-82b1-e8993be28759 (api:52)<br = class=3D"">29882:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) = [vdsm.api] START multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D9b781324-b50b-49a9-bf31-5eca835dc75c (api:46)<br = class=3D"">29883:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) = [vdsm.api] FINISH multipath_health return=3D{} = from=3D::ffff:192.168.0.4,49914, = task_id=3D9b781324-b50b-49a9-bf31-5eca835dc75c (api:52)<br = class=3D"">29884:2018-03-21 10:46:43,188+0200 INFO (jsonrpc/1) = [api.host] FINISH getStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.40', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.33'}, '10': {'cpuUser': = '0.60', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '98.93'}, '13': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '99.73'}, '12': {'cpuUser': '1.46', 'nodeIndex': 0, 'cpuSys': '0.60', = 'cpuIdle': '97.94'}, '15': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': = '0.33', 'cpuIdle': '99.47'}, '14': {'cpuUser': '0.40', 'nodeIndex': 0, = 'cpuSys': '0.47', 'cpuIdle': '99.13'}, '1': {'cpuUser': '1.33', = 'nodeIndex': 1, 'cpuSys': '1.93', 'cpuIdle': '96.74'}, '0': {'cpuUser': = '0.60', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.13'}, '3': = {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': = '99.40'}, '2': {'cpuUser': '6.99', 'nodeIndex': 0, 'cpuSys': '0.40', = 'cpuIdle': '92.61'}, '5': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': = '0.20', 'cpuIdle': '99.67'}, '4': {'cpuUser': '0.27', 'nodeIndex': 0, = 'cpuSys': '0.27', 'cpuIdle': '99.46'}, '7': {'cpuUser': '0.47', = 'nodeIndex': 1, 'cpuSys': '0.60', 'cpuIdle': '98.93'}, '6': {'cpuUser': = '0.53', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.20'}, '9': = {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': = '99.20'}, '8': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', = 'cpuIdle': '99.80'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, = 'memFree': '15131'}, '0': {'memPercent': 5, 'memFree': '15357'}}, = 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, = 'vmCount': 0, 'memUsed': '3', 'storageDomains': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, = 'version': 4, 'acquired': True, 'delay': '0.000486101', 'lastCheck': = '3.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.00043444', 'lastCheck': '3.4', 'valid': True}}, = 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'enp3s0f0', = 'tx': '1091404888', 'txDropped': '0', 'rx': '11914970667', 'rxErrors': = '0', 'speed': '100', 'rxDropped': '1455'}, 'ovirtmgmt': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': = 'ovirtmgmt', 'tx': '1048752406', 'txDropped': '0', 'rx': '11615298347', = 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'lo', 'tx': = '58215969455', 'txDropped': '0', 'rx': '58215969455', 'rxErrors': '0', = 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'enp3s0f1', = 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': = '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': = 'down', 'sampleTime': 1521621990.363726, 'name': ';vdsmdummy;', 'tx': = '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621990.363726, 'name': 'enp4s0f0', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621990.363726, 'name': 'enp4s0f1', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '206', = 'ksmPages': 100, 'elapsedTime': '76562.02', 'cpuLoad': '0.18', 'cpuSys': = '0.45', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': = {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.27', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1455', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:46:43 GMT', 'cpuUser': '0.87', = 'memFree': 31336, 'cpuIdle': '98.68', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '1.20'}} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29885:2018-03-21 10:46:43,190+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573)<br class=3D"">29886:2018-03-21 10:46:45,288+0200 = INFO (jsonrpc/5) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3Dffe57349-2532-4b64-b32c-8a558273d8e5 (api:46)<br = class=3D"">29887:2018-03-21 10:46:45,292+0200 INFO (jsonrpc/5) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3Dffe57349-2532-4b64-b32c-8a558273d8e5 (api:52)<br = class=3D"">29888:2018-03-21 10:46:45,293+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.00 seconds (__init__:573)<br class=3D"">29889:2018-03-21 = 10:46:45,298+0200 INFO (jsonrpc/3) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3Db2ea28f0-abf0-4258-8152-6035d93aa5dc (api:46)<br = class=3D"">29890:2018-03-21 10:46:45,305+0200 INFO (jsonrpc/3) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3Db2ea28f0-abf0-4258-8152-6035d93aa5dc (api:52)<br = class=3D"">29891:2018-03-21 10:46:45,306+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29892:2018-03-21 10:46:45,395+0200 = INFO (periodic/2) [vdsm.api] START repoStats(domains=3D()) = from=3Dinternal, task_id=3D4010bf72-5c86-4daf-a8f9-12ec229a86e4 = (api:46)<br class=3D"">29893:2018-03-21 10:46:45,395+0200 INFO = (periodic/2) [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000486101', = 'lastCheck': '5.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': = '5.7', 'valid': True}} from=3Dinternal, = task_id=3D4010bf72-5c86-4daf-a8f9-12ec229a86e4 (api:52)<br = class=3D"">29894:2018-03-21 10:46:45,396+0200 INFO (periodic/2) = [vdsm.api] START multipath_health() from=3Dinternal, = task_id=3D37a1132f-452b-4805-9cc0-b8e74e8d6b02 (api:46)<br = class=3D"">29895:2018-03-21 10:46:45,396+0200 INFO (periodic/2) = [vdsm.api] FINISH multipath_health return=3D{} from=3Dinternal, = task_id=3D37a1132f-452b-4805-9cc0-b8e74e8d6b02 (api:52)<br = class=3D"">29896:2018-03-21 10:46:50,467+0200 INFO (jsonrpc/0) = [api.host] START getAllVmStats() from=3D::ffff:192.168.0.4,49914 = (api:46)<br class=3D"">29897:2018-03-21 10:46:50,468+0200 INFO = (jsonrpc/0) [throttled] Current getAllVmStats: {} (throttledlog:103)<br = class=3D"">29898:2018-03-21 10:46:50,468+0200 INFO (jsonrpc/0) = [api.host] FINISH getAllVmStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 = (api:52)<br class=3D"">29899:2018-03-21 10:46:50,468+0200 INFO = (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats = succeeded in 0.01 seconds (__init__:573)<br class=3D"">29900:2018-03-21 = 10:46:51,967+0200 INFO (jsonrpc/6) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46)<br class=3D"">29901:2018-03-21 = 10:46:51,968+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats = return=3D{'status': {'message': 'Done', 'code': 0}, 'statsList': = (suppressed)} from=3D::1,36114 (api:52)<br class=3D"">29902:2018-03-21 = 10:46:51,968+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br = class=3D"">29903:2018-03-21 10:46:55,648+0200 INFO (jsonrpc/4) = [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3Da975635f-59d7-4f40-b9a7-a5ea7150368e (api:46)<br = class=3D"">29904:2018-03-21 10:46:55,653+0200 INFO (jsonrpc/4) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3Da975635f-59d7-4f40-b9a7-a5ea7150368e (api:52)<br = class=3D"">29905:2018-03-21 10:46:55,653+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.00 seconds (__init__:573)<br class=3D"">29906:2018-03-21 = 10:46:55,659+0200 INFO (jsonrpc/2) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D97d4ae6c-3719-4149-85fd-28e924ad8883 (api:46)<br = class=3D"">29907:2018-03-21 10:46:55,666+0200 INFO (jsonrpc/2) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D97d4ae6c-3719-4149-85fd-28e924ad8883 (api:52)<br = class=3D"">29908:2018-03-21 10:46:55,667+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29909:2018-03-21 10:46:59,473+0200 = INFO (jsonrpc/7) [api.host] START getStats() = from=3D::ffff:192.168.0.4,49914 (api:46)<br class=3D"">29910:2018-03-21 = 10:46:59,474+0200 INFO (jsonrpc/7) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D177b551b-9bd3-45a4-84d0-5244630c08e5 (api:46)<br = class=3D"">29911:2018-03-21 10:46:59,474+0200 INFO (jsonrpc/7) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000470906', = 'lastCheck': '9.9', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000334928', 'lastCheck': = '9.7', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D177b551b-9bd3-45a4-84d0-5244630c08e5 (api:52)<br = class=3D"">29912:2018-03-21 10:46:59,475+0200 INFO (jsonrpc/7) = [vdsm.api] START multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D30753d5c-9bfd-4b0f-ac83-e32278d82b63 (api:46)<br = class=3D"">29913:2018-03-21 10:46:59,475+0200 INFO (jsonrpc/7) = [vdsm.api] FINISH multipath_health return=3D{} = from=3D::ffff:192.168.0.4,49914, = task_id=3D30753d5c-9bfd-4b0f-ac83-e32278d82b63 (api:52)<br = class=3D"">29914:2018-03-21 10:46:59,482+0200 INFO (jsonrpc/7) = [api.host] FINISH getStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '10': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '13': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.86'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', = 'cpuIdle': '99.93'}, '15': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.87'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, = 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '1': {'cpuUser': '1.07', = 'nodeIndex': 1, 'cpuSys': '1.07', 'cpuIdle': '97.86'}, '0': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '3': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.86'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', = 'cpuIdle': '99.87'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.80'}, '4': {'cpuUser': '0.47', 'nodeIndex': 0, = 'cpuSys': '0.27', 'cpuIdle': '99.26'}, '7': {'cpuUser': '0.00', = 'nodeIndex': 1, 'cpuSys': '0.47', 'cpuIdle': '99.53'}, '6': {'cpuUser': = '0.20', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.73'}, '9': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.80'}, '8': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', = 'cpuIdle': '99.93'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, = 'memFree': '15125'}, '0': {'memPercent': 5, 'memFree': '15364'}}, = 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, = 'vmCount': 0, 'memUsed': '3', 'storageDomains': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, = 'version': 4, 'acquired': True, 'delay': '0.000470906', 'lastCheck': = '9.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.000334928', 'lastCheck': '9.7', 'valid': True}}, = 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'enp3s0f0', = 'tx': '1091412332', 'txDropped': '0', 'rx': '11914976557', 'rxErrors': = '0', 'speed': '100', 'rxDropped': '1456'}, 'ovirtmgmt': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': = 'ovirtmgmt', 'tx': '1048761326', 'txDropped': '0', 'rx': '11615304242', = 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'lo', 'tx': = '58228460774', 'txDropped': '0', 'rx': '58228460774', 'rxErrors': '0', = 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'enp3s0f1', = 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': = '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': = 'down', 'sampleTime': 1521622005.383289, 'name': ';vdsmdummy;', 'tx': = '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521622005.383289, 'name': 'enp4s0f0', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521622005.383289, 'name': 'enp4s0f1', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '208', = 'ksmPages': 100, 'elapsedTime': '76578.31', 'cpuLoad': '0.17', 'cpuSys': = '0.20', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': = {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.13', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31078, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1456', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:46:59 GMT', 'cpuUser': '0.15', = 'memFree': 31334, 'cpuIdle': '99.65', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.60'}} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29915:2018-03-21 10:46:59,484+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573)</div><div class=3D""><br class=3D""><div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D"">On 21 = Mar 2018, at 11:33, Fred Rolland <<a = href=3D"mailto:frolland@redhat.com" class=3D"">frolland@redhat.com</a>>= wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = dir=3D"ltr" class=3D""><div class=3D"">Can you provide the vdsm logs = from the host.<br class=3D""></div>It looks the vdsm failed to connect = to the server.<br class=3D""></div><div class=3D"gmail_extra"><br = class=3D""><div class=3D"gmail_quote">On Wed, Mar 21, 2018 at 11:21 AM, = Andrei Verovski <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:andreil1@starlett.lv" target=3D"_blank" = class=3D"">andreil1@starlett.lv</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div = style=3D"word-wrap:break-word;line-break:after-white-space" = class=3D"">Hi,<div class=3D""><br class=3D""></div><div class=3D"">I = have 2-host oVirt setup with 2 Data Centers, one with local storage = domain (DC #1) for VMs + Export domain on NFS, another with all NFS = shared (DC #2).</div><div class=3D"">Trying to export VMs from DC = #1 to DC #2.</div><div class=3D"">VMs are exported to DC #1 export = domain (NFS), then domain put into maintenance mode and detached from DC = #1.</div><div class=3D""><br class=3D""></div><div = class=3D"">Unfortunately, attaching it to DC #2 failed. Logs attached. = Tried to run this command twice.</div><div class=3D"">Workaround are = possible in order to accomplish this task, yet it would be better to do = in a way as it was designed.<br class=3D""> Thanks.<br class=3D""><br class=3D""><br class=3D""><div = class=3D"">2018-03-21 10:43:24,024+02 ERROR = [org.ovirt.engine.core.dal.<wbr class=3D"">dbbroker.auditloghandling.<wbr = class=3D"">AuditLogDirector] (EE-ManagedThreadFactory-<wbr = class=3D"">engine-Thread-128904) [<font color=3D"#942192" = class=3D"">570ec5d9-fff5-4656-afbd-<wbr class=3D"">90b3207a616e</font>] = EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_<wbr class=3D"">POOL_FAILED(963), = Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center = StrDataCenter11. (User: admin@internal-authz)<br class=3D"">2018-03-21 = 10:46:16,512+02 ERROR [org.ovirt.engine.core.dal.<wbr = class=3D"">dbbroker.auditloghandling.<wbr class=3D"">AuditLogDirector] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128956) = [1435fc81] EVENT_ID: VDS_STORAGES_CONNECTION_<wbr class=3D"">FAILED(188), = Failed to connect Host node11 to the Storage Domains = node10-NFS-EXPORTS.<br class=3D"">2018-03-21 10:46:16,651+02 ERROR = [org.ovirt.engine.core.dal.<wbr class=3D"">dbbroker.auditloghandling.<wbr = class=3D"">AuditLogDirector] (EE-ManagedThreadFactory-<wbr = class=3D"">engine-Thread-128955) [<font color=3D"#942192" = class=3D"">a81ffa4a-5a58-41a0-888a-<wbr class=3D"">f0edc321609b</font>] = EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_<wbr class=3D"">POOL_FAILED(963), = Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center = StrDataCenter11. (User: admin@internal-authz)<br class=3D""><br = class=3D""><br class=3D""><font color=3D"#aa7942" class=3D""><b = class=3D"">tail -n 1000 engine.log | grep 570ec5d9-fff5-4656-afbd-<wbr = class=3D"">90b3207a616e</b></font><br class=3D"">2018-03-21 = 10:41:14,643+02 INFO [org.ovirt.engine.core.bll.<wbr = class=3D"">storage.domain.<wbr = class=3D"">AttachStorageDomainToPoolComma<wbr class=3D"">nd] (default = task-2) [570ec5d9-fff5-4656-afbd-<wbr class=3D"">90b3207a616e] Lock = Acquired to object 'EngineLock:{exclusiveLocks=3D'[<wbr = class=3D"">1d7208ce-d3a1-4406-9638-<wbr class=3D"">fe7051562994=3DSTORAGE]= ', sharedLocks=3D''}'<br class=3D"">2018-03-21 10:41:16,129+02 = INFO [org.ovirt.engine.core.bll.<wbr class=3D"">storage.domain.<wbr = class=3D"">AttachStorageDomainToPoolComma<wbr class=3D"">nd] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-<wbr class=3D"">90b3207a616e] Running command: = AttachStorageDomainToPoolComma<wbr class=3D"">nd internal: false. = Entities affected : ID: 1d7208ce-d3a1-4406-9638-<wbr = class=3D"">fe7051562994 Type: StorageAction group = MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: = 80cc922f-8dea-4fed-b951-<wbr class=3D"">1060ba116ad5 Type: = StoragePoolAction group MANIPULATE_STORAGE_DOMAIN with role type = ADMIN<br class=3D"">2018-03-21 10:43:23,564+02 ERROR = [org.ovirt.engine.core.bll.<wbr class=3D"">storage.domain.<wbr = class=3D"">AttachStorageDomainToPoolComma<wbr class=3D"">nd] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-<wbr class=3D"">90b3207a616e] Cannot connect = storage connection server, aborting attach storage domain operation.<br = class=3D"">2018-03-21 10:43:23,567+02 INFO = [org.ovirt.engine.core.bll.<wbr class=3D"">CommandCompensator] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-<wbr class=3D"">90b3207a616e] Command = [id=3D921ca7cd-4f93-46aa-8de2-<wbr class=3D"">91b13b8f96cb]: = Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.<wbr = class=3D"">businessentities.<wbr class=3D"">StoragePoolIsoMap; snapshot: = StoragePoolIsoMapId:{<wbr class=3D"">storagePoolId=3D'80cc922f-8dea-<wbr = class=3D"">4fed-b951-1060ba116ad5', storageId=3D'1d7208ce-d3a1-4406-<wbr = class=3D"">9638-fe7051562994'}.<br class=3D"">2018-03-21 10:43:24,024+02 = ERROR [org.ovirt.engine.core.dal.<wbr = class=3D"">dbbroker.auditloghandling.<wbr class=3D"">AuditLogDirector] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-<wbr class=3D"">90b3207a616e] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_<wbr class=3D"">POOL_FAILED(963), Failed = to attach Storage Domain node10-NFS-EXPORTS to Data Center = StrDataCenter11. (User: admin@internal-authz)<br class=3D"">2018-03-21 = 10:43:24,114+02 INFO [org.ovirt.engine.core.bll.<wbr = class=3D"">storage.domain.<wbr = class=3D"">AttachStorageDomainToPoolComma<wbr class=3D"">nd] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-<wbr class=3D"">90b3207a616e] Lock freed to = object 'EngineLock:{exclusiveLocks=3D'[<wbr = class=3D"">1d7208ce-d3a1-4406-9638-<wbr class=3D"">fe7051562994=3DSTORAGE]= ', sharedLocks=3D''}'<br class=3D""><br class=3D""><br class=3D""><br = class=3D""><font color=3D"#aa7942" class=3D""><b class=3D"">[root@node00 = ovirt-engine]# tail -n 1000 engine.log | grep = a81ffa4a-5a58-41a0-888a-<wbr class=3D"">f0edc321609b</b></font><br = class=3D"">2018-03-21 10:44:11,025+02 INFO = [org.ovirt.engine.core.bll.<wbr class=3D"">storage.domain.<wbr = class=3D"">AttachStorageDomainToPoolComma<wbr class=3D"">nd] (default = task-16) [a81ffa4a-5a58-41a0-888a-<wbr class=3D"">f0edc321609b] Lock = Acquired to object 'EngineLock:{exclusiveLocks=3D'[<wbr = class=3D"">1d7208ce-d3a1-4406-9638-<wbr class=3D"">fe7051562994=3DSTORAGE]= ', sharedLocks=3D''}'<br class=3D"">2018-03-21 10:44:11,236+02 = INFO [org.ovirt.engine.core.bll.<wbr class=3D"">storage.domain.<wbr = class=3D"">AttachStorageDomainToPoolComma<wbr class=3D"">nd] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-<wbr class=3D"">f0edc321609b] Running command: = AttachStorageDomainToPoolComma<wbr class=3D"">nd internal: false. = Entities affected : ID: 1d7208ce-d3a1-4406-9638-<wbr = class=3D"">fe7051562994 Type: StorageAction group = MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: = 80cc922f-8dea-4fed-b951-<wbr class=3D"">1060ba116ad5 Type: = StoragePoolAction group MANIPULATE_STORAGE_DOMAIN with role type = ADMIN<br class=3D"">2018-03-21 10:46:16,567+02 ERROR = [org.ovirt.engine.core.bll.<wbr class=3D"">storage.domain.<wbr = class=3D"">AttachStorageDomainToPoolComma<wbr class=3D"">nd] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-<wbr class=3D"">f0edc321609b] Cannot connect = storage connection server, aborting attach storage domain operation.<br = class=3D"">2018-03-21 10:46:16,568+02 INFO = [org.ovirt.engine.core.bll.<wbr class=3D"">CommandCompensator] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-<wbr class=3D"">f0edc321609b] Command = [id=3Db5c25100-1a8a-4db0-9509-<wbr class=3D"">99cfa60995b1]: = Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.<wbr = class=3D"">businessentities.<wbr class=3D"">StoragePoolIsoMap; snapshot: = StoragePoolIsoMapId:{<wbr class=3D"">storagePoolId=3D'80cc922f-8dea-<wbr = class=3D"">4fed-b951-1060ba116ad5', storageId=3D'1d7208ce-d3a1-4406-<wbr = class=3D"">9638-fe7051562994'}.<br class=3D"">2018-03-21 10:46:16,651+02 = ERROR [org.ovirt.engine.core.dal.<wbr = class=3D"">dbbroker.auditloghandling.<wbr class=3D"">AuditLogDirector] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-<wbr class=3D"">f0edc321609b] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_<wbr class=3D"">POOL_FAILED(963), Failed = to attach Storage Domain node10-NFS-EXPORTS to Data Center = StrDataCenter11. (User: admin@internal-authz)<br class=3D"">2018-03-21 = 10:46:16,681+02 INFO [org.ovirt.engine.core.bll.<wbr = class=3D"">storage.domain.<wbr = class=3D"">AttachStorageDomainToPoolComma<wbr class=3D"">nd] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-<wbr class=3D"">f0edc321609b] Lock freed to = object 'EngineLock:{exclusiveLocks=3D'[<wbr = class=3D"">1d7208ce-d3a1-4406-9638-<wbr class=3D"">fe7051562994=3DSTORAGE]= ', sharedLocks=3D'=E2=80=99}'</div><div class=3D""><br = class=3D""></div></div></div><br = class=3D"">______________________________<wbr = class=3D"">_________________<br class=3D""> Users mailing list<br class=3D""> <a href=3D"mailto:Users@ovirt.org" class=3D"">Users@ovirt.org</a><br = class=3D""> <a href=3D"http://lists.ovirt.org/mailman/listinfo/users" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/users</a><br class=3D""> <br class=3D""></blockquote></div><br class=3D""></div> </div></blockquote></div><br class=3D""></div></body></html>= --Apple-Mail=_C5052CF0-37B9-4927-AE48-DF656F09C5B6--