Attach Export Domain to another Data Center - Failure
--Apple-Mail=_2F3B6B23-5CA4-462A-87E0-5B084B91DCB9 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi, I have 2-host oVirt setup with 2 Data Centers, one with local storage = domain (DC #1) for VMs + Export domain on NFS, another with all NFS = shared (DC #2). Trying to export VMs from DC #1 to DC #2. VMs are exported to DC #1 export domain (NFS), then domain put into = maintenance mode and detached from DC #1. Unfortunately, attaching it to DC #2 failed. Logs attached. Tried to run = this command twice. Workaround are possible in order to accomplish this task, yet it would = be better to do in a way as it was designed. Thanks. 2018-03-21 10:43:24,024+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage = Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: = admin@internal-authz) 2018-03-21 10:46:16,512+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128956) [1435fc81] EVENT_ID: = VDS_STORAGES_CONNECTION_FAILED(188), Failed to connect Host node11 to = the Storage Domains node10-NFS-EXPORTS. 2018-03-21 10:46:16,651+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage = Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: = admin@internal-authz) tail -n 1000 engine.log | grep 570ec5d9-fff5-4656-afbd-90b3207a616e 2018-03-21 10:41:14,643+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (default task-2) [570ec5d9-fff5-4656-afbd-90b3207a616e] Lock Acquired = to object = 'EngineLock:{exclusiveLocks=3D'[1d7208ce-d3a1-4406-9638-fe7051562994=3DSTO= RAGE]', sharedLocks=3D''}' 2018-03-21 10:41:16,129+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] Running command: = AttachStorageDomainToPoolCommand internal: false. Entities affected : = ID: 1d7208ce-d3a1-4406-9638-fe7051562994 Type: StorageAction group = MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: = 80cc922f-8dea-4fed-b951-1060ba116ad5 Type: StoragePoolAction group = MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2018-03-21 10:43:23,564+02 ERROR = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] Cannot connect storage connection = server, aborting attach storage domain operation. 2018-03-21 10:43:23,567+02 INFO = [org.ovirt.engine.core.bll.CommandCompensator] = (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] Command = [id=3D921ca7cd-4f93-46aa-8de2-91b13b8f96cb]: Compensating NEW_ENTITY_ID = of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; = snapshot: = StoragePoolIsoMapId:{storagePoolId=3D'80cc922f-8dea-4fed-b951-1060ba116ad5= ', storageId=3D'1d7208ce-d3a1-4406-9638-fe7051562994'}. 2018-03-21 10:43:24,024+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage = Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: = admin@internal-authz) 2018-03-21 10:43:24,114+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] Lock freed to object = 'EngineLock:{exclusiveLocks=3D'[1d7208ce-d3a1-4406-9638-fe7051562994=3DSTO= RAGE]', sharedLocks=3D''}' [root@node00 ovirt-engine]# tail -n 1000 engine.log | grep = a81ffa4a-5a58-41a0-888a-f0edc321609b 2018-03-21 10:44:11,025+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (default task-16) [a81ffa4a-5a58-41a0-888a-f0edc321609b] Lock Acquired = to object = 'EngineLock:{exclusiveLocks=3D'[1d7208ce-d3a1-4406-9638-fe7051562994=3DSTO= RAGE]', sharedLocks=3D''}' 2018-03-21 10:44:11,236+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] Running command: = AttachStorageDomainToPoolCommand internal: false. Entities affected : = ID: 1d7208ce-d3a1-4406-9638-fe7051562994 Type: StorageAction group = MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: = 80cc922f-8dea-4fed-b951-1060ba116ad5 Type: StoragePoolAction group = MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2018-03-21 10:46:16,567+02 ERROR = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] Cannot connect storage connection = server, aborting attach storage domain operation. 2018-03-21 10:46:16,568+02 INFO = [org.ovirt.engine.core.bll.CommandCompensator] = (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] Command = [id=3Db5c25100-1a8a-4db0-9509-99cfa60995b1]: Compensating NEW_ENTITY_ID = of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; = snapshot: = StoragePoolIsoMapId:{storagePoolId=3D'80cc922f-8dea-4fed-b951-1060ba116ad5= ', storageId=3D'1d7208ce-d3a1-4406-9638-fe7051562994'}. 2018-03-21 10:46:16,651+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage = Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: = admin@internal-authz) 2018-03-21 10:46:16,681+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] Lock freed to object = 'EngineLock:{exclusiveLocks=3D'[1d7208ce-d3a1-4406-9638-fe7051562994=3DSTO= RAGE]', sharedLocks=3D'=E2=80=99}' --Apple-Mail=_2F3B6B23-5CA4-462A-87E0-5B084B91DCB9 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" = class=3D"">Hi,<div class=3D""><br class=3D""></div><div class=3D"">I = have 2-host oVirt setup with 2 Data Centers, one with local storage = domain (DC #1) for VMs + Export domain on NFS, another with all NFS = shared (DC #2).</div><div class=3D"">Trying to export VMs from DC = #1 to DC #2.</div><div class=3D"">VMs are exported to DC #1 export = domain (NFS), then domain put into maintenance mode and detached from DC = #1.</div><div class=3D""><br class=3D""></div><div = class=3D"">Unfortunately, attaching it to DC #2 failed. Logs attached. = Tried to run this command twice.</div><div class=3D"">Workaround are = possible in order to accomplish this task, yet it would be better to do = in a way as it was designed.<br class=3D""> Thanks.<br class=3D""><br class=3D""><br class=3D""><div = class=3D"">2018-03-21 10:43:24,024+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128904) [<font color=3D"#942192" = class=3D"">570ec5d9-fff5-4656-afbd-90b3207a616e</font>] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage = Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: = admin@internal-authz)<br class=3D"">2018-03-21 10:46:16,512+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128956) [1435fc81] EVENT_ID: = VDS_STORAGES_CONNECTION_FAILED(188), Failed to connect Host node11 to = the Storage Domains node10-NFS-EXPORTS.<br class=3D"">2018-03-21 = 10:46:16,651+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128955) [<font color=3D"#942192" = class=3D"">a81ffa4a-5a58-41a0-888a-f0edc321609b</font>] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage = Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: = admin@internal-authz)<br class=3D""><br class=3D""><br class=3D""><font = color=3D"#aa7942" class=3D""><b class=3D"">tail -n 1000 engine.log | = grep 570ec5d9-fff5-4656-afbd-90b3207a616e</b></font><br = class=3D"">2018-03-21 10:41:14,643+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (default task-2) [570ec5d9-fff5-4656-afbd-90b3207a616e] Lock Acquired = to object = 'EngineLock:{exclusiveLocks=3D'[1d7208ce-d3a1-4406-9638-fe7051562994=3DSTO= RAGE]', sharedLocks=3D''}'<br class=3D"">2018-03-21 10:41:16,129+02 = INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] Running command: = AttachStorageDomainToPoolCommand internal: false. Entities affected = : ID: 1d7208ce-d3a1-4406-9638-fe7051562994 Type: StorageAction = group MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: = 80cc922f-8dea-4fed-b951-1060ba116ad5 Type: StoragePoolAction group = MANIPULATE_STORAGE_DOMAIN with role type ADMIN<br class=3D"">2018-03-21 = 10:43:23,564+02 ERROR = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] Cannot connect storage connection = server, aborting attach storage domain operation.<br class=3D"">2018-03-21= 10:43:23,567+02 INFO = [org.ovirt.engine.core.bll.CommandCompensator] = (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] Command = [id=3D921ca7cd-4f93-46aa-8de2-91b13b8f96cb]: Compensating NEW_ENTITY_ID = of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; = snapshot: = StoragePoolIsoMapId:{storagePoolId=3D'80cc922f-8dea-4fed-b951-1060ba116ad5= ', storageId=3D'1d7208ce-d3a1-4406-9638-fe7051562994'}.<br = class=3D"">2018-03-21 10:43:24,024+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage = Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: = admin@internal-authz)<br class=3D"">2018-03-21 10:43:24,114+02 = INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] Lock freed to object = 'EngineLock:{exclusiveLocks=3D'[1d7208ce-d3a1-4406-9638-fe7051562994=3DSTO= RAGE]', sharedLocks=3D''}'<br class=3D""><br class=3D""><br class=3D""><br= class=3D""><font color=3D"#aa7942" class=3D""><b class=3D"">[root@node00 = ovirt-engine]# tail -n 1000 engine.log | grep = a81ffa4a-5a58-41a0-888a-f0edc321609b</b></font><br class=3D"">2018-03-21 = 10:44:11,025+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (default task-16) [a81ffa4a-5a58-41a0-888a-f0edc321609b] Lock Acquired = to object = 'EngineLock:{exclusiveLocks=3D'[1d7208ce-d3a1-4406-9638-fe7051562994=3DSTO= RAGE]', sharedLocks=3D''}'<br class=3D"">2018-03-21 10:44:11,236+02 = INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] Running command: = AttachStorageDomainToPoolCommand internal: false. Entities affected = : ID: 1d7208ce-d3a1-4406-9638-fe7051562994 Type: StorageAction = group MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: = 80cc922f-8dea-4fed-b951-1060ba116ad5 Type: StoragePoolAction group = MANIPULATE_STORAGE_DOMAIN with role type ADMIN<br class=3D"">2018-03-21 = 10:46:16,567+02 ERROR = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] Cannot connect storage connection = server, aborting attach storage domain operation.<br class=3D"">2018-03-21= 10:46:16,568+02 INFO = [org.ovirt.engine.core.bll.CommandCompensator] = (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] Command = [id=3Db5c25100-1a8a-4db0-9509-99cfa60995b1]: Compensating NEW_ENTITY_ID = of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; = snapshot: = StoragePoolIsoMapId:{storagePoolId=3D'80cc922f-8dea-4fed-b951-1060ba116ad5= ', storageId=3D'1d7208ce-d3a1-4406-9638-fe7051562994'}.<br = class=3D"">2018-03-21 10:46:16,651+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage = Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: = admin@internal-authz)<br class=3D"">2018-03-21 10:46:16,681+02 = INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] Lock freed to object = 'EngineLock:{exclusiveLocks=3D'[1d7208ce-d3a1-4406-9638-fe7051562994=3DSTO= RAGE]', sharedLocks=3D'=E2=80=99}'</div><div class=3D""><br = class=3D""></div></div></body></html>= --Apple-Mail=_2F3B6B23-5CA4-462A-87E0-5B084B91DCB9--
Can you provide the vdsm logs from the host. It looks the vdsm failed to connect to the server. On Wed, Mar 21, 2018 at 11:21 AM, Andrei Verovski <andreil1@starlett.lv> wrote:
Hi,
I have 2-host oVirt setup with 2 Data Centers, one with local storage domain (DC #1) for VMs + Export domain on NFS, another with all NFS shared (DC #2). Trying to export VMs from DC #1 to DC #2. VMs are exported to DC #1 export domain (NFS), then domain put into maintenance mode and detached from DC #1.
Unfortunately, attaching it to DC #2 failed. Logs attached. Tried to run this command twice. Workaround are possible in order to accomplish this task, yet it would be better to do in a way as it was designed. Thanks.
2018-03-21 10:43:24,024+02 ERROR [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-128904) [570ec5d9-fff5-4656-afbd-90b3207a616e] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz) 2018-03-21 10:46:16,512+02 ERROR [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-128956) [1435fc81] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Failed to connect Host node11 to the Storage Domains node10-NFS-EXPORTS. 2018-03-21 10:46:16,651+02 ERROR [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc321609b] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz)
*tail -n 1000 engine.log | grep 570ec5d9-fff5-4656-afbd-90b3207a616e* 2018-03-21 10:41:14,643+02 INFO [org.ovirt.engine.core.bll. storage.domain.AttachStorageDomainToPoolCommand] (default task-2) [570ec5d9-fff5-4656-afbd-90b3207a616e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[1d7208ce-d3a1-4406-9638-fe7051562994=STORAGE]', sharedLocks=''}' 2018-03-21 10:41:16,129+02 INFO [org.ovirt.engine.core.bll. storage.domain.AttachStorageDomainToPoolCommand] (EE-ManagedThreadFactory-engine-Thread-128904) [570ec5d9-fff5-4656-afbd-90b3207a616e] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 1d7208ce-d3a1-4406-9638-fe7051562994 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: 80cc922f-8dea-4fed-b951-1060ba116ad5 Type: StoragePoolAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2018-03-21 10:43:23,564+02 ERROR [org.ovirt.engine.core.bll. storage.domain.AttachStorageDomainToPoolCommand] (EE-ManagedThreadFactory-engine-Thread-128904) [570ec5d9-fff5-4656-afbd-90b3207a616e] Cannot connect storage connection server, aborting attach storage domain operation. 2018-03-21 10:43:23,567+02 INFO [org.ovirt.engine.core.bll.CommandCompensator] (EE-ManagedThreadFactory-engine-Thread-128904) [570ec5d9-fff5-4656-afbd-90b3207a616e] Command [id=921ca7cd-4f93-46aa-8de2-91b13b8f96cb]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: StoragePoolIsoMapId:{storagePoolId='80cc922f-8dea-4fed-b951-1060ba116ad5', storageId='1d7208ce-d3a1-4406-9638-fe7051562994'}. 2018-03-21 10:43:24,024+02 ERROR [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-128904) [570ec5d9-fff5-4656-afbd-90b3207a616e] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz) 2018-03-21 10:43:24,114+02 INFO [org.ovirt.engine.core.bll. storage.domain.AttachStorageDomainToPoolCommand] (EE-ManagedThreadFactory-engine-Thread-128904) [570ec5d9-fff5-4656-afbd-90b3207a616e] Lock freed to object 'EngineLock:{exclusiveLocks='[1d7208ce-d3a1-4406-9638-fe7051562994=STORAGE]', sharedLocks=''}'
*[root@node00 ovirt-engine]# tail -n 1000 engine.log | grep a81ffa4a-5a58-41a0-888a-f0edc321609b* 2018-03-21 10:44:11,025+02 INFO [org.ovirt.engine.core.bll. storage.domain.AttachStorageDomainToPoolCommand] (default task-16) [a81ffa4a-5a58-41a0-888a-f0edc321609b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[1d7208ce-d3a1-4406-9638-fe7051562994=STORAGE]', sharedLocks=''}' 2018-03-21 10:44:11,236+02 INFO [org.ovirt.engine.core.bll. storage.domain.AttachStorageDomainToPoolCommand] (EE-ManagedThreadFactory-engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc321609b] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 1d7208ce-d3a1-4406-9638-fe7051562994 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: 80cc922f-8dea-4fed-b951-1060ba116ad5 Type: StoragePoolAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2018-03-21 10:46:16,567+02 ERROR [org.ovirt.engine.core.bll. storage.domain.AttachStorageDomainToPoolCommand] (EE-ManagedThreadFactory-engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc321609b] Cannot connect storage connection server, aborting attach storage domain operation. 2018-03-21 10:46:16,568+02 INFO [org.ovirt.engine.core.bll.CommandCompensator] (EE-ManagedThreadFactory-engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc321609b] Command [id=b5c25100-1a8a-4db0-9509-99cfa60995b1]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: StoragePoolIsoMapId:{storagePoolId='80cc922f-8dea-4fed-b951-1060ba116ad5', storageId='1d7208ce-d3a1-4406-9638-fe7051562994'}. 2018-03-21 10:46:16,651+02 ERROR [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc321609b] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz) 2018-03-21 10:46:16,681+02 INFO [org.ovirt.engine.core.bll. storage.domain.AttachStorageDomainToPoolCommand] (EE-ManagedThreadFactory-engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc321609b] Lock freed to object 'EngineLock:{exclusiveLocks='[1d7208ce-d3a1-4406-9638-fe7051562994=STORAGE]', sharedLocks='’}'
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_C5052CF0-37B9-4927-AE48-DF656F09C5B6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi, Errors occurred at 10.43 AM and 10.46 AM node00.starlett.lv - 192.168.0.4 - oVirt host engine (separate PC, not = hosted) node10.starlett.lv - 192.168.0.5 - host #1 of DC #1, export domain from = which was detached from DC #1 node11.starlett.lv - 192.168.0.6 - host #1 of DC #2,=20 Logs from DC#2 node11, to which I=E2=80=99m trying to attach export = domain located at NFS share node10.
grep 570ec5d9-fff5-4656-afbd-90b3207a616e within vdsm.log returned nothing, so I did
On 21 Mar 2018, at 11:33, Fred Rolland <frolland@redhat.com> wrote: =20 Can you provide the vdsm logs from the host. It looks the vdsm failed to connect to the server. =20 On Wed, Mar 21, 2018 at 11:21 AM, Andrei Verovski = <andreil1@starlett.lv <mailto:andreil1@starlett.lv>> wrote: Hi, =20 I have 2-host oVirt setup with 2 Data Centers, one with local storage = domain (DC #1) for VMs + Export domain on NFS, another with all NFS = shared (DC #2). Trying to export VMs from DC #1 to DC #2. VMs are exported to DC #1 export domain (NFS), then domain put into =
=20 Unfortunately, attaching it to DC #2 failed. Logs attached. Tried to = run this command twice. Workaround are possible in order to accomplish this task, yet it would = be better to do in a way as it was designed. Thanks. =20 =20 2018-03-21 10:43:24,024+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage = Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: = admin@internal-authz) 2018-03-21 10:46:16,512+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128956) [1435fc81] EVENT_ID: = VDS_STORAGES_CONNECTION_FAILED(188), Failed to connect Host node11 to =
grep -n 10:43 vdsm.log | tail -1000 1011:2018-03-21 06:10:43,077+0200 INFO (jsonrpc/3) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 1012:2018-03-21 06:10:43,077+0200 INFO (jsonrpc/3) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 1013:2018-03-21 06:10:43,077+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 = seconds (__init__:573) 1014:2018-03-21 06:10:43,868+0200 INFO (jsonrpc/7) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 1015:2018-03-21 06:10:43,868+0200 INFO (jsonrpc/7) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 1016:2018-03-21 06:10:43,868+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 19729:2018-03-21 09:10:43,641+0200 INFO (jsonrpc/5) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 19730:2018-03-21 09:10:43,641+0200 INFO (jsonrpc/5) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 19731:2018-03-21 09:10:43,642+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29302:2018-03-21 10:43:00,085+0200 INFO (periodic/2) [vdsm.api] START = repoStats(domains=3D()) from=3Dinternal, = task_id=3D7b690c8a-5470-44da-a8f3-7e7e9b018e88 (api:46) 29303:2018-03-21 10:43:00,085+0200 INFO (periodic/2) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000477038', = 'lastCheck': '0.5', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': = '0.4', 'valid': True}} from=3Dinternal, = task_id=3D7b690c8a-5470-44da-a8f3-7e7e9b018e88 (api:52) 29304:2018-03-21 10:43:00,086+0200 INFO (periodic/2) [vdsm.api] START = multipath_health() from=3Dinternal, = task_id=3D9a625d43-a03c-429c-856f-9aa8e5ff65b5 (api:46) 29305:2018-03-21 10:43:00,086+0200 INFO (periodic/2) [vdsm.api] FINISH = multipath_health return=3D{} from=3Dinternal, = task_id=3D9a625d43-a03c-429c-856f-9aa8e5ff65b5 (api:52) 29306:2018-03-21 10:43:05,382+0200 INFO (jsonrpc/5) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29307:2018-03-21 10:43:05,383+0200 INFO (jsonrpc/5) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 29308:2018-03-21 10:43:05,383+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29309:2018-03-21 10:43:06,248+0200 INFO (jsonrpc/3) [api.host] START = getStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29310:2018-03-21 10:43:06,249+0200 INFO (jsonrpc/3) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3Df80f9bb8-2e84-4f45-ab58-09a88e808cf3 (api:46) 29311:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000477038', = 'lastCheck': '6.7', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': = '6.6', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3Df80f9bb8-2e84-4f45-ab58-09a88e808cf3 (api:52) 29312:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) [vdsm.api] START = multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3Dddd7064d-cd12-43bf-a177-b2a93cc9035e (api:46) 29313:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) [vdsm.api] FINISH = multipath_health return=3D{} from=3D::ffff:192.168.0.4,49914, = task_id=3Dddd7064d-cd12-43bf-a177-b2a93cc9035e (api:52) 29314:2018-03-21 10:43:06,256+0200 INFO (jsonrpc/3) [api.host] FINISH = getStats return=3D{'status': {'message': 'Done', 'code': 0}, 'info': = {'cpuStatistics': {'11': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': = '0.27', 'cpuIdle': '99.60'}, '10': {'cpuUser': '0.13', 'nodeIndex': 0, = 'cpuSys': '0.40', 'cpuIdle': '99.47'}, '13': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '12': {'cpuUser': = '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '15': = {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.47', 'cpuIdle': = '99.33'}, '14': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.27', = 'cpuIdle': '99.60'}, '1': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': = '1.60', 'cpuIdle': '97.27'}, '0': {'cpuUser': '0.47', 'nodeIndex': 0, = 'cpuSys': '0.27', 'cpuIdle': '99.26'}, '3': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '2': {'cpuUser': = '0.33', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.54'}, '5': = {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.74'}, '4': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', = 'cpuIdle': '99.67'}, '7': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': = '0.60', 'cpuIdle': '99.27'}, '6': {'cpuUser': '0.13', 'nodeIndex': 0, = 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '9': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '8': {'cpuUser': = '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}}, = 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15142'}, '0': = {'memPercent': 5, 'memFree': '15349'}}, 'memShared': 0, 'thpState': = 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', = 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000477038', = 'lastCheck': '6.7', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': = '6.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': = {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621780.072418, 'name': 'enp3s0f0', 'tx': '1091287978', 'txDropped': = '0', 'rx': '11914868563', 'rxErrors': '0', 'speed': '100', 'rxDropped': = '1439'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621780.072418, 'name': 'ovirtmgmt', 'tx': '1048639112', 'txDropped': = '0', 'rx': '11615209092', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621780.072418, 'name': 'lo', 'tx': '58057075945', 'txDropped': '0', = 'rx': '58057075945', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621780.072418, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621780.072418, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621780.072418, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621780.072418, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, = 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, = 'elapsedTime': '76345.09', 'cpuLoad': '0.16', 'cpuSys': '0.36', = 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': = '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31081, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1439', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, = 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, = 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:43:06 GMT', 'cpuUser': '0.21', = 'memFree': 31337, 'cpuIdle': '99.43', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.93'}} from=3D::ffff:192.168.0.4,49914 (api:52) 29315:2018-03-21 10:43:06,258+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573) 29316:2018-03-21 10:43:06,632+0200 INFO (jsonrpc/0) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 29317:2018-03-21 10:43:06,632+0200 INFO (jsonrpc/0) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 29318:2018-03-21 10:43:06,633+0200 INFO (jsonrpc/0) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29319:2018-03-21 10:43:07,415+0200 INFO (jsonrpc/4) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D276c13c7-61ec-4782-a0d9-5cd0f0b0a729 (api:46) 29320:2018-03-21 10:43:07,420+0200 INFO (jsonrpc/4) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D276c13c7-61ec-4782-a0d9-5cd0f0b0a729 (api:52) 29321:2018-03-21 10:43:07,420+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573) 29322:2018-03-21 10:43:07,427+0200 INFO (jsonrpc/6) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D344b18df-a428-429a-b49f-381f901d1618 (api:46) 29323:2018-03-21 10:43:07,434+0200 INFO (jsonrpc/6) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D344b18df-a428-429a-b49f-381f901d1618 (api:52) 29324:2018-03-21 10:43:07,435+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573) 29325:2018-03-21 10:43:13,839+0200 WARN (vdsm.Scheduler) [Executor] = Worker blocked: <Worker name=3Djsonrpc/1 running <Task <JsonRpcTask = {'params': {u'connectionParams': [{u'id': = u'461f65a9-3a81-4f3f-a46d-c5ed12520524', u'connection': = u'node10.starlett.lv:/vmdata/nfs/exports', u'iqn': u'', u'user': u'', = u'tpgt': u'1', u'protocol_version': u'auto', u'password': '********', = u'port': u''}], u'storagepoolID': = u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': = '2.0', 'method': u'StoragePool.connectStorageServer', 'id': = u'5f98dfd5-998f-44a6-8594-e5f40144b0ed'} at 0x4076150> timeout=3D60, = duration=3D120 at 0x402fe50> task#=3D3754 at 0x353e690>, traceback: 29376:2018-03-21 10:43:15,104+0200 INFO (periodic/2) [vdsm.api] START = repoStats(domains=3D()) from=3Dinternal, = task_id=3D672a883e-1d60-4981-9eee-bdcac2ec30c1 (api:46) 29377:2018-03-21 10:43:15,105+0200 INFO (periodic/2) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000488777', = 'lastCheck': '5.6', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000481962', 'lastCheck': = '5.4', 'valid': True}} from=3Dinternal, = task_id=3D672a883e-1d60-4981-9eee-bdcac2ec30c1 (api:52) 29378:2018-03-21 10:43:15,105+0200 INFO (periodic/2) [vdsm.api] START = multipath_health() from=3Dinternal, = task_id=3D55f1b797-da2a-4b07-affe-5b9e41a74307 (api:46) 29379:2018-03-21 10:43:15,106+0200 INFO (periodic/2) [vdsm.api] FINISH = multipath_health return=3D{} from=3Dinternal, = task_id=3D55f1b797-da2a-4b07-affe-5b9e41a74307 (api:52) 29380:2018-03-21 10:43:17,645+0200 INFO (jsonrpc/7) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D52a6ddeb-3c20-4f3f-b27d-94d19580b473 (api:46) 29381:2018-03-21 10:43:17,649+0200 INFO (jsonrpc/7) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D52a6ddeb-3c20-4f3f-b27d-94d19580b473 (api:52) 29382:2018-03-21 10:43:17,650+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573) 29383:2018-03-21 10:43:17,656+0200 INFO (jsonrpc/2) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D1a103883-6a6a-4d61-ade6-96559591ddb0 (api:46) 29384:2018-03-21 10:43:17,663+0200 INFO (jsonrpc/2) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D1a103883-6a6a-4d61-ade6-96559591ddb0 (api:52) 29385:2018-03-21 10:43:17,663+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573) 29386:2018-03-21 10:43:18,882+0200 ERROR (jsonrpc/1) [storage.HSM] Could = not connect to storageServer (hsm:2407) 29406:2018-03-21 10:43:18,883+0200 INFO (jsonrpc/1) [vdsm.api] FINISH = connectStorageServer return=3D{'statuslist': [{'status': 477, 'id': = u'461f65a9-3a81-4f3f-a46d-c5ed12520524'}]} = from=3D::ffff:192.168.0.4,49914, flow_id=3D4a53b512, = task_id=3De598cbe0-cde8-4c73-b526-4398df05e67f (api:52) 29407:2018-03-21 10:43:18,883+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer = succeeded in 125.04 seconds (__init__:573) 29408:2018-03-21 10:43:20,387+0200 INFO (jsonrpc/5) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29409:2018-03-21 10:43:20,388+0200 INFO (jsonrpc/5) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 29410:2018-03-21 10:43:20,388+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 = seconds (__init__:573) 29411:2018-03-21 10:43:21,497+0200 INFO (jsonrpc/3) [api.host] START = getStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29412:2018-03-21 10:43:21,498+0200 INFO (jsonrpc/3) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3Da7469c1f-3d6d-42e7-8798-5c5335da4a9b (api:46) 29413:2018-03-21 10:43:21,498+0200 INFO (jsonrpc/3) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000445096', = 'lastCheck': '1.9', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000296586', 'lastCheck': = '1.8', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3Da7469c1f-3d6d-42e7-8798-5c5335da4a9b (api:52) 29414:2018-03-21 10:43:21,499+0200 INFO (jsonrpc/3) [vdsm.api] START = multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D9fa5dae3-e2bd-4186-9a1d-afb0aa6e2511 (api:46) 29415:2018-03-21 10:43:21,499+0200 INFO (jsonrpc/3) [vdsm.api] FINISH = multipath_health return=3D{} from=3D::ffff:192.168.0.4,49914, = task_id=3D9fa5dae3-e2bd-4186-9a1d-afb0aa6e2511 (api:52) 29416:2018-03-21 10:43:21,505+0200 INFO (jsonrpc/3) [api.host] FINISH = getStats return=3D{'status': {'message': 'Done', 'code': 0}, 'info': = {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.07', 'cpuIdle': '99.86'}, '10': {'cpuUser': '0.07', 'nodeIndex': 0, = 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '13': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '12': {'cpuUser': = '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '15': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.87'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.33', = 'cpuIdle': '99.60'}, '1': {'cpuUser': '1.00', 'nodeIndex': 1, 'cpuSys': = '1.00', 'cpuIdle': '98.00'}, '0': {'cpuUser': '0.07', 'nodeIndex': 0, = 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '3': {'cpuUser': '0.00', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '2': {'cpuUser': = '0.27', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.66'}, '5': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.86'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', = 'cpuIdle': '99.86'}, '7': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.27', 'cpuIdle': '99.66'}, '6': {'cpuUser': '0.07', 'nodeIndex': 0, = 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '9': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '8': {'cpuUser': = '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}}, = 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15142'}, '0': = {'memPercent': 5, 'memFree': '15351'}}, 'memShared': 0, 'thpState': = 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', = 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000445096', = 'lastCheck': '1.9', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000296586', 'lastCheck': = '1.8', 'valid': True}}, 'incomingVmMigrations': 0, 'network': = {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621795.091263, 'name': 'enp3s0f0', 'tx': '1091295628', 'txDropped': = '0', 'rx': '11914875210', 'rxErrors': '0', 'speed': '100', 'rxDropped': = '1442'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621795.091263, 'name': 'ovirtmgmt', 'tx': '1048646480', 'txDropped': = '0', 'rx': '11615214750', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621795.091263, 'name': 'lo', 'tx': '58066397668', 'txDropped': '0', = 'rx': '58066397668', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621795.091263, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621795.091263, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621795.091263, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621795.091263, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, = 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, = 'elapsedTime': '76360.34', 'cpuLoad': '0.15', 'cpuSys': '0.18', = 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': = '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31031, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1442', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, = 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, = 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:43:21 GMT', 'cpuUser': '0.15', = 'memFree': 31287, 'cpuIdle': '99.67', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.67'}} from=3D::ffff:192.168.0.4,49914 (api:52) 29417:2018-03-21 10:43:21,508+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.02 seconds = (__init__:573) 29418:2018-03-21 10:43:21,651+0200 INFO (jsonrpc/0) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 29419:2018-03-21 10:43:21,651+0200 INFO (jsonrpc/0) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 29420:2018-03-21 10:43:21,652+0200 INFO (jsonrpc/0) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29421:2018-03-21 10:43:27,764+0200 INFO (jsonrpc/4) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D46d19cab-8903-4c52-9bb2-4dd8f370997e (api:46) 29422:2018-03-21 10:43:27,769+0200 INFO (jsonrpc/4) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D46d19cab-8903-4c52-9bb2-4dd8f370997e (api:52) 29423:2018-03-21 10:43:27,770+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573) 29424:2018-03-21 10:43:27,815+0200 INFO (jsonrpc/6) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D290acc73-8f25-4fd7-9263-7bd05d46430a (api:46) 29425:2018-03-21 10:43:27,822+0200 INFO (jsonrpc/6) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D290acc73-8f25-4fd7-9263-7bd05d46430a (api:52) 29426:2018-03-21 10:43:27,823+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573) 29427:2018-03-21 10:43:30,121+0200 INFO (periodic/3) [vdsm.api] START = repoStats(domains=3D()) from=3Dinternal, = task_id=3D0abdebcf-2b0b-4d33-b552-c0f2ed3e567b (api:46) 29428:2018-03-21 10:43:30,122+0200 INFO (periodic/3) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000402077', = 'lastCheck': '0.6', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': = '0.5', 'valid': True}} from=3Dinternal, = task_id=3D0abdebcf-2b0b-4d33-b552-c0f2ed3e567b (api:52) 29429:2018-03-21 10:43:30,122+0200 INFO (periodic/3) [vdsm.api] START = multipath_health() from=3Dinternal, = task_id=3Dacde7e7d-ab4a-4811-934f-4d312da33208 (api:46) 29430:2018-03-21 10:43:30,123+0200 INFO (periodic/3) [vdsm.api] FINISH = multipath_health return=3D{} from=3Dinternal, = task_id=3Dacde7e7d-ab4a-4811-934f-4d312da33208 (api:52) 29431:2018-03-21 10:43:35,393+0200 INFO (jsonrpc/7) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29432:2018-03-21 10:43:35,394+0200 INFO (jsonrpc/7) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 29433:2018-03-21 10:43:35,394+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29434:2018-03-21 10:43:36,671+0200 INFO (jsonrpc/2) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 29435:2018-03-21 10:43:36,671+0200 INFO (jsonrpc/2) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 29436:2018-03-21 10:43:36,672+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29437:2018-03-21 10:43:37,356+0200 INFO (jsonrpc/1) [api.host] START = getStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29438:2018-03-21 10:43:37,357+0200 INFO (jsonrpc/1) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3Dbc14e253-d3d4-4cc3-9ae1-c0e6ac6e37f7 (api:46) 29439:2018-03-21 10:43:37,357+0200 INFO (jsonrpc/1) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000402077', = 'lastCheck': '7.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': = '7.7', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3Dbc14e253-d3d4-4cc3-9ae1-c0e6ac6e37f7 (api:52) 29440:2018-03-21 10:43:37,358+0200 INFO (jsonrpc/1) [vdsm.api] START = multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D23c698b8-d5bd-4746-b738-97b10f3e4bfb (api:46) 29441:2018-03-21 10:43:37,358+0200 INFO (jsonrpc/1) [vdsm.api] FINISH = multipath_health return=3D{} from=3D::ffff:192.168.0.4,49914, = task_id=3D23c698b8-d5bd-4746-b738-97b10f3e4bfb (api:52) 29442:2018-03-21 10:43:37,365+0200 INFO (jsonrpc/1) [api.host] FINISH = getStats return=3D{'status': {'message': 'Done', 'code': 0}, 'info': = {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.07', 'cpuIdle': '99.86'}, '10': {'cpuUser': '3.06', 'nodeIndex': 0, = 'cpuSys': '0.80', 'cpuIdle': '96.14'}, '13': {'cpuUser': '0.00', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '12': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '15': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.87'}, '14': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.40', = 'cpuIdle': '99.33'}, '1': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': = '1.33', 'cpuIdle': '97.54'}, '0': {'cpuUser': '0.47', 'nodeIndex': 0, = 'cpuSys': '0.33', 'cpuIdle': '99.20'}, '3': {'cpuUser': '0.00', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '2': {'cpuUser': = '0.67', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '98.93'}, '5': = {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.74'}, '4': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.40', = 'cpuIdle': '99.13'}, '7': {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': = '0.73', 'cpuIdle': '98.94'}, '6': {'cpuUser': '5.39', 'nodeIndex': 0, = 'cpuSys': '0.47', 'cpuIdle': '94.14'}, '9': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '8': {'cpuUser': = '0.53', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.07'}}, = 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15143'}, '0': = {'memPercent': 5, 'memFree': '15348'}}, 'memShared': 0, 'thpState': = 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', = 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000402077', = 'lastCheck': '7.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': = '7.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': = {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621810.108511, 'name': 'enp3s0f0', 'tx': '1091305295', 'txDropped': = '0', 'rx': '11914880953', 'rxErrors': '0', 'speed': '100', 'rxDropped': = '1442'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621810.108511, 'name': 'ovirtmgmt', 'tx': '1048655897', 'txDropped': = '0', 'rx': '11615220341', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621810.108511, 'name': 'lo', 'tx': '58078891247', 'txDropped': '0', = 'rx': '58078891247', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621810.108511, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621810.108511, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621810.108511, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621810.108511, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, = 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, = 'elapsedTime': '76376.19', 'cpuLoad': '0.16', 'cpuSys': '0.39', = 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': = '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.00', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1442', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, = 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, = 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:43:37 GMT', 'cpuUser': '0.79', = 'memFree': 31336, 'cpuIdle': '98.82', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.73'}} from=3D::ffff:192.168.0.4,49914 (api:52) 29443:2018-03-21 10:43:37,367+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573) 29444:2018-03-21 10:43:38,124+0200 INFO (jsonrpc/5) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D9d2a82dd-c76c-4eb5-a1e1-b49bf004384e (api:46) 29445:2018-03-21 10:43:38,128+0200 INFO (jsonrpc/5) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D9d2a82dd-c76c-4eb5-a1e1-b49bf004384e (api:52) 29446:2018-03-21 10:43:38,129+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573) 29447:2018-03-21 10:43:38,171+0200 INFO (jsonrpc/3) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D62d44b0b-20b2-47ae-bfb8-d4717e301b69 (api:46) 29448:2018-03-21 10:43:38,177+0200 INFO (jsonrpc/3) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D62d44b0b-20b2-47ae-bfb8-d4717e301b69 (api:52) 29449:2018-03-21 10:43:38,178+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573) 29450:2018-03-21 10:43:45,147+0200 INFO (periodic/2) [vdsm.api] START = repoStats(domains=3D()) from=3Dinternal, = task_id=3D49c33285-a222-464d-8984-4e75a5c6354a (api:46) 29451:2018-03-21 10:43:45,148+0200 INFO (periodic/2) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000454102', = 'lastCheck': '5.6', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000347381', 'lastCheck': = '5.5', 'valid': True}} from=3Dinternal, = task_id=3D49c33285-a222-464d-8984-4e75a5c6354a (api:52) 29452:2018-03-21 10:43:45,148+0200 INFO (periodic/2) [vdsm.api] START = multipath_health() from=3Dinternal, = task_id=3Deed6b564-76b1-4f9b-80a2-1f0f6d9bbe32 (api:46) 29453:2018-03-21 10:43:45,149+0200 INFO (periodic/2) [vdsm.api] FINISH = multipath_health return=3D{} from=3Dinternal, = task_id=3Deed6b564-76b1-4f9b-80a2-1f0f6d9bbe32 (api:52) 29454:2018-03-21 10:43:48,405+0200 INFO (jsonrpc/0) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D84651a37-7d13-41a9-9dcf-2169a9c46fc5 (api:46) 29455:2018-03-21 10:43:48,410+0200 INFO (jsonrpc/0) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D84651a37-7d13-41a9-9dcf-2169a9c46fc5 (api:52) 29456:2018-03-21 10:43:48,411+0200 INFO (jsonrpc/0) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573) 29457:2018-03-21 10:43:48,417+0200 INFO (jsonrpc/4) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D4ebd1d1a-11ac-41ee-b459-7e45f008776b (api:46) 29458:2018-03-21 10:43:48,423+0200 INFO (jsonrpc/4) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D4ebd1d1a-11ac-41ee-b459-7e45f008776b (api:52) 29459:2018-03-21 10:43:48,424+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573) 29460:2018-03-21 10:43:50,399+0200 INFO (jsonrpc/6) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29461:2018-03-21 10:43:50,400+0200 INFO (jsonrpc/6) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 29462:2018-03-21 10:43:50,400+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29463:2018-03-21 10:43:51,694+0200 INFO (jsonrpc/7) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 29464:2018-03-21 10:43:51,694+0200 INFO (jsonrpc/7) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 29465:2018-03-21 10:43:51,695+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29466:2018-03-21 10:43:52,516+0200 INFO (jsonrpc/2) [api.host] START = getStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29467:2018-03-21 10:43:52,517+0200 INFO (jsonrpc/2) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D0a88f0dc-eccf-4d24-9ac2-22e079a1480a (api:46) 29468:2018-03-21 10:43:52,517+0200 INFO (jsonrpc/2) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000487937', = 'lastCheck': '3.0', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000455956', 'lastCheck': = '2.9', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D0a88f0dc-eccf-4d24-9ac2-22e079a1480a (api:52) 29469:2018-03-21 10:43:52,518+0200 INFO (jsonrpc/2) [vdsm.api] START = multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3Da7a44ff7-a137-41b6-9ad5-2c0b32ab79ec (api:46) 29470:2018-03-21 10:43:52,518+0200 INFO (jsonrpc/2) [vdsm.api] FINISH = multipath_health return=3D{} from=3D::ffff:192.168.0.4,49914, = task_id=3Da7a44ff7-a137-41b6-9ad5-2c0b32ab79ec (api:52) 29471:2018-03-21 10:43:52,524+0200 INFO (jsonrpc/2) [api.host] FINISH = getStats return=3D{'status': {'message': 'Done', 'code': 0}, 'info': = {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.80'}, '10': {'cpuUser': '0.00', 'nodeIndex': 0, = 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '13': {'cpuUser': '0.00', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '12': {'cpuUser': = '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '15': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.86'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', = 'cpuIdle': '99.87'}, '1': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': = '1.13', 'cpuIdle': '97.74'}, '0': {'cpuUser': '0.33', 'nodeIndex': 0, = 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '3': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '2': {'cpuUser': = '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '5': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.87'}, '4': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.13', = 'cpuIdle': '99.60'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': = '0.33', 'cpuIdle': '99.67'}, '6': {'cpuUser': '0.00', 'nodeIndex': 0, = 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '9': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '8': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}}, = 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15143'}, '0': = {'memPercent': 5, 'memFree': '15349'}}, 'memShared': 0, 'thpState': = 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', = 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000487937', = 'lastCheck': '3.0', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000455956', 'lastCheck': = '2.9', 'valid': True}}, 'incomingVmMigrations': 0, 'network': = {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621825.13588, 'name': 'enp3s0f0', 'tx': '1091312739', 'txDropped': = '0', 'rx': '11914899051', 'rxErrors': '0', 'speed': '100', 'rxDropped': = '1443'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621825.13588, 'name': 'ovirtmgmt', 'tx': '1048663107', 'txDropped': = '0', 'rx': '11615236520', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621825.13588, 'name': 'lo', 'tx': '58090295494', 'txDropped': '0', = 'rx': '58090295494', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621825.13588, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': = '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621825.13588, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621825.13588, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': = '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': = {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621825.13588, = 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': = '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', = 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76391.35', = 'cpuLoad': '0.15', 'cpuSys': '0.20', 'diskStats': {'/var/log': {'free': = '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, = 'cpuUserVdsmd': '1.13', 'netConfigDirty': 'False', 'memCommitted': 0, = 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, = 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': = False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': = False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': = '1443', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': = 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:43:52 GMT', 'cpuUser': '0.14', = 'memFree': 31336, 'cpuIdle': '99.66', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.73'}} from=3D::ffff:192.168.0.4,49914 (api:52) 29472:2018-03-21 10:43:52,526+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573) 29473:2018-03-21 10:43:58,686+0200 INFO (jsonrpc/1) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3Dd6d29cf6-c083-4639-91a2-f062eae4e629 (api:46) 29474:2018-03-21 10:43:58,690+0200 INFO (jsonrpc/1) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3Dd6d29cf6-c083-4639-91a2-f062eae4e629 (api:52) 29475:2018-03-21 10:43:58,690+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573) 29476:2018-03-21 10:43:58,697+0200 INFO (jsonrpc/5) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D9dbd91df-dd9c-469c-8308-bfbfc228937c (api:46) 29477:2018-03-21 10:43:58,703+0200 INFO (jsonrpc/5) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D9dbd91df-dd9c-469c-8308-bfbfc228937c (api:52) 29478:2018-03-21 10:43:58,704+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573) grep -n 10:46 vdsm.log | tail -1000 1017:2018-03-21 06:10:46,882+0200 INFO (jsonrpc/2) [api.host] START = getStats() from=3D::ffff:192.168.0.4,49914 (api:46) 1018:2018-03-21 06:10:46,883+0200 INFO (jsonrpc/2) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D6e045b00-88b2-469b-80d6-af38303f8c32 (api:46) 1019:2018-03-21 06:10:46,883+0200 INFO (jsonrpc/2) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000332785', = 'lastCheck': '7.3', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000203914', 'lastCheck': = '5.6', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D6e045b00-88b2-469b-80d6-af38303f8c32 (api:52) 1020:2018-03-21 06:10:46,884+0200 INFO (jsonrpc/2) [vdsm.api] START = multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D475a5b40-4a43-42ee-9524-2bdd220aed83 (api:46) 1021:2018-03-21 06:10:46,884+0200 INFO (jsonrpc/2) [vdsm.api] FINISH = multipath_health return=3D{} from=3D::ffff:192.168.0.4,49914, = task_id=3D475a5b40-4a43-42ee-9524-2bdd220aed83 (api:52) 1022:2018-03-21 06:10:46,889+0200 INFO (jsonrpc/2) [api.host] FINISH = getStats return=3D{'status': {'message': 'Done', 'code': 0}, 'info': = {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.20', 'cpuIdle': '99.73'}, '10': {'cpuUser': '0.13', 'nodeIndex': 0, = 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '13': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '12': {'cpuUser': = '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '15': = {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '99.40'}, '14': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.40', = 'cpuIdle': '99.13'}, '1': {'cpuUser': '3.00', 'nodeIndex': 1, 'cpuSys': = '1.40', 'cpuIdle': '95.60'}, '0': {'cpuUser': '0.73', 'nodeIndex': 0, = 'cpuSys': '0.20', 'cpuIdle': '99.07'}, '3': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '2': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '5': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': = '99.73'}, '4': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', = 'cpuIdle': '99.54'}, '7': {'cpuUser': '0.80', 'nodeIndex': 1, 'cpuSys': = '0.80', 'cpuIdle': '98.40'}, '6': {'cpuUser': '5.46', 'nodeIndex': 0, = 'cpuSys': '0.27', 'cpuIdle': '94.27'}, '9': {'cpuUser': '0.20', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '8': {'cpuUser': = '1.27', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '98.26'}}, = 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15128'}, '0': = {'memPercent': 5, 'memFree': '15366'}}, 'memShared': 0, 'thpState': = 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', = 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000332785', = 'lastCheck': '7.3', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000203914', 'lastCheck': = '5.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': = {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521605437.203776, 'name': 'enp3s0f0', 'tx': '1082264809', 'txDropped': = '0', 'rx': '11907188594', 'rxErrors': '0', 'speed': '100', 'rxDropped': = '289'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521605437.203776, 'name': 'ovirtmgmt', 'tx': '1039869554', 'txDropped': = '0', 'rx': '11608697383', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521605437.203776, 'name': 'lo', 'tx': '45587320171', 'txDropped': '0', = 'rx': '45587320171', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521605437.203776, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '53620', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521605437.203776, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521605437.203776, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521605437.203776, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, = 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, = 'elapsedTime': '60005.72', 'cpuLoad': '0.06', 'cpuSys': '0.38', = 'diskStats': {'/var/log': {'free': '7357'}, '/var/run/vdsm/': {'free': = '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31084, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '289', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, = 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, = 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T04:10:46 GMT', 'cpuUser': '0.83', = 'memFree': 31340, 'cpuIdle': '98.79', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.80'}} from=3D::ffff:192.168.0.4,49914 (api:52) 1023:2018-03-21 06:10:46,891+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573) 13522:2018-03-21 08:10:46,827+0200 INFO (jsonrpc/7) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 13523:2018-03-21 08:10:46,828+0200 INFO (jsonrpc/7) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 13524:2018-03-21 08:10:46,828+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29734:2018-03-21 10:46:00,338+0200 INFO (periodic/2) [vdsm.api] START = repoStats(domains=3D()) from=3Dinternal, = task_id=3Daf099ce6-0f44-461b-9e8e-99de63b3884f (api:46) 29735:2018-03-21 10:46:00,339+0200 INFO (periodic/2) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470245', = 'lastCheck': '0.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000476935', 'lastCheck': = '0.7', 'valid': True}} from=3Dinternal, = task_id=3Daf099ce6-0f44-461b-9e8e-99de63b3884f (api:52) 29736:2018-03-21 10:46:00,339+0200 INFO (periodic/2) [vdsm.api] START = multipath_health() from=3Dinternal, = task_id=3D4a5709d9-e322-44eb-a8a4-bbe3f5e5e1bd (api:46) 29737:2018-03-21 10:46:00,340+0200 INFO (periodic/2) [vdsm.api] FINISH = multipath_health return=3D{} from=3Dinternal, = task_id=3D4a5709d9-e322-44eb-a8a4-bbe3f5e5e1bd (api:52) 29738:2018-03-21 10:46:04,637+0200 INFO (jsonrpc/7) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D5b9300f7-f8c4-4f63-a49c-45ca8119d3b9 (api:46) 29739:2018-03-21 10:46:04,642+0200 INFO (jsonrpc/7) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D5b9300f7-f8c4-4f63-a49c-45ca8119d3b9 (api:52) 29740:2018-03-21 10:46:04,642+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573) 29741:2018-03-21 10:46:04,648+0200 INFO (jsonrpc/1) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3Df4db5bfa-e131-4f9a-baaa-e7555c996ab0 (api:46) 29742:2018-03-21 10:46:04,655+0200 INFO (jsonrpc/1) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3Df4db5bfa-e131-4f9a-baaa-e7555c996ab0 (api:52) 29743:2018-03-21 10:46:04,656+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 = seconds (__init__:573) 29744:2018-03-21 10:46:05,451+0200 INFO (jsonrpc/5) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29745:2018-03-21 10:46:05,452+0200 INFO (jsonrpc/5) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 29746:2018-03-21 10:46:05,452+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29747:2018-03-21 10:46:06,903+0200 INFO (jsonrpc/3) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 29748:2018-03-21 10:46:06,904+0200 INFO (jsonrpc/3) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 29749:2018-03-21 10:46:06,904+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29750:2018-03-21 10:46:08,791+0200 WARN (vdsm.Scheduler) [Executor] = Worker blocked: <Worker name=3Djsonrpc/6 running <Task <JsonRpcTask = {'params': {u'connectionParams': [{u'id': = u'461f65a9-3a81-4f3f-a46d-c5ed12520524', u'connection': = u'node10.starlett.lv:/vmdata/nfs/exports', u'iqn': u'', u'user': u'', = u'tpgt': u'1', u'protocol_version': u'auto', u'password': '********', = u'port': u''}], u'storagepoolID': = u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': = '2.0', 'method': u'StoragePool.connectStorageServer', 'id': = u'71143eef-aac6-4996-9783-3e0e3da180c3'} at 0x3604fd0> timeout=3D60, = duration=3D120 at 0x3604e10> task#=3D3762 at 0x3541650>, traceback: 29801:2018-03-21 10:46:12,852+0200 INFO (jsonrpc/0) [api.host] START = getStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29802:2018-03-21 10:46:12,853+0200 INFO (jsonrpc/0) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D391fdab9-d3c2-4075-9c34-177a07a21ec3 (api:46) 29803:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000492277', = 'lastCheck': '3.3', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': = '3.2', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D391fdab9-d3c2-4075-9c34-177a07a21ec3 (api:52) 29804:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) [vdsm.api] START = multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D7e53adbc-3974-4f51-bfc8-8fbd3ca6b749 (api:46) 29805:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) [vdsm.api] FINISH = multipath_health return=3D{} from=3D::ffff:192.168.0.4,49914, = task_id=3D7e53adbc-3974-4f51-bfc8-8fbd3ca6b749 (api:52) 29806:2018-03-21 10:46:12,861+0200 INFO (jsonrpc/0) [api.host] FINISH = getStats return=3D{'status': {'message': 'Done', 'code': 0}, 'info': = {'cpuStatistics': {'11': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': = '0.40', 'cpuIdle': '99.13'}, '10': {'cpuUser': '0.13', 'nodeIndex': 0, = 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '13': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '12': {'cpuUser': = '0.20', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '15': = {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '99.60'}, '14': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', = 'cpuIdle': '99.54'}, '1': {'cpuUser': '1.00', 'nodeIndex': 1, 'cpuSys': = '1.33', 'cpuIdle': '97.67'}, '0': {'cpuUser': '0.13', 'nodeIndex': 0, = 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '3': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '2': {'cpuUser': = '0.20', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.47'}, '5': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.80'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', = 'cpuIdle': '99.73'}, '7': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.40', 'cpuIdle': '99.53'}, '6': {'cpuUser': '0.47', 'nodeIndex': 0, = 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '9': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '8': {'cpuUser': = '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}}, = 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15142'}, '0': = {'memPercent': 5, 'memFree': '15348'}}, 'memShared': 0, 'thpState': = 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', = 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000492277', = 'lastCheck': '3.3', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': = '3.2', 'valid': True}}, 'incomingVmMigrations': 0, 'network': = {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621960.326184, 'name': 'enp3s0f0', 'tx': '1091387790', 'txDropped': = '0', 'rx': '11914959714', 'rxErrors': '0', 'speed': '100', 'rxDropped': = '1451'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621960.326184, 'name': 'ovirtmgmt', 'tx': '1048735814', 'txDropped': = '0', 'rx': '11615289173', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621960.326184, 'name': 'lo', 'tx': '58192068637', 'txDropped': '0', = 'rx': '58192068637', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621960.326184, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621960.326184, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621960.326184, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621960.326184, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, = 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, = 'elapsedTime': '76531.69', 'cpuLoad': '0.17', 'cpuSys': '0.32', = 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': = '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.00', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31081, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1451', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, = 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, = 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:46:12 GMT', 'cpuUser': '0.21', = 'memFree': 31337, 'cpuIdle': '99.46', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.80'}} from=3D::ffff:192.168.0.4,49914 (api:52) 29807:2018-03-21 10:46:12,863+0200 INFO (jsonrpc/0) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573) 29808:2018-03-21 10:46:13,829+0200 ERROR (jsonrpc/6) [storage.HSM] Could = not connect to storageServer (hsm:2407) 29828:2018-03-21 10:46:13,829+0200 INFO (jsonrpc/6) [vdsm.api] FINISH = connectStorageServer return=3D{'statuslist': [{'status': 477, 'id': = u'461f65a9-3a81-4f3f-a46d-c5ed12520524'}]} = from=3D::ffff:192.168.0.4,49914, flow_id=3D1435fc81, = task_id=3D0a828d2c-d9f4-4f83-a9e9-7393159d5323 (api:52) 29829:2018-03-21 10:46:13,830+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer = succeeded in 125.04 seconds (__init__:573) 29830:2018-03-21 10:46:14,767+0200 INFO (jsonrpc/4) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D49cd2ae6-7943-47c9-ad86-9e0b7e58bca3 (api:46) 29831:2018-03-21 10:46:14,772+0200 INFO (jsonrpc/4) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D49cd2ae6-7943-47c9-ad86-9e0b7e58bca3 (api:52) 29832:2018-03-21 10:46:14,773+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573) 29833:2018-03-21 10:46:14,815+0200 INFO (jsonrpc/2) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D19909616-efb4-490d-982c-66eda9ca4381 (api:46) 29834:2018-03-21 10:46:14,822+0200 INFO (jsonrpc/2) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D19909616-efb4-490d-982c-66eda9ca4381 (api:52) 29835:2018-03-21 10:46:14,823+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573) 29836:2018-03-21 10:46:15,360+0200 INFO (periodic/2) [vdsm.api] START = repoStats(domains=3D()) from=3Dinternal, = task_id=3Dda74ea9f-164b-450d-b836-6818caa3fdc5 (api:46) 29837:2018-03-21 10:46:15,360+0200 INFO (periodic/2) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000492277', = 'lastCheck': '5.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': = '5.7', 'valid': True}} from=3Dinternal, = task_id=3Dda74ea9f-164b-450d-b836-6818caa3fdc5 (api:52) 29838:2018-03-21 10:46:15,361+0200 INFO (periodic/2) [vdsm.api] START = multipath_health() from=3Dinternal, = task_id=3D06245e2b-e8ca-41c4-90bf-4294d0c699b8 (api:46) 29839:2018-03-21 10:46:15,361+0200 INFO (periodic/2) [vdsm.api] FINISH = multipath_health return=3D{} from=3Dinternal, = task_id=3D06245e2b-e8ca-41c4-90bf-4294d0c699b8 (api:52) 29840:2018-03-21 10:46:20,456+0200 INFO (jsonrpc/7) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29841:2018-03-21 10:46:20,457+0200 INFO (jsonrpc/7) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 29842:2018-03-21 10:46:20,457+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 = seconds (__init__:573) 29843:2018-03-21 10:46:21,925+0200 INFO (jsonrpc/1) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 29844:2018-03-21 10:46:21,926+0200 INFO (jsonrpc/1) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 29845:2018-03-21 10:46:21,926+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29846:2018-03-21 10:46:24,930+0200 INFO (jsonrpc/5) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3Dd964535d-e5eb-42bc-b324-432a05c364da (api:46) 29847:2018-03-21 10:46:24,935+0200 INFO (jsonrpc/5) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3Dd964535d-e5eb-42bc-b324-432a05c364da (api:52) 29848:2018-03-21 10:46:24,935+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.00 seconds (__init__:573) 29849:2018-03-21 10:46:24,987+0200 INFO (jsonrpc/3) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D167dcd23-04b5-4de3-9c81-7d25ba56407a (api:46) 29850:2018-03-21 10:46:25,022+0200 INFO (jsonrpc/3) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D167dcd23-04b5-4de3-9c81-7d25ba56407a (api:52) 29851:2018-03-21 10:46:25,023+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.03 = seconds (__init__:573) 29852:2018-03-21 10:46:28,010+0200 INFO (jsonrpc/0) [api.host] START = getStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29853:2018-03-21 10:46:28,011+0200 INFO (jsonrpc/0) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D011616ce-53c9-4cfc-8b29-247b1be03409 (api:46) 29854:2018-03-21 10:46:28,011+0200 INFO (jsonrpc/0) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000218955', = 'lastCheck': '8.4', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000351741', 'lastCheck': = '8.4', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D011616ce-53c9-4cfc-8b29-247b1be03409 (api:52) 29855:2018-03-21 10:46:28,012+0200 INFO (jsonrpc/0) [vdsm.api] START = multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D85b42a8d-1bf1-495d-ad32-a1f9710a6468 (api:46) 29856:2018-03-21 10:46:28,012+0200 INFO (jsonrpc/0) [vdsm.api] FINISH = multipath_health return=3D{} from=3D::ffff:192.168.0.4,49914, = task_id=3D85b42a8d-1bf1-495d-ad32-a1f9710a6468 (api:52) 29857:2018-03-21 10:46:28,018+0200 INFO (jsonrpc/0) [api.host] FINISH = getStats return=3D{'status': {'message': 'Done', 'code': 0}, 'info': = {'cpuStatistics': {'11': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': = '0.20', 'cpuIdle': '99.67'}, '10': {'cpuUser': '1.73', 'nodeIndex': 0, = 'cpuSys': '0.53', 'cpuIdle': '97.74'}, '13': {'cpuUser': '0.20', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '12': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '15': = {'cpuUser': '0.67', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '99.06'}, '14': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.33', = 'cpuIdle': '99.47'}, '1': {'cpuUser': '1.26', 'nodeIndex': 1, 'cpuSys': = '1.33', 'cpuIdle': '97.41'}, '0': {'cpuUser': '0.40', 'nodeIndex': 0, = 'cpuSys': '0.07', 'cpuIdle': '99.53'}, '3': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '2': {'cpuUser': = '0.00', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.73'}, '5': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.87'}, '4': {'cpuUser': '13.78', 'nodeIndex': 0, 'cpuSys': '0.80', = 'cpuIdle': '85.42'}, '7': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': = '0.67', 'cpuIdle': '98.86'}, '6': {'cpuUser': '1.07', 'nodeIndex': 0, = 'cpuSys': '0.27', 'cpuIdle': '98.66'}, '9': {'cpuUser': '1.20', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '98.53'}, '8': {'cpuUser': = '0.80', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '98.87'}}, = 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15136'}, '0': = {'memPercent': 5, 'memFree': '15357'}}, 'memShared': 0, 'thpState': = 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', = 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000218955', = 'lastCheck': '8.4', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000351741', 'lastCheck': = '8.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': = {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621975.346633, 'name': 'enp3s0f0', 'tx': '1091397599', 'txDropped': = '0', 'rx': '11914966591', 'rxErrors': '0', 'speed': '100', 'rxDropped': = '1454'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621975.346633, 'name': 'ovirtmgmt', 'tx': '1048745347', 'txDropped': = '0', 'rx': '11615294897', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621975.346633, 'name': 'lo', 'tx': '58205583384', 'txDropped': '0', = 'rx': '58205583384', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621975.346633, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621975.346633, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621975.346633, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621975.346633, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, = 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, = 'elapsedTime': '76546.85', 'cpuLoad': '0.18', 'cpuSys': '0.39', = 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': = '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.20', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31031, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1454', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, = 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, = 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:46:28 GMT', 'cpuUser': '1.38', = 'memFree': 31287, 'cpuIdle': '98.23', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.87'}} from=3D::ffff:192.168.0.4,49914 (api:52) 29858:2018-03-21 10:46:28,020+0200 INFO (jsonrpc/0) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573) 29859:2018-03-21 10:46:29,667+0200 INFO (itmap/0) [IOProcessClient] = Starting client ioprocess-575 (__init__:308) 29860:2018-03-21 10:46:29,700+0200 INFO (itmap/1) [IOProcessClient] = Starting client ioprocess-576 (__init__:308) 29861:2018-03-21 10:46:29,714+0200 INFO (ioprocess/1711202) [IOProcess] = Starting ioprocess (__init__:437) 29862:2018-03-21 10:46:29,725+0200 INFO (ioprocess/1711208) [IOProcess] = Starting ioprocess (__init__:437) 29863:2018-03-21 10:46:30,375+0200 INFO (periodic/0) [vdsm.api] START = repoStats(domains=3D()) from=3Dinternal, = task_id=3Dbdacf1c0-3fcb-43b1-82e2-a041a26d0d7f (api:46) 29864:2018-03-21 10:46:30,376+0200 INFO (periodic/0) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00031111', = 'lastCheck': '0.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.00045594', 'lastCheck': = '0.6', 'valid': True}} from=3Dinternal, = task_id=3Dbdacf1c0-3fcb-43b1-82e2-a041a26d0d7f (api:52) 29865:2018-03-21 10:46:30,376+0200 INFO (periodic/0) [vdsm.api] START = multipath_health() from=3Dinternal, = task_id=3D3d3cad70-e5c8-49b2-9f5d-92a249cc102d (api:46) 29866:2018-03-21 10:46:30,377+0200 INFO (periodic/0) [vdsm.api] FINISH = multipath_health return=3D{} from=3Dinternal, = task_id=3D3d3cad70-e5c8-49b2-9f5d-92a249cc102d (api:52) 29867:2018-03-21 10:46:35,108+0200 INFO (jsonrpc/6) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D304d7e7c-03fc-44d6-be32-d083ada09b30 (api:46) 29868:2018-03-21 10:46:35,114+0200 INFO (jsonrpc/6) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D304d7e7c-03fc-44d6-be32-d083ada09b30 (api:52) 29869:2018-03-21 10:46:35,114+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.00 seconds (__init__:573) 29870:2018-03-21 10:46:35,170+0200 INFO (jsonrpc/4) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D76736a93-d4c4-4b2f-9848-7b8731fe2b67 (api:46) 29871:2018-03-21 10:46:35,176+0200 INFO (jsonrpc/4) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D76736a93-d4c4-4b2f-9848-7b8731fe2b67 (api:52) 29872:2018-03-21 10:46:35,177+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573) 29873:2018-03-21 10:46:35,462+0200 INFO (jsonrpc/2) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29874:2018-03-21 10:46:35,462+0200 INFO (jsonrpc/2) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 29875:2018-03-21 10:46:35,463+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29876:2018-03-21 10:46:36,948+0200 INFO (jsonrpc/7) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 29877:2018-03-21 10:46:36,948+0200 INFO (jsonrpc/7) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 29878:2018-03-21 10:46:36,949+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29879:2018-03-21 10:46:43,179+0200 INFO (jsonrpc/1) [api.host] START = getStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29880:2018-03-21 10:46:43,180+0200 INFO (jsonrpc/1) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D83bbc95c-6d59-4eec-82b1-e8993be28759 (api:46) 29881:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000486101', = 'lastCheck': '3.6', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': = '3.4', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D83bbc95c-6d59-4eec-82b1-e8993be28759 (api:52) 29882:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) [vdsm.api] START = multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D9b781324-b50b-49a9-bf31-5eca835dc75c (api:46) 29883:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) [vdsm.api] FINISH = multipath_health return=3D{} from=3D::ffff:192.168.0.4,49914, = task_id=3D9b781324-b50b-49a9-bf31-5eca835dc75c (api:52) 29884:2018-03-21 10:46:43,188+0200 INFO (jsonrpc/1) [api.host] FINISH = getStats return=3D{'status': {'message': 'Done', 'code': 0}, 'info': = {'cpuStatistics': {'11': {'cpuUser': '0.40', 'nodeIndex': 1, 'cpuSys': = '0.27', 'cpuIdle': '99.33'}, '10': {'cpuUser': '0.60', 'nodeIndex': 0, = 'cpuSys': '0.47', 'cpuIdle': '98.93'}, '13': {'cpuUser': '0.00', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.73'}, '12': {'cpuUser': = '1.46', 'nodeIndex': 0, 'cpuSys': '0.60', 'cpuIdle': '97.94'}, '15': = {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': = '99.47'}, '14': {'cpuUser': '0.40', 'nodeIndex': 0, 'cpuSys': '0.47', = 'cpuIdle': '99.13'}, '1': {'cpuUser': '1.33', 'nodeIndex': 1, 'cpuSys': = '1.93', 'cpuIdle': '96.74'}, '0': {'cpuUser': '0.60', 'nodeIndex': 0, = 'cpuSys': '0.27', 'cpuIdle': '99.13'}, '3': {'cpuUser': '0.20', = 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.40'}, '2': {'cpuUser': = '6.99', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '92.61'}, '5': = {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': = '99.67'}, '4': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.27', = 'cpuIdle': '99.46'}, '7': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': = '0.60', 'cpuIdle': '98.93'}, '6': {'cpuUser': '0.53', 'nodeIndex': 0, = 'cpuSys': '0.27', 'cpuIdle': '99.20'}, '9': {'cpuUser': '0.47', = 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '99.20'}, '8': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}}, = 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15131'}, '0': = {'memPercent': 5, 'memFree': '15357'}}, 'memShared': 0, 'thpState': = 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', = 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000486101', = 'lastCheck': '3.6', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': = '3.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': = {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621990.363726, 'name': 'enp3s0f0', 'tx': '1091404888', 'txDropped': = '0', 'rx': '11914970667', 'rxErrors': '0', 'speed': '100', 'rxDropped': = '1455'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621990.363726, 'name': 'ovirtmgmt', 'tx': '1048752406', 'txDropped': = '0', 'rx': '11615298347', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621990.363726, 'name': 'lo', 'tx': '58215969455', 'txDropped': '0', = 'rx': '58215969455', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521621990.363726, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621990.363726, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621990.363726, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521621990.363726, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, = 'txDropped': '0', 'anonHugePages': '206', 'ksmPages': 100, = 'elapsedTime': '76562.02', 'cpuLoad': '0.18', 'cpuSys': '0.45', = 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': = '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.27', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1455', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, = 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, = 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:46:43 GMT', 'cpuUser': '0.87', = 'memFree': 31336, 'cpuIdle': '98.68', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '1.20'}} from=3D::ffff:192.168.0.4,49914 (api:52) 29885:2018-03-21 10:46:43,190+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573) 29886:2018-03-21 10:46:45,288+0200 INFO (jsonrpc/5) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3Dffe57349-2532-4b64-b32c-8a558273d8e5 (api:46) 29887:2018-03-21 10:46:45,292+0200 INFO (jsonrpc/5) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3Dffe57349-2532-4b64-b32c-8a558273d8e5 (api:52) 29888:2018-03-21 10:46:45,293+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.00 seconds (__init__:573) 29889:2018-03-21 10:46:45,298+0200 INFO (jsonrpc/3) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3Db2ea28f0-abf0-4258-8152-6035d93aa5dc (api:46) 29890:2018-03-21 10:46:45,305+0200 INFO (jsonrpc/3) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3Db2ea28f0-abf0-4258-8152-6035d93aa5dc (api:52) 29891:2018-03-21 10:46:45,306+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 = seconds (__init__:573) 29892:2018-03-21 10:46:45,395+0200 INFO (periodic/2) [vdsm.api] START = repoStats(domains=3D()) from=3Dinternal, = task_id=3D4010bf72-5c86-4daf-a8f9-12ec229a86e4 (api:46) 29893:2018-03-21 10:46:45,395+0200 INFO (periodic/2) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000486101', = 'lastCheck': '5.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': = '5.7', 'valid': True}} from=3Dinternal, = task_id=3D4010bf72-5c86-4daf-a8f9-12ec229a86e4 (api:52) 29894:2018-03-21 10:46:45,396+0200 INFO (periodic/2) [vdsm.api] START = multipath_health() from=3Dinternal, = task_id=3D37a1132f-452b-4805-9cc0-b8e74e8d6b02 (api:46) 29895:2018-03-21 10:46:45,396+0200 INFO (periodic/2) [vdsm.api] FINISH = multipath_health return=3D{} from=3Dinternal, = task_id=3D37a1132f-452b-4805-9cc0-b8e74e8d6b02 (api:52) 29896:2018-03-21 10:46:50,467+0200 INFO (jsonrpc/0) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29897:2018-03-21 10:46:50,468+0200 INFO (jsonrpc/0) [throttled] Current = getAllVmStats: {} (throttledlog:103) 29898:2018-03-21 10:46:50,468+0200 INFO (jsonrpc/0) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52) 29899:2018-03-21 10:46:50,468+0200 INFO (jsonrpc/0) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 = seconds (__init__:573) 29900:2018-03-21 10:46:51,967+0200 INFO (jsonrpc/6) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46) 29901:2018-03-21 10:46:51,968+0200 INFO (jsonrpc/6) [api.host] FINISH = getAllVmStats return=3D{'status': {'message': 'Done', 'code': 0}, = 'statsList': (suppressed)} from=3D::1,36114 (api:52) 29902:2018-03-21 10:46:51,968+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573) 29903:2018-03-21 10:46:55,648+0200 INFO (jsonrpc/4) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3Da975635f-59d7-4f40-b9a7-a5ea7150368e (api:46) 29904:2018-03-21 10:46:55,653+0200 INFO (jsonrpc/4) [vdsm.api] FINISH = getSpmStatus return=3D{'spm_st': {'spmId': 1, 'spmStatus': 'SPM', = 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3Da975635f-59d7-4f40-b9a7-a5ea7150368e (api:52) 29905:2018-03-21 10:46:55,653+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.00 seconds (__init__:573) 29906:2018-03-21 10:46:55,659+0200 INFO (jsonrpc/2) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D97d4ae6c-3719-4149-85fd-28e924ad8883 (api:46) 29907:2018-03-21 10:46:55,666+0200 INFO (jsonrpc/2) [vdsm.api] FINISH = getStoragePoolInfo return=3D{'info': {'name': 'No Description', = 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D97d4ae6c-3719-4149-85fd-28e924ad8883 (api:52) 29908:2018-03-21 10:46:55,667+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 = seconds (__init__:573) 29909:2018-03-21 10:46:59,473+0200 INFO (jsonrpc/7) [api.host] START = getStats() from=3D::ffff:192.168.0.4,49914 (api:46) 29910:2018-03-21 10:46:59,474+0200 INFO (jsonrpc/7) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D177b551b-9bd3-45a4-84d0-5244630c08e5 (api:46) 29911:2018-03-21 10:46:59,474+0200 INFO (jsonrpc/7) [vdsm.api] FINISH = repoStats return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470906', = 'lastCheck': '9.9', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000334928', 'lastCheck': = '9.7', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D177b551b-9bd3-45a4-84d0-5244630c08e5 (api:52) 29912:2018-03-21 10:46:59,475+0200 INFO (jsonrpc/7) [vdsm.api] START = multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D30753d5c-9bfd-4b0f-ac83-e32278d82b63 (api:46) 29913:2018-03-21 10:46:59,475+0200 INFO (jsonrpc/7) [vdsm.api] FINISH = multipath_health return=3D{} from=3D::ffff:192.168.0.4,49914, = task_id=3D30753d5c-9bfd-4b0f-ac83-e32278d82b63 (api:52) 29914:2018-03-21 10:46:59,482+0200 INFO (jsonrpc/7) [api.host] FINISH = getStats return=3D{'status': {'message': 'Done', 'code': 0}, 'info': = {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.80'}, '10': {'cpuUser': '0.07', 'nodeIndex': 0, = 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '13': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '12': {'cpuUser': = '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '15': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.87'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', = 'cpuIdle': '99.73'}, '1': {'cpuUser': '1.07', 'nodeIndex': 1, 'cpuSys': = '1.07', 'cpuIdle': '97.86'}, '0': {'cpuUser': '0.07', 'nodeIndex': 0, = 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '3': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '2': {'cpuUser': = '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '5': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.80'}, '4': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.27', = 'cpuIdle': '99.26'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': = '0.47', 'cpuIdle': '99.53'}, '6': {'cpuUser': '0.20', 'nodeIndex': 0, = 'cpuSys': '0.07', 'cpuIdle': '99.73'}, '9': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '8': {'cpuUser': = '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}}, = 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15125'}, '0': = {'memPercent': 5, 'memFree': '15364'}}, 'memShared': 0, 'thpState': = 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', = 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, = 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470906', = 'lastCheck': '9.9', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000334928', 'lastCheck': = '9.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': = {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521622005.383289, 'name': 'enp3s0f0', 'tx': '1091412332', 'txDropped': = '0', 'rx': '11914976557', 'rxErrors': '0', 'speed': '100', 'rxDropped': = '1456'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521622005.383289, 'name': 'ovirtmgmt', 'tx': '1048761326', 'txDropped': = '0', 'rx': '11615304242', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521622005.383289, 'name': 'lo', 'tx': '58228460774', 'txDropped': '0', = 'rx': '58228460774', 'rxErrors': '0', 'speed': '1000', 'rxDropped': = '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': = 1521622005.383289, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521622005.383289, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521622005.383289, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, = 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': = 1521622005.383289, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', = 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, = 'txDropped': '0', 'anonHugePages': '208', 'ksmPages': 100, = 'elapsedTime': '76578.31', 'cpuLoad': '0.17', 'cpuSys': '0.20', = 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': = '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.13', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31078, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1456', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, = 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, = 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:46:59 GMT', 'cpuUser': '0.15', = 'memFree': 31334, 'cpuIdle': '99.65', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.60'}} from=3D::ffff:192.168.0.4,49914 (api:52) 29915:2018-03-21 10:46:59,484+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573) maintenance mode and detached from DC #1. the Storage Domains node10-NFS-EXPORTS.
2018-03-21 10:46:16,651+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage = Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: = admin@internal-authz) =20 =20 tail -n 1000 engine.log | grep 570ec5d9-fff5-4656-afbd-90b3207a616e 2018-03-21 10:41:14,643+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (default task-2) [570ec5d9-fff5-4656-afbd-90b3207a616e] Lock Acquired = to object = 'EngineLock:{exclusiveLocks=3D'[1d7208ce-d3a1-4406-9638-fe7051562994=3DSTO= RAGE]', sharedLocks=3D''}' 2018-03-21 10:41:16,129+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] Running command: = AttachStorageDomainToPoolCommand internal: false. Entities affected : = ID: 1d7208ce-d3a1-4406-9638-fe7051562994 Type: StorageAction group = MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: = 80cc922f-8dea-4fed-b951-1060ba116ad5 Type: StoragePoolAction group = MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2018-03-21 10:43:23,564+02 ERROR = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] Cannot connect storage connection = server, aborting attach storage domain operation. 2018-03-21 10:43:23,567+02 INFO = [org.ovirt.engine.core.bll.CommandCompensator] = (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] Command = [id=3D921ca7cd-4f93-46aa-8de2-91b13b8f96cb]: Compensating NEW_ENTITY_ID = of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; = snapshot: = StoragePoolIsoMapId:{storagePoolId=3D'80cc922f-8dea-4fed-b951-1060ba116ad5= ', storageId=3D'1d7208ce-d3a1-4406-9638-fe7051562994'}. 2018-03-21 10:43:24,024+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage = Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: = admin@internal-authz) 2018-03-21 10:43:24,114+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-90b3207a616e] Lock freed to object = 'EngineLock:{exclusiveLocks=3D'[1d7208ce-d3a1-4406-9638-fe7051562994=3DSTO= RAGE]', sharedLocks=3D''}' =20 =20 =20 [root@node00 ovirt-engine]# tail -n 1000 engine.log | grep = a81ffa4a-5a58-41a0-888a-f0edc321609b 2018-03-21 10:44:11,025+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (default task-16) [a81ffa4a-5a58-41a0-888a-f0edc321609b] Lock Acquired = to object = 'EngineLock:{exclusiveLocks=3D'[1d7208ce-d3a1-4406-9638-fe7051562994=3DSTO= RAGE]', sharedLocks=3D''}' 2018-03-21 10:44:11,236+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] Running command: = AttachStorageDomainToPoolCommand internal: false. Entities affected : = ID: 1d7208ce-d3a1-4406-9638-fe7051562994 Type: StorageAction group = MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: = 80cc922f-8dea-4fed-b951-1060ba116ad5 Type: StoragePoolAction group = MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2018-03-21 10:46:16,567+02 ERROR = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] Cannot connect storage connection = server, aborting attach storage domain operation. 2018-03-21 10:46:16,568+02 INFO = [org.ovirt.engine.core.bll.CommandCompensator] = (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] Command = [id=3Db5c25100-1a8a-4db0-9509-99cfa60995b1]: Compensating NEW_ENTITY_ID = of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; = snapshot: = StoragePoolIsoMapId:{storagePoolId=3D'80cc922f-8dea-4fed-b951-1060ba116ad5= ', storageId=3D'1d7208ce-d3a1-4406-9638-fe7051562994'}. 2018-03-21 10:46:16,651+02 ERROR = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage = Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: = admin@internal-authz) 2018-03-21 10:46:16,681+02 INFO = [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand= ] (EE-ManagedThreadFactory-engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-f0edc321609b] Lock freed to object = 'EngineLock:{exclusiveLocks=3D'[1d7208ce-d3a1-4406-9638-fe7051562994=3DSTO= RAGE]', sharedLocks=3D'=E2=80=99}' =20 =20 _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users = <http://lists.ovirt.org/mailman/listinfo/users> =20 =20
--Apple-Mail=_C5052CF0-37B9-4927-AE48-DF656F09C5B6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D""><div = class=3D"">Hi,</div><div class=3D""><br class=3D""></div><div = class=3D""><br class=3D""></div><div class=3D"">Errors occurred at 10.43 = AM and 10.46 AM</div><div class=3D"">node00.starlett.lv - 192.168.0.4 - = oVirt host engine (separate PC, not hosted)</div><div = class=3D"">node10.starlett.lv - 192.168.0.5 - host #1 of DC #1, export = domain from which was detached from DC #1</div><div = class=3D"">node11.starlett.lv - 192.168.0.6 - host #1 of DC = #2, </div><div class=3D""><br class=3D""></div>Logs from DC#2 = node11, to which I=E2=80=99m trying to attach export domain located at = NFS share node10.<div class=3D""><br class=3D""></div><div = class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"gmail_extra"><div class=3D"gmail_quote"><blockquote = class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex; = border-left-width: 1px; border-left-style: solid; border-left-color: = rgb(204, 204, 204); padding-left: 1ex;"><div class=3D"" = style=3D"word-wrap: break-word; line-break: after-white-space;"><div = class=3D""><div class=3D""><font color=3D"#aa7942" class=3D""><b = class=3D"">grep 570ec5d9-fff5-4656-afbd-<wbr = class=3D"">90b3207a616e</b></font></div></div></div></blockquote></div></d= iv></blockquote>within vdsm.log returned nothing, so I did</div><div = class=3D""><br class=3D""></div><div class=3D""><br class=3D""></div><div = class=3D""><font color=3D"#942192" class=3D""><b class=3D"">grep -n = 10:43 vdsm.log | tail -1000</b></font><br class=3D"">1011:2018-03-21 = 06:10:43,077+0200 INFO (jsonrpc/3) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46)<br class=3D"">1012:2018-03-21 = 06:10:43,077+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats = return=3D{'status': {'message': 'Done', 'code': 0}, 'statsList': = (suppressed)} from=3D::1,36114 (api:52)<br class=3D"">1013:2018-03-21 = 06:10:43,077+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573)<br = class=3D"">1014:2018-03-21 06:10:43,868+0200 INFO (jsonrpc/7) = [api.host] START getAllVmStats() from=3D::ffff:192.168.0.4,49914 = (api:46)<br class=3D"">1015:2018-03-21 06:10:43,868+0200 INFO = (jsonrpc/7) [api.host] FINISH getAllVmStats return=3D{'status': = {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} = from=3D::ffff:192.168.0.4,49914 (api:52)<br class=3D"">1016:2018-03-21 = 06:10:43,868+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br = class=3D"">19729:2018-03-21 09:10:43,641+0200 INFO (jsonrpc/5) = [api.host] START getAllVmStats() from=3D::1,36114 (api:46)<br = class=3D"">19730:2018-03-21 09:10:43,641+0200 INFO (jsonrpc/5) = [api.host] FINISH getAllVmStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'statsList': (suppressed)} from=3D::1,36114 (api:52)<br = class=3D"">19731:2018-03-21 09:10:43,642+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29302:2018-03-21 10:43:00,085+0200 = INFO (periodic/2) [vdsm.api] START repoStats(domains=3D()) = from=3Dinternal, task_id=3D7b690c8a-5470-44da-a8f3-7e7e9b018e88 = (api:46)<br class=3D"">29303:2018-03-21 10:43:00,085+0200 INFO = (periodic/2) [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000477038', = 'lastCheck': '0.5', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': = '0.4', 'valid': True}} from=3Dinternal, = task_id=3D7b690c8a-5470-44da-a8f3-7e7e9b018e88 (api:52)<br = class=3D"">29304:2018-03-21 10:43:00,086+0200 INFO (periodic/2) = [vdsm.api] START multipath_health() from=3Dinternal, = task_id=3D9a625d43-a03c-429c-856f-9aa8e5ff65b5 (api:46)<br = class=3D"">29305:2018-03-21 10:43:00,086+0200 INFO (periodic/2) = [vdsm.api] FINISH multipath_health return=3D{} from=3Dinternal, = task_id=3D9a625d43-a03c-429c-856f-9aa8e5ff65b5 (api:52)<br = class=3D"">29306:2018-03-21 10:43:05,382+0200 INFO (jsonrpc/5) = [api.host] START getAllVmStats() from=3D::ffff:192.168.0.4,49914 = (api:46)<br class=3D"">29307:2018-03-21 10:43:05,383+0200 INFO = (jsonrpc/5) [api.host] FINISH getAllVmStats return=3D{'status': = {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} = from=3D::ffff:192.168.0.4,49914 (api:52)<br class=3D"">29308:2018-03-21 = 10:43:05,383+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br = class=3D"">29309:2018-03-21 10:43:06,248+0200 INFO (jsonrpc/3) = [api.host] START getStats() from=3D::ffff:192.168.0.4,49914 (api:46)<br = class=3D"">29310:2018-03-21 10:43:06,249+0200 INFO (jsonrpc/3) = [vdsm.api] START repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3Df80f9bb8-2e84-4f45-ab58-09a88e808cf3 (api:46)<br = class=3D"">29311:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000477038', = 'lastCheck': '6.7', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': = '6.6', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3Df80f9bb8-2e84-4f45-ab58-09a88e808cf3 (api:52)<br = class=3D"">29312:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) = [vdsm.api] START multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3Dddd7064d-cd12-43bf-a177-b2a93cc9035e (api:46)<br = class=3D"">29313:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) = [vdsm.api] FINISH multipath_health return=3D{} = from=3D::ffff:192.168.0.4,49914, = task_id=3Dddd7064d-cd12-43bf-a177-b2a93cc9035e (api:52)<br = class=3D"">29314:2018-03-21 10:43:06,256+0200 INFO (jsonrpc/3) = [api.host] FINISH getStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.13', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '10': {'cpuUser': = '0.13', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.47'}, '13': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': = '99.73'}, '12': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', = 'cpuIdle': '99.67'}, '15': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': = '0.47', 'cpuIdle': '99.33'}, '14': {'cpuUser': '0.13', 'nodeIndex': 0, = 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '1': {'cpuUser': '1.13', = 'nodeIndex': 1, 'cpuSys': '1.60', 'cpuIdle': '97.27'}, '0': {'cpuUser': = '0.47', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.26'}, '3': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': = '99.73'}, '2': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.13', = 'cpuIdle': '99.54'}, '5': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.74'}, '4': {'cpuUser': '0.13', 'nodeIndex': 0, = 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '7': {'cpuUser': '0.13', = 'nodeIndex': 1, 'cpuSys': '0.60', 'cpuIdle': '99.27'}, '6': {'cpuUser': = '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '9': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '99.66'}, '8': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', = 'cpuIdle': '99.54'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, = 'memFree': '15142'}, '0': {'memPercent': 5, 'memFree': '15349'}}, = 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, = 'vmCount': 0, 'memUsed': '3', 'storageDomains': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, = 'version': 4, 'acquired': True, 'delay': '0.000477038', 'lastCheck': = '6.7', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.000498252', 'lastCheck': '6.6', 'valid': True}}, = 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'enp3s0f0', = 'tx': '1091287978', 'txDropped': '0', 'rx': '11914868563', 'rxErrors': = '0', 'speed': '100', 'rxDropped': '1439'}, 'ovirtmgmt': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': = 'ovirtmgmt', 'tx': '1048639112', 'txDropped': '0', 'rx': '11615209092', = 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'lo', 'tx': = '58057075945', 'txDropped': '0', 'rx': '58057075945', 'rxErrors': '0', = 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'enp3s0f1', = 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': = '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': = 'down', 'sampleTime': 1521621780.072418, 'name': ';vdsmdummy;', 'tx': = '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621780.072418, 'name': 'enp4s0f0', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621780.072418, 'name': 'enp4s0f1', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', = 'ksmPages': 100, 'elapsedTime': '76345.09', 'cpuLoad': '0.16', 'cpuSys': = '0.36', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': = {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31081, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1439', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:43:06 GMT', 'cpuUser': '0.21', = 'memFree': 31337, 'cpuIdle': '99.43', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.93'}} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29315:2018-03-21 10:43:06,258+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573)<br class=3D"">29316:2018-03-21 10:43:06,632+0200 = INFO (jsonrpc/0) [api.host] START getAllVmStats() from=3D::1,36114 = (api:46)<br class=3D"">29317:2018-03-21 10:43:06,632+0200 INFO = (jsonrpc/0) [api.host] FINISH getAllVmStats return=3D{'status': = {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} = from=3D::1,36114 (api:52)<br class=3D"">29318:2018-03-21 = 10:43:06,633+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br = class=3D"">29319:2018-03-21 10:43:07,415+0200 INFO (jsonrpc/4) = [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D276c13c7-61ec-4782-a0d9-5cd0f0b0a729 (api:46)<br = class=3D"">29320:2018-03-21 10:43:07,420+0200 INFO (jsonrpc/4) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D276c13c7-61ec-4782-a0d9-5cd0f0b0a729 (api:52)<br = class=3D"">29321:2018-03-21 10:43:07,420+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573)<br class=3D"">29322:2018-03-21 = 10:43:07,427+0200 INFO (jsonrpc/6) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D344b18df-a428-429a-b49f-381f901d1618 (api:46)<br = class=3D"">29323:2018-03-21 10:43:07,434+0200 INFO (jsonrpc/6) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D344b18df-a428-429a-b49f-381f901d1618 (api:52)<br = class=3D"">29324:2018-03-21 10:43:07,435+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573)<br class=3D"">29325:2018-03-21 10:43:13,839+0200 = WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker = name=3Djsonrpc/1 running <Task <JsonRpcTask {'params': = {u'connectionParams': [{u'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524', = u'connection': u'node10.starlett.lv:/vmdata/nfs/exports', u'iqn': u'', = u'user': u'', u'tpgt': u'1', u'protocol_version': u'auto', u'password': = '********', u'port': u''}], u'storagepoolID': = u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': = '2.0', 'method': u'StoragePool.connectStorageServer', 'id': = u'5f98dfd5-998f-44a6-8594-e5f40144b0ed'} at 0x4076150> timeout=3D60, = duration=3D120 at 0x402fe50> task#=3D3754 at 0x353e690>, = traceback:<br class=3D"">29376:2018-03-21 10:43:15,104+0200 INFO = (periodic/2) [vdsm.api] START repoStats(domains=3D()) from=3Dinternal, = task_id=3D672a883e-1d60-4981-9eee-bdcac2ec30c1 (api:46)<br = class=3D"">29377:2018-03-21 10:43:15,105+0200 INFO (periodic/2) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000488777', = 'lastCheck': '5.6', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000481962', 'lastCheck': = '5.4', 'valid': True}} from=3Dinternal, = task_id=3D672a883e-1d60-4981-9eee-bdcac2ec30c1 (api:52)<br = class=3D"">29378:2018-03-21 10:43:15,105+0200 INFO (periodic/2) = [vdsm.api] START multipath_health() from=3Dinternal, = task_id=3D55f1b797-da2a-4b07-affe-5b9e41a74307 (api:46)<br = class=3D"">29379:2018-03-21 10:43:15,106+0200 INFO (periodic/2) = [vdsm.api] FINISH multipath_health return=3D{} from=3Dinternal, = task_id=3D55f1b797-da2a-4b07-affe-5b9e41a74307 (api:52)<br = class=3D"">29380:2018-03-21 10:43:17,645+0200 INFO (jsonrpc/7) = [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D52a6ddeb-3c20-4f3f-b27d-94d19580b473 (api:46)<br = class=3D"">29381:2018-03-21 10:43:17,649+0200 INFO (jsonrpc/7) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D52a6ddeb-3c20-4f3f-b27d-94d19580b473 (api:52)<br = class=3D"">29382:2018-03-21 10:43:17,650+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573)<br class=3D"">29383:2018-03-21 = 10:43:17,656+0200 INFO (jsonrpc/2) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D1a103883-6a6a-4d61-ade6-96559591ddb0 (api:46)<br = class=3D"">29384:2018-03-21 10:43:17,663+0200 INFO (jsonrpc/2) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D1a103883-6a6a-4d61-ade6-96559591ddb0 (api:52)<br = class=3D"">29385:2018-03-21 10:43:17,663+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573)<br class=3D"">29386:2018-03-21 10:43:18,882+0200 = ERROR (jsonrpc/1) [storage.HSM] Could not connect to storageServer = (hsm:2407)<br class=3D"">29406:2018-03-21 10:43:18,883+0200 INFO = (jsonrpc/1) [vdsm.api] FINISH connectStorageServer return=3D{'statuslist':= [{'status': 477, 'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524'}]} = from=3D::ffff:192.168.0.4,49914, flow_id=3D4a53b512, = task_id=3De598cbe0-cde8-4c73-b526-4398df05e67f (api:52)<br = class=3D"">29407:2018-03-21 10:43:18,883+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer = succeeded in 125.04 seconds (__init__:573)<br class=3D"">29408:2018-03-21 = 10:43:20,387+0200 INFO (jsonrpc/5) [api.host] START = getAllVmStats() from=3D::ffff:192.168.0.4,49914 (api:46)<br = class=3D"">29409:2018-03-21 10:43:20,388+0200 INFO (jsonrpc/5) = [api.host] FINISH getAllVmStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 = (api:52)<br class=3D"">29410:2018-03-21 10:43:20,388+0200 INFO = (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats = succeeded in 0.01 seconds (__init__:573)<br class=3D"">29411:2018-03-21 = 10:43:21,497+0200 INFO (jsonrpc/3) [api.host] START getStats() = from=3D::ffff:192.168.0.4,49914 (api:46)<br class=3D"">29412:2018-03-21 = 10:43:21,498+0200 INFO (jsonrpc/3) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3Da7469c1f-3d6d-42e7-8798-5c5335da4a9b (api:46)<br = class=3D"">29413:2018-03-21 10:43:21,498+0200 INFO (jsonrpc/3) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000445096', = 'lastCheck': '1.9', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000296586', 'lastCheck': = '1.8', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3Da7469c1f-3d6d-42e7-8798-5c5335da4a9b (api:52)<br = class=3D"">29414:2018-03-21 10:43:21,499+0200 INFO (jsonrpc/3) = [vdsm.api] START multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D9fa5dae3-e2bd-4186-9a1d-afb0aa6e2511 (api:46)<br = class=3D"">29415:2018-03-21 10:43:21,499+0200 INFO (jsonrpc/3) = [vdsm.api] FINISH multipath_health return=3D{} = from=3D::ffff:192.168.0.4,49914, = task_id=3D9fa5dae3-e2bd-4186-9a1d-afb0aa6e2511 (api:52)<br = class=3D"">29416:2018-03-21 10:43:21,505+0200 INFO (jsonrpc/3) = [api.host] FINISH getStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '10': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '13': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.86'}, '12': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', = 'cpuIdle': '99.47'}, '15': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.87'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, = 'cpuSys': '0.33', 'cpuIdle': '99.60'}, '1': {'cpuUser': '1.00', = 'nodeIndex': 1, 'cpuSys': '1.00', 'cpuIdle': '98.00'}, '0': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '3': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.93'}, '2': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.07', = 'cpuIdle': '99.66'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.07', 'cpuIdle': '99.86'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, = 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '7': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '6': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '9': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.80'}, '8': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', = 'cpuIdle': '99.93'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, = 'memFree': '15142'}, '0': {'memPercent': 5, 'memFree': '15351'}}, = 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, = 'vmCount': 0, 'memUsed': '3', 'storageDomains': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, = 'version': 4, 'acquired': True, 'delay': '0.000445096', 'lastCheck': = '1.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.000296586', 'lastCheck': '1.8', 'valid': True}}, = 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'enp3s0f0', = 'tx': '1091295628', 'txDropped': '0', 'rx': '11914875210', 'rxErrors': = '0', 'speed': '100', 'rxDropped': '1442'}, 'ovirtmgmt': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': = 'ovirtmgmt', 'tx': '1048646480', 'txDropped': '0', 'rx': '11615214750', = 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'lo', 'tx': = '58066397668', 'txDropped': '0', 'rx': '58066397668', 'rxErrors': '0', = 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'enp3s0f1', = 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': = '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': = 'down', 'sampleTime': 1521621795.091263, 'name': ';vdsmdummy;', 'tx': = '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621795.091263, 'name': 'enp4s0f0', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621795.091263, 'name': 'enp4s0f1', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', = 'ksmPages': 100, 'elapsedTime': '76360.34', 'cpuLoad': '0.15', 'cpuSys': = '0.18', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': = {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31031, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1442', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:43:21 GMT', 'cpuUser': '0.15', = 'memFree': 31287, 'cpuIdle': '99.67', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.67'}} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29417:2018-03-21 10:43:21,508+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.02 seconds = (__init__:573)<br class=3D"">29418:2018-03-21 10:43:21,651+0200 = INFO (jsonrpc/0) [api.host] START getAllVmStats() from=3D::1,36114 = (api:46)<br class=3D"">29419:2018-03-21 10:43:21,651+0200 INFO = (jsonrpc/0) [api.host] FINISH getAllVmStats return=3D{'status': = {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} = from=3D::1,36114 (api:52)<br class=3D"">29420:2018-03-21 = 10:43:21,652+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br = class=3D"">29421:2018-03-21 10:43:27,764+0200 INFO (jsonrpc/4) = [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D46d19cab-8903-4c52-9bb2-4dd8f370997e (api:46)<br = class=3D"">29422:2018-03-21 10:43:27,769+0200 INFO (jsonrpc/4) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D46d19cab-8903-4c52-9bb2-4dd8f370997e (api:52)<br = class=3D"">29423:2018-03-21 10:43:27,770+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573)<br class=3D"">29424:2018-03-21 = 10:43:27,815+0200 INFO (jsonrpc/6) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D290acc73-8f25-4fd7-9263-7bd05d46430a (api:46)<br = class=3D"">29425:2018-03-21 10:43:27,822+0200 INFO (jsonrpc/6) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D290acc73-8f25-4fd7-9263-7bd05d46430a (api:52)<br = class=3D"">29426:2018-03-21 10:43:27,823+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573)<br class=3D"">29427:2018-03-21 10:43:30,121+0200 = INFO (periodic/3) [vdsm.api] START repoStats(domains=3D()) = from=3Dinternal, task_id=3D0abdebcf-2b0b-4d33-b552-c0f2ed3e567b = (api:46)<br class=3D"">29428:2018-03-21 10:43:30,122+0200 INFO = (periodic/3) [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000402077', = 'lastCheck': '0.6', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': = '0.5', 'valid': True}} from=3Dinternal, = task_id=3D0abdebcf-2b0b-4d33-b552-c0f2ed3e567b (api:52)<br = class=3D"">29429:2018-03-21 10:43:30,122+0200 INFO (periodic/3) = [vdsm.api] START multipath_health() from=3Dinternal, = task_id=3Dacde7e7d-ab4a-4811-934f-4d312da33208 (api:46)<br = class=3D"">29430:2018-03-21 10:43:30,123+0200 INFO (periodic/3) = [vdsm.api] FINISH multipath_health return=3D{} from=3Dinternal, = task_id=3Dacde7e7d-ab4a-4811-934f-4d312da33208 (api:52)<br = class=3D"">29431:2018-03-21 10:43:35,393+0200 INFO (jsonrpc/7) = [api.host] START getAllVmStats() from=3D::ffff:192.168.0.4,49914 = (api:46)<br class=3D"">29432:2018-03-21 10:43:35,394+0200 INFO = (jsonrpc/7) [api.host] FINISH getAllVmStats return=3D{'status': = {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} = from=3D::ffff:192.168.0.4,49914 (api:52)<br class=3D"">29433:2018-03-21 = 10:43:35,394+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br = class=3D"">29434:2018-03-21 10:43:36,671+0200 INFO (jsonrpc/2) = [api.host] START getAllVmStats() from=3D::1,36114 (api:46)<br = class=3D"">29435:2018-03-21 10:43:36,671+0200 INFO (jsonrpc/2) = [api.host] FINISH getAllVmStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'statsList': (suppressed)} from=3D::1,36114 (api:52)<br = class=3D"">29436:2018-03-21 10:43:36,672+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29437:2018-03-21 10:43:37,356+0200 = INFO (jsonrpc/1) [api.host] START getStats() = from=3D::ffff:192.168.0.4,49914 (api:46)<br class=3D"">29438:2018-03-21 = 10:43:37,357+0200 INFO (jsonrpc/1) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3Dbc14e253-d3d4-4cc3-9ae1-c0e6ac6e37f7 (api:46)<br = class=3D"">29439:2018-03-21 10:43:37,357+0200 INFO (jsonrpc/1) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000402077', = 'lastCheck': '7.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': = '7.7', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3Dbc14e253-d3d4-4cc3-9ae1-c0e6ac6e37f7 (api:52)<br = class=3D"">29440:2018-03-21 10:43:37,358+0200 INFO (jsonrpc/1) = [vdsm.api] START multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D23c698b8-d5bd-4746-b738-97b10f3e4bfb (api:46)<br = class=3D"">29441:2018-03-21 10:43:37,358+0200 INFO (jsonrpc/1) = [vdsm.api] FINISH multipath_health return=3D{} = from=3D::ffff:192.168.0.4,49914, = task_id=3D23c698b8-d5bd-4746-b738-97b10f3e4bfb (api:52)<br = class=3D"">29442:2018-03-21 10:43:37,365+0200 INFO (jsonrpc/1) = [api.host] FINISH getStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '10': {'cpuUser': = '3.06', 'nodeIndex': 0, 'cpuSys': '0.80', 'cpuIdle': '96.14'}, '13': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.93'}, '12': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', = 'cpuIdle': '99.73'}, '15': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.87'}, '14': {'cpuUser': '0.27', 'nodeIndex': 0, = 'cpuSys': '0.40', 'cpuIdle': '99.33'}, '1': {'cpuUser': '1.13', = 'nodeIndex': 1, 'cpuSys': '1.33', 'cpuIdle': '97.54'}, '0': {'cpuUser': = '0.47', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.20'}, '3': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.93'}, '2': {'cpuUser': '0.67', 'nodeIndex': 0, 'cpuSys': '0.40', = 'cpuIdle': '98.93'}, '5': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.74'}, '4': {'cpuUser': '0.47', 'nodeIndex': 0, = 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '7': {'cpuUser': '0.33', = 'nodeIndex': 1, 'cpuSys': '0.73', 'cpuIdle': '98.94'}, '6': {'cpuUser': = '5.39', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '94.14'}, '9': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '99.66'}, '8': {'cpuUser': '0.53', 'nodeIndex': 0, 'cpuSys': '0.40', = 'cpuIdle': '99.07'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, = 'memFree': '15143'}, '0': {'memPercent': 5, 'memFree': '15348'}}, = 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, = 'vmCount': 0, 'memUsed': '3', 'storageDomains': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, = 'version': 4, 'acquired': True, 'delay': '0.000402077', 'lastCheck': = '7.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.000344445', 'lastCheck': '7.7', 'valid': True}}, = 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'enp3s0f0', = 'tx': '1091305295', 'txDropped': '0', 'rx': '11914880953', 'rxErrors': = '0', 'speed': '100', 'rxDropped': '1442'}, 'ovirtmgmt': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': = 'ovirtmgmt', 'tx': '1048655897', 'txDropped': '0', 'rx': '11615220341', = 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'lo', 'tx': = '58078891247', 'txDropped': '0', 'rx': '58078891247', 'rxErrors': '0', = 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'enp3s0f1', = 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': = '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': = 'down', 'sampleTime': 1521621810.108511, 'name': ';vdsmdummy;', 'tx': = '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621810.108511, 'name': 'enp4s0f0', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621810.108511, 'name': 'enp4s0f1', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', = 'ksmPages': 100, 'elapsedTime': '76376.19', 'cpuLoad': '0.16', 'cpuSys': = '0.39', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': = {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.00', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1442', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:43:37 GMT', 'cpuUser': '0.79', = 'memFree': 31336, 'cpuIdle': '98.82', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.73'}} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29443:2018-03-21 10:43:37,367+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573)<br class=3D"">29444:2018-03-21 10:43:38,124+0200 = INFO (jsonrpc/5) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D9d2a82dd-c76c-4eb5-a1e1-b49bf004384e (api:46)<br = class=3D"">29445:2018-03-21 10:43:38,128+0200 INFO (jsonrpc/5) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D9d2a82dd-c76c-4eb5-a1e1-b49bf004384e (api:52)<br = class=3D"">29446:2018-03-21 10:43:38,129+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573)<br class=3D"">29447:2018-03-21 = 10:43:38,171+0200 INFO (jsonrpc/3) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D62d44b0b-20b2-47ae-bfb8-d4717e301b69 (api:46)<br = class=3D"">29448:2018-03-21 10:43:38,177+0200 INFO (jsonrpc/3) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D62d44b0b-20b2-47ae-bfb8-d4717e301b69 (api:52)<br = class=3D"">29449:2018-03-21 10:43:38,178+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573)<br class=3D"">29450:2018-03-21 10:43:45,147+0200 = INFO (periodic/2) [vdsm.api] START repoStats(domains=3D()) = from=3Dinternal, task_id=3D49c33285-a222-464d-8984-4e75a5c6354a = (api:46)<br class=3D"">29451:2018-03-21 10:43:45,148+0200 INFO = (periodic/2) [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000454102', = 'lastCheck': '5.6', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000347381', 'lastCheck': = '5.5', 'valid': True}} from=3Dinternal, = task_id=3D49c33285-a222-464d-8984-4e75a5c6354a (api:52)<br = class=3D"">29452:2018-03-21 10:43:45,148+0200 INFO (periodic/2) = [vdsm.api] START multipath_health() from=3Dinternal, = task_id=3Deed6b564-76b1-4f9b-80a2-1f0f6d9bbe32 (api:46)<br = class=3D"">29453:2018-03-21 10:43:45,149+0200 INFO (periodic/2) = [vdsm.api] FINISH multipath_health return=3D{} from=3Dinternal, = task_id=3Deed6b564-76b1-4f9b-80a2-1f0f6d9bbe32 (api:52)<br = class=3D"">29454:2018-03-21 10:43:48,405+0200 INFO (jsonrpc/0) = [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D84651a37-7d13-41a9-9dcf-2169a9c46fc5 (api:46)<br = class=3D"">29455:2018-03-21 10:43:48,410+0200 INFO (jsonrpc/0) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D84651a37-7d13-41a9-9dcf-2169a9c46fc5 (api:52)<br = class=3D"">29456:2018-03-21 10:43:48,411+0200 INFO (jsonrpc/0) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573)<br class=3D"">29457:2018-03-21 = 10:43:48,417+0200 INFO (jsonrpc/4) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D4ebd1d1a-11ac-41ee-b459-7e45f008776b (api:46)<br = class=3D"">29458:2018-03-21 10:43:48,423+0200 INFO (jsonrpc/4) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D4ebd1d1a-11ac-41ee-b459-7e45f008776b (api:52)<br = class=3D"">29459:2018-03-21 10:43:48,424+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573)<br class=3D"">29460:2018-03-21 10:43:50,399+0200 = INFO (jsonrpc/6) [api.host] START getAllVmStats() = from=3D::ffff:192.168.0.4,49914 (api:46)<br class=3D"">29461:2018-03-21 = 10:43:50,400+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats = return=3D{'status': {'message': 'Done', 'code': 0}, 'statsList': = (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29462:2018-03-21 10:43:50,400+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29463:2018-03-21 10:43:51,694+0200 = INFO (jsonrpc/7) [api.host] START getAllVmStats() from=3D::1,36114 = (api:46)<br class=3D"">29464:2018-03-21 10:43:51,694+0200 INFO = (jsonrpc/7) [api.host] FINISH getAllVmStats return=3D{'status': = {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} = from=3D::1,36114 (api:52)<br class=3D"">29465:2018-03-21 = 10:43:51,695+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br = class=3D"">29466:2018-03-21 10:43:52,516+0200 INFO (jsonrpc/2) = [api.host] START getStats() from=3D::ffff:192.168.0.4,49914 (api:46)<br = class=3D"">29467:2018-03-21 10:43:52,517+0200 INFO (jsonrpc/2) = [vdsm.api] START repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D0a88f0dc-eccf-4d24-9ac2-22e079a1480a (api:46)<br = class=3D"">29468:2018-03-21 10:43:52,517+0200 INFO (jsonrpc/2) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000487937', = 'lastCheck': '3.0', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000455956', 'lastCheck': = '2.9', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D0a88f0dc-eccf-4d24-9ac2-22e079a1480a (api:52)<br = class=3D"">29469:2018-03-21 10:43:52,518+0200 INFO (jsonrpc/2) = [vdsm.api] START multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3Da7a44ff7-a137-41b6-9ad5-2c0b32ab79ec (api:46)<br = class=3D"">29470:2018-03-21 10:43:52,518+0200 INFO (jsonrpc/2) = [vdsm.api] FINISH multipath_health return=3D{} = from=3D::ffff:192.168.0.4,49914, = task_id=3Da7a44ff7-a137-41b6-9ad5-2c0b32ab79ec (api:52)<br = class=3D"">29471:2018-03-21 10:43:52,524+0200 INFO (jsonrpc/2) = [api.host] FINISH getStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '10': {'cpuUser': = '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '13': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.93'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', = 'cpuIdle': '99.87'}, '15': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.07', 'cpuIdle': '99.86'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, = 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '1': {'cpuUser': '1.13', = 'nodeIndex': 1, 'cpuSys': '1.13', 'cpuIdle': '97.74'}, '0': {'cpuUser': = '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '3': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': = '99.93'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', = 'cpuIdle': '99.87'}, '5': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.87'}, '4': {'cpuUser': '0.27', 'nodeIndex': 0, = 'cpuSys': '0.13', 'cpuIdle': '99.60'}, '7': {'cpuUser': '0.00', = 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '99.67'}, '6': {'cpuUser': = '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '9': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.80'}, '8': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', = 'cpuIdle': '99.80'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, = 'memFree': '15143'}, '0': {'memPercent': 5, 'memFree': '15349'}}, = 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, = 'vmCount': 0, 'memUsed': '3', 'storageDomains': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, = 'version': 4, 'acquired': True, 'delay': '0.000487937', 'lastCheck': = '3.0', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.000455956', 'lastCheck': '2.9', 'valid': True}}, = 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'enp3s0f0', 'tx': = '1091312739', 'txDropped': '0', 'rx': '11914899051', 'rxErrors': '0', = 'speed': '100', 'rxDropped': '1443'}, 'ovirtmgmt': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'ovirtmgmt', = 'tx': '1048663107', 'txDropped': '0', 'rx': '11615236520', 'rxErrors': = '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'lo', 'tx': = '58090295494', 'txDropped': '0', 'rx': '58090295494', 'rxErrors': '0', = 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'enp3s0f1', 'tx': = '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621825.13588, 'name': ';vdsmdummy;', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621825.13588, 'name': 'enp4s0f0', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621825.13588, 'name': 'enp4s0f1', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', = 'ksmPages': 100, 'elapsedTime': '76391.35', 'cpuLoad': '0.15', 'cpuSys': = '0.20', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': = {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.13', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1443', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:43:52 GMT', 'cpuUser': '0.14', = 'memFree': 31336, 'cpuIdle': '99.66', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.73'}} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29472:2018-03-21 10:43:52,526+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573)<br class=3D"">29473:2018-03-21 10:43:58,686+0200 = INFO (jsonrpc/1) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3Dd6d29cf6-c083-4639-91a2-f062eae4e629 (api:46)<br = class=3D"">29474:2018-03-21 10:43:58,690+0200 INFO (jsonrpc/1) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3Dd6d29cf6-c083-4639-91a2-f062eae4e629 (api:52)<br = class=3D"">29475:2018-03-21 10:43:58,690+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573)<br class=3D"">29476:2018-03-21 = 10:43:58,697+0200 INFO (jsonrpc/5) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D9dbd91df-dd9c-469c-8308-bfbfc228937c (api:46)<br = class=3D"">29477:2018-03-21 10:43:58,703+0200 INFO (jsonrpc/5) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D9dbd91df-dd9c-469c-8308-bfbfc228937c (api:52)<br = class=3D"">29478:2018-03-21 10:43:58,704+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573)<br class=3D""><br class=3D""><br class=3D""><br = class=3D""><b class=3D""><font color=3D"#942192" class=3D"">grep -n = 10:46 vdsm.log | tail -1000</font></b><br class=3D"">1017:2018-03-21 = 06:10:46,882+0200 INFO (jsonrpc/2) [api.host] START getStats() = from=3D::ffff:192.168.0.4,49914 (api:46)<br class=3D"">1018:2018-03-21 = 06:10:46,883+0200 INFO (jsonrpc/2) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D6e045b00-88b2-469b-80d6-af38303f8c32 (api:46)<br = class=3D"">1019:2018-03-21 06:10:46,883+0200 INFO (jsonrpc/2) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000332785', = 'lastCheck': '7.3', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000203914', 'lastCheck': = '5.6', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D6e045b00-88b2-469b-80d6-af38303f8c32 (api:52)<br = class=3D"">1020:2018-03-21 06:10:46,884+0200 INFO (jsonrpc/2) = [vdsm.api] START multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D475a5b40-4a43-42ee-9524-2bdd220aed83 (api:46)<br = class=3D"">1021:2018-03-21 06:10:46,884+0200 INFO (jsonrpc/2) = [vdsm.api] FINISH multipath_health return=3D{} = from=3D::ffff:192.168.0.4,49914, = task_id=3D475a5b40-4a43-42ee-9524-2bdd220aed83 (api:52)<br = class=3D"">1022:2018-03-21 06:10:46,889+0200 INFO (jsonrpc/2) = [api.host] FINISH getStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '10': {'cpuUser': = '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '13': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '99.66'}, '12': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', = 'cpuIdle': '99.47'}, '15': {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': = '0.27', 'cpuIdle': '99.40'}, '14': {'cpuUser': '0.47', 'nodeIndex': 0, = 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '1': {'cpuUser': '3.00', = 'nodeIndex': 1, 'cpuSys': '1.40', 'cpuIdle': '95.60'}, '0': {'cpuUser': = '0.73', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.07'}, '3': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.86'}, '2': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.27', = 'cpuIdle': '99.66'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.20', 'cpuIdle': '99.73'}, '4': {'cpuUser': '0.13', 'nodeIndex': 0, = 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '7': {'cpuUser': '0.80', = 'nodeIndex': 1, 'cpuSys': '0.80', 'cpuIdle': '98.40'}, '6': {'cpuUser': = '5.46', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '94.27'}, '9': = {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '99.53'}, '8': {'cpuUser': '1.27', 'nodeIndex': 0, 'cpuSys': '0.47', = 'cpuIdle': '98.26'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, = 'memFree': '15128'}, '0': {'memPercent': 5, 'memFree': '15366'}}, = 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, = 'vmCount': 0, 'memUsed': '3', 'storageDomains': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, = 'version': 4, 'acquired': True, 'delay': '0.000332785', 'lastCheck': = '7.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.000203914', 'lastCheck': '5.6', 'valid': True}}, = 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'enp3s0f0', = 'tx': '1082264809', 'txDropped': '0', 'rx': '11907188594', 'rxErrors': = '0', 'speed': '100', 'rxDropped': '289'}, 'ovirtmgmt': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'ovirtmgmt', = 'tx': '1039869554', 'txDropped': '0', 'rx': '11608697383', 'rxErrors': = '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'lo', 'tx': = '45587320171', 'txDropped': '0', 'rx': '45587320171', 'rxErrors': '0', = 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'enp3s0f1', = 'tx': '0', 'txDropped': '0', 'rx': '53620', 'rxErrors': '0', 'speed': = '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': = 'down', 'sampleTime': 1521605437.203776, 'name': ';vdsmdummy;', 'tx': = '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521605437.203776, 'name': 'enp4s0f0', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521605437.203776, 'name': 'enp4s0f1', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', = 'ksmPages': 100, 'elapsedTime': '60005.72', 'cpuLoad': '0.06', 'cpuSys': = '0.38', 'diskStats': {'/var/log': {'free': '7357'}, '/var/run/vdsm/': = {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31084, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '289', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T04:10:46 GMT', 'cpuUser': '0.83', = 'memFree': 31340, 'cpuIdle': '98.79', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.80'}} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">1023:2018-03-21 06:10:46,891+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573)<br class=3D"">13522:2018-03-21 08:10:46,827+0200 = INFO (jsonrpc/7) [api.host] START getAllVmStats() = from=3D::ffff:192.168.0.4,49914 (api:46)<br class=3D"">13523:2018-03-21 = 08:10:46,828+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats = return=3D{'status': {'message': 'Done', 'code': 0}, 'statsList': = (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">13524:2018-03-21 08:10:46,828+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29734:2018-03-21 10:46:00,338+0200 = INFO (periodic/2) [vdsm.api] START repoStats(domains=3D()) = from=3Dinternal, task_id=3Daf099ce6-0f44-461b-9e8e-99de63b3884f = (api:46)<br class=3D"">29735:2018-03-21 10:46:00,339+0200 INFO = (periodic/2) [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000470245', = 'lastCheck': '0.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000476935', 'lastCheck': = '0.7', 'valid': True}} from=3Dinternal, = task_id=3Daf099ce6-0f44-461b-9e8e-99de63b3884f (api:52)<br = class=3D"">29736:2018-03-21 10:46:00,339+0200 INFO (periodic/2) = [vdsm.api] START multipath_health() from=3Dinternal, = task_id=3D4a5709d9-e322-44eb-a8a4-bbe3f5e5e1bd (api:46)<br = class=3D"">29737:2018-03-21 10:46:00,340+0200 INFO (periodic/2) = [vdsm.api] FINISH multipath_health return=3D{} from=3Dinternal, = task_id=3D4a5709d9-e322-44eb-a8a4-bbe3f5e5e1bd (api:52)<br = class=3D"">29738:2018-03-21 10:46:04,637+0200 INFO (jsonrpc/7) = [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D5b9300f7-f8c4-4f63-a49c-45ca8119d3b9 (api:46)<br = class=3D"">29739:2018-03-21 10:46:04,642+0200 INFO (jsonrpc/7) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D5b9300f7-f8c4-4f63-a49c-45ca8119d3b9 (api:52)<br = class=3D"">29740:2018-03-21 10:46:04,642+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573)<br class=3D"">29741:2018-03-21 = 10:46:04,648+0200 INFO (jsonrpc/1) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3Df4db5bfa-e131-4f9a-baaa-e7555c996ab0 (api:46)<br = class=3D"">29742:2018-03-21 10:46:04,655+0200 INFO (jsonrpc/1) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3Df4db5bfa-e131-4f9a-baaa-e7555c996ab0 (api:52)<br = class=3D"">29743:2018-03-21 10:46:04,656+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29744:2018-03-21 10:46:05,451+0200 = INFO (jsonrpc/5) [api.host] START getAllVmStats() = from=3D::ffff:192.168.0.4,49914 (api:46)<br class=3D"">29745:2018-03-21 = 10:46:05,452+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats = return=3D{'status': {'message': 'Done', 'code': 0}, 'statsList': = (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29746:2018-03-21 10:46:05,452+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29747:2018-03-21 10:46:06,903+0200 = INFO (jsonrpc/3) [api.host] START getAllVmStats() from=3D::1,36114 = (api:46)<br class=3D"">29748:2018-03-21 10:46:06,904+0200 INFO = (jsonrpc/3) [api.host] FINISH getAllVmStats return=3D{'status': = {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} = from=3D::1,36114 (api:52)<br class=3D"">29749:2018-03-21 = 10:46:06,904+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br = class=3D"">29750:2018-03-21 10:46:08,791+0200 WARN = (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=3Djsonrpc/6 = running <Task <JsonRpcTask {'params': {u'connectionParams': = [{u'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524', u'connection': = u'node10.starlett.lv:/vmdata/nfs/exports', u'iqn': u'', u'user': u'', = u'tpgt': u'1', u'protocol_version': u'auto', u'password': '********', = u'port': u''}], u'storagepoolID': = u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': = '2.0', 'method': u'StoragePool.connectStorageServer', 'id': = u'71143eef-aac6-4996-9783-3e0e3da180c3'} at 0x3604fd0> timeout=3D60, = duration=3D120 at 0x3604e10> task#=3D3762 at 0x3541650>, = traceback:<br class=3D"">29801:2018-03-21 10:46:12,852+0200 INFO = (jsonrpc/0) [api.host] START getStats() from=3D::ffff:192.168.0.4,49914 = (api:46)<br class=3D"">29802:2018-03-21 10:46:12,853+0200 INFO = (jsonrpc/0) [vdsm.api] START repoStats(domains=3D()) = from=3D::ffff:192.168.0.4,49914, = task_id=3D391fdab9-d3c2-4075-9c34-177a07a21ec3 (api:46)<br = class=3D"">29803:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000492277', = 'lastCheck': '3.3', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': = '3.2', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D391fdab9-d3c2-4075-9c34-177a07a21ec3 (api:52)<br = class=3D"">29804:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) = [vdsm.api] START multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D7e53adbc-3974-4f51-bfc8-8fbd3ca6b749 (api:46)<br = class=3D"">29805:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) = [vdsm.api] FINISH multipath_health return=3D{} = from=3D::ffff:192.168.0.4,49914, = task_id=3D7e53adbc-3974-4f51-bfc8-8fbd3ca6b749 (api:52)<br = class=3D"">29806:2018-03-21 10:46:12,861+0200 INFO (jsonrpc/0) = [api.host] FINISH getStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.47', = 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '10': {'cpuUser': = '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '13': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': = '99.73'}, '12': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.27', = 'cpuIdle': '99.53'}, '15': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': = '0.27', 'cpuIdle': '99.60'}, '14': {'cpuUser': '0.13', 'nodeIndex': 0, = 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '1': {'cpuUser': '1.00', = 'nodeIndex': 1, 'cpuSys': '1.33', 'cpuIdle': '97.67'}, '0': {'cpuUser': = '0.13', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '3': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': = '99.93'}, '2': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.33', = 'cpuIdle': '99.47'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.80'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, = 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '7': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.53'}, '6': {'cpuUser': = '0.47', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '9': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': = '99.73'}, '8': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', = 'cpuIdle': '99.54'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, = 'memFree': '15142'}, '0': {'memPercent': 5, 'memFree': '15348'}}, = 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, = 'vmCount': 0, 'memUsed': '3', 'storageDomains': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, = 'version': 4, 'acquired': True, 'delay': '0.000492277', 'lastCheck': = '3.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.000479153', 'lastCheck': '3.2', 'valid': True}}, = 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'enp3s0f0', = 'tx': '1091387790', 'txDropped': '0', 'rx': '11914959714', 'rxErrors': = '0', 'speed': '100', 'rxDropped': '1451'}, 'ovirtmgmt': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': = 'ovirtmgmt', 'tx': '1048735814', 'txDropped': '0', 'rx': '11615289173', = 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'lo', 'tx': = '58192068637', 'txDropped': '0', 'rx': '58192068637', 'rxErrors': '0', = 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'enp3s0f1', = 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': = '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': = 'down', 'sampleTime': 1521621960.326184, 'name': ';vdsmdummy;', 'tx': = '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621960.326184, 'name': 'enp4s0f0', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621960.326184, 'name': 'enp4s0f1', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', = 'ksmPages': 100, 'elapsedTime': '76531.69', 'cpuLoad': '0.17', 'cpuSys': = '0.32', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': = {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.00', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31081, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1451', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:46:12 GMT', 'cpuUser': '0.21', = 'memFree': 31337, 'cpuIdle': '99.46', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.80'}} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29807:2018-03-21 10:46:12,863+0200 INFO (jsonrpc/0) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573)<br class=3D"">29808:2018-03-21 10:46:13,829+0200 ERROR = (jsonrpc/6) [storage.HSM] Could not connect to storageServer = (hsm:2407)<br class=3D"">29828:2018-03-21 10:46:13,829+0200 INFO = (jsonrpc/6) [vdsm.api] FINISH connectStorageServer return=3D{'statuslist':= [{'status': 477, 'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524'}]} = from=3D::ffff:192.168.0.4,49914, flow_id=3D1435fc81, = task_id=3D0a828d2c-d9f4-4f83-a9e9-7393159d5323 (api:52)<br = class=3D"">29829:2018-03-21 10:46:13,830+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer = succeeded in 125.04 seconds (__init__:573)<br class=3D"">29830:2018-03-21 = 10:46:14,767+0200 INFO (jsonrpc/4) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D49cd2ae6-7943-47c9-ad86-9e0b7e58bca3 (api:46)<br = class=3D"">29831:2018-03-21 10:46:14,772+0200 INFO (jsonrpc/4) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D49cd2ae6-7943-47c9-ad86-9e0b7e58bca3 (api:52)<br = class=3D"">29832:2018-03-21 10:46:14,773+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.01 seconds (__init__:573)<br class=3D"">29833:2018-03-21 = 10:46:14,815+0200 INFO (jsonrpc/2) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D19909616-efb4-490d-982c-66eda9ca4381 (api:46)<br = class=3D"">29834:2018-03-21 10:46:14,822+0200 INFO (jsonrpc/2) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D19909616-efb4-490d-982c-66eda9ca4381 (api:52)<br = class=3D"">29835:2018-03-21 10:46:14,823+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573)<br class=3D"">29836:2018-03-21 10:46:15,360+0200 = INFO (periodic/2) [vdsm.api] START repoStats(domains=3D()) = from=3Dinternal, task_id=3Dda74ea9f-164b-450d-b836-6818caa3fdc5 = (api:46)<br class=3D"">29837:2018-03-21 10:46:15,360+0200 INFO = (periodic/2) [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000492277', = 'lastCheck': '5.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': = '5.7', 'valid': True}} from=3Dinternal, = task_id=3Dda74ea9f-164b-450d-b836-6818caa3fdc5 (api:52)<br = class=3D"">29838:2018-03-21 10:46:15,361+0200 INFO (periodic/2) = [vdsm.api] START multipath_health() from=3Dinternal, = task_id=3D06245e2b-e8ca-41c4-90bf-4294d0c699b8 (api:46)<br = class=3D"">29839:2018-03-21 10:46:15,361+0200 INFO (periodic/2) = [vdsm.api] FINISH multipath_health return=3D{} from=3Dinternal, = task_id=3D06245e2b-e8ca-41c4-90bf-4294d0c699b8 (api:52)<br = class=3D"">29840:2018-03-21 10:46:20,456+0200 INFO (jsonrpc/7) = [api.host] START getAllVmStats() from=3D::ffff:192.168.0.4,49914 = (api:46)<br class=3D"">29841:2018-03-21 10:46:20,457+0200 INFO = (jsonrpc/7) [api.host] FINISH getAllVmStats return=3D{'status': = {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} = from=3D::ffff:192.168.0.4,49914 (api:52)<br class=3D"">29842:2018-03-21 = 10:46:20,457+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573)<br = class=3D"">29843:2018-03-21 10:46:21,925+0200 INFO (jsonrpc/1) = [api.host] START getAllVmStats() from=3D::1,36114 (api:46)<br = class=3D"">29844:2018-03-21 10:46:21,926+0200 INFO (jsonrpc/1) = [api.host] FINISH getAllVmStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'statsList': (suppressed)} from=3D::1,36114 (api:52)<br = class=3D"">29845:2018-03-21 10:46:21,926+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29846:2018-03-21 10:46:24,930+0200 = INFO (jsonrpc/5) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3Dd964535d-e5eb-42bc-b324-432a05c364da (api:46)<br = class=3D"">29847:2018-03-21 10:46:24,935+0200 INFO (jsonrpc/5) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3Dd964535d-e5eb-42bc-b324-432a05c364da (api:52)<br = class=3D"">29848:2018-03-21 10:46:24,935+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.00 seconds (__init__:573)<br class=3D"">29849:2018-03-21 = 10:46:24,987+0200 INFO (jsonrpc/3) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D167dcd23-04b5-4de3-9c81-7d25ba56407a (api:46)<br = class=3D"">29850:2018-03-21 10:46:25,022+0200 INFO (jsonrpc/3) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D167dcd23-04b5-4de3-9c81-7d25ba56407a (api:52)<br = class=3D"">29851:2018-03-21 10:46:25,023+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.03 = seconds (__init__:573)<br class=3D"">29852:2018-03-21 10:46:28,010+0200 = INFO (jsonrpc/0) [api.host] START getStats() = from=3D::ffff:192.168.0.4,49914 (api:46)<br class=3D"">29853:2018-03-21 = 10:46:28,011+0200 INFO (jsonrpc/0) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D011616ce-53c9-4cfc-8b29-247b1be03409 (api:46)<br = class=3D"">29854:2018-03-21 10:46:28,011+0200 INFO (jsonrpc/0) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000218955', = 'lastCheck': '8.4', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000351741', 'lastCheck': = '8.4', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D011616ce-53c9-4cfc-8b29-247b1be03409 (api:52)<br = class=3D"">29855:2018-03-21 10:46:28,012+0200 INFO (jsonrpc/0) = [vdsm.api] START multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D85b42a8d-1bf1-495d-ad32-a1f9710a6468 (api:46)<br = class=3D"">29856:2018-03-21 10:46:28,012+0200 INFO (jsonrpc/0) = [vdsm.api] FINISH multipath_health return=3D{} = from=3D::ffff:192.168.0.4,49914, = task_id=3D85b42a8d-1bf1-495d-ad32-a1f9710a6468 (api:52)<br = class=3D"">29857:2018-03-21 10:46:28,018+0200 INFO (jsonrpc/0) = [api.host] FINISH getStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.13', = 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '10': {'cpuUser': = '1.73', 'nodeIndex': 0, 'cpuSys': '0.53', 'cpuIdle': '97.74'}, '13': = {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '99.53'}, '12': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', = 'cpuIdle': '99.73'}, '15': {'cpuUser': '0.67', 'nodeIndex': 1, 'cpuSys': = '0.27', 'cpuIdle': '99.06'}, '14': {'cpuUser': '0.20', 'nodeIndex': 0, = 'cpuSys': '0.33', 'cpuIdle': '99.47'}, '1': {'cpuUser': '1.26', = 'nodeIndex': 1, 'cpuSys': '1.33', 'cpuIdle': '97.41'}, '0': {'cpuUser': = '0.40', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.53'}, '3': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.80'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.27', = 'cpuIdle': '99.73'}, '5': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.87'}, '4': {'cpuUser': '13.78', 'nodeIndex': 0, = 'cpuSys': '0.80', 'cpuIdle': '85.42'}, '7': {'cpuUser': '0.47', = 'nodeIndex': 1, 'cpuSys': '0.67', 'cpuIdle': '98.86'}, '6': {'cpuUser': = '1.07', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '98.66'}, '9': = {'cpuUser': '1.20', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '98.53'}, '8': {'cpuUser': '0.80', 'nodeIndex': 0, 'cpuSys': '0.33', = 'cpuIdle': '98.87'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, = 'memFree': '15136'}, '0': {'memPercent': 5, 'memFree': '15357'}}, = 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, = 'vmCount': 0, 'memUsed': '3', 'storageDomains': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, = 'version': 4, 'acquired': True, 'delay': '0.000218955', 'lastCheck': = '8.4', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.000351741', 'lastCheck': '8.4', 'valid': True}}, = 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'enp3s0f0', = 'tx': '1091397599', 'txDropped': '0', 'rx': '11914966591', 'rxErrors': = '0', 'speed': '100', 'rxDropped': '1454'}, 'ovirtmgmt': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': = 'ovirtmgmt', 'tx': '1048745347', 'txDropped': '0', 'rx': '11615294897', = 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'lo', 'tx': = '58205583384', 'txDropped': '0', 'rx': '58205583384', 'rxErrors': '0', = 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'enp3s0f1', = 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': = '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': = 'down', 'sampleTime': 1521621975.346633, 'name': ';vdsmdummy;', 'tx': = '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621975.346633, 'name': 'enp4s0f0', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621975.346633, 'name': 'enp4s0f1', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', = 'ksmPages': 100, 'elapsedTime': '76546.85', 'cpuLoad': '0.18', 'cpuSys': = '0.39', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': = {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.20', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31031, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1454', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:46:28 GMT', 'cpuUser': '1.38', = 'memFree': 31287, 'cpuIdle': '98.23', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.87'}} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29858:2018-03-21 10:46:28,020+0200 INFO (jsonrpc/0) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573)<br class=3D"">29859:2018-03-21 10:46:29,667+0200 = INFO (itmap/0) [IOProcessClient] Starting client ioprocess-575 = (__init__:308)<br class=3D"">29860:2018-03-21 10:46:29,700+0200 = INFO (itmap/1) [IOProcessClient] Starting client ioprocess-576 = (__init__:308)<br class=3D"">29861:2018-03-21 10:46:29,714+0200 = INFO (ioprocess/1711202) [IOProcess] Starting ioprocess = (__init__:437)<br class=3D"">29862:2018-03-21 10:46:29,725+0200 = INFO (ioprocess/1711208) [IOProcess] Starting ioprocess = (__init__:437)<br class=3D"">29863:2018-03-21 10:46:30,375+0200 = INFO (periodic/0) [vdsm.api] START repoStats(domains=3D()) = from=3Dinternal, task_id=3Dbdacf1c0-3fcb-43b1-82e2-a041a26d0d7f = (api:46)<br class=3D"">29864:2018-03-21 10:46:30,376+0200 INFO = (periodic/0) [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.00031111', = 'lastCheck': '0.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.00045594', 'lastCheck': = '0.6', 'valid': True}} from=3Dinternal, = task_id=3Dbdacf1c0-3fcb-43b1-82e2-a041a26d0d7f (api:52)<br = class=3D"">29865:2018-03-21 10:46:30,376+0200 INFO (periodic/0) = [vdsm.api] START multipath_health() from=3Dinternal, = task_id=3D3d3cad70-e5c8-49b2-9f5d-92a249cc102d (api:46)<br = class=3D"">29866:2018-03-21 10:46:30,377+0200 INFO (periodic/0) = [vdsm.api] FINISH multipath_health return=3D{} from=3Dinternal, = task_id=3D3d3cad70-e5c8-49b2-9f5d-92a249cc102d (api:52)<br = class=3D"">29867:2018-03-21 10:46:35,108+0200 INFO (jsonrpc/6) = [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3D304d7e7c-03fc-44d6-be32-d083ada09b30 (api:46)<br = class=3D"">29868:2018-03-21 10:46:35,114+0200 INFO (jsonrpc/6) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3D304d7e7c-03fc-44d6-be32-d083ada09b30 (api:52)<br = class=3D"">29869:2018-03-21 10:46:35,114+0200 INFO (jsonrpc/6) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.00 seconds (__init__:573)<br class=3D"">29870:2018-03-21 = 10:46:35,170+0200 INFO (jsonrpc/4) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D76736a93-d4c4-4b2f-9848-7b8731fe2b67 (api:46)<br = class=3D"">29871:2018-03-21 10:46:35,176+0200 INFO (jsonrpc/4) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D76736a93-d4c4-4b2f-9848-7b8731fe2b67 (api:52)<br = class=3D"">29872:2018-03-21 10:46:35,177+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 = seconds (__init__:573)<br class=3D"">29873:2018-03-21 10:46:35,462+0200 = INFO (jsonrpc/2) [api.host] START getAllVmStats() = from=3D::ffff:192.168.0.4,49914 (api:46)<br class=3D"">29874:2018-03-21 = 10:46:35,462+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats = return=3D{'status': {'message': 'Done', 'code': 0}, 'statsList': = (suppressed)} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29875:2018-03-21 10:46:35,463+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29876:2018-03-21 10:46:36,948+0200 = INFO (jsonrpc/7) [api.host] START getAllVmStats() from=3D::1,36114 = (api:46)<br class=3D"">29877:2018-03-21 10:46:36,948+0200 INFO = (jsonrpc/7) [api.host] FINISH getAllVmStats return=3D{'status': = {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} = from=3D::1,36114 (api:52)<br class=3D"">29878:2018-03-21 = 10:46:36,949+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br = class=3D"">29879:2018-03-21 10:46:43,179+0200 INFO (jsonrpc/1) = [api.host] START getStats() from=3D::ffff:192.168.0.4,49914 (api:46)<br = class=3D"">29880:2018-03-21 10:46:43,180+0200 INFO (jsonrpc/1) = [vdsm.api] START repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D83bbc95c-6d59-4eec-82b1-e8993be28759 (api:46)<br = class=3D"">29881:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000486101', = 'lastCheck': '3.6', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': = '3.4', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D83bbc95c-6d59-4eec-82b1-e8993be28759 (api:52)<br = class=3D"">29882:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) = [vdsm.api] START multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D9b781324-b50b-49a9-bf31-5eca835dc75c (api:46)<br = class=3D"">29883:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) = [vdsm.api] FINISH multipath_health return=3D{} = from=3D::ffff:192.168.0.4,49914, = task_id=3D9b781324-b50b-49a9-bf31-5eca835dc75c (api:52)<br = class=3D"">29884:2018-03-21 10:46:43,188+0200 INFO (jsonrpc/1) = [api.host] FINISH getStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.40', = 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.33'}, '10': {'cpuUser': = '0.60', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '98.93'}, '13': = {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': = '99.73'}, '12': {'cpuUser': '1.46', 'nodeIndex': 0, 'cpuSys': '0.60', = 'cpuIdle': '97.94'}, '15': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': = '0.33', 'cpuIdle': '99.47'}, '14': {'cpuUser': '0.40', 'nodeIndex': 0, = 'cpuSys': '0.47', 'cpuIdle': '99.13'}, '1': {'cpuUser': '1.33', = 'nodeIndex': 1, 'cpuSys': '1.93', 'cpuIdle': '96.74'}, '0': {'cpuUser': = '0.60', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.13'}, '3': = {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': = '99.40'}, '2': {'cpuUser': '6.99', 'nodeIndex': 0, 'cpuSys': '0.40', = 'cpuIdle': '92.61'}, '5': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': = '0.20', 'cpuIdle': '99.67'}, '4': {'cpuUser': '0.27', 'nodeIndex': 0, = 'cpuSys': '0.27', 'cpuIdle': '99.46'}, '7': {'cpuUser': '0.47', = 'nodeIndex': 1, 'cpuSys': '0.60', 'cpuIdle': '98.93'}, '6': {'cpuUser': = '0.53', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.20'}, '9': = {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': = '99.20'}, '8': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', = 'cpuIdle': '99.80'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, = 'memFree': '15131'}, '0': {'memPercent': 5, 'memFree': '15357'}}, = 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, = 'vmCount': 0, 'memUsed': '3', 'storageDomains': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, = 'version': 4, 'acquired': True, 'delay': '0.000486101', 'lastCheck': = '3.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.00043444', 'lastCheck': '3.4', 'valid': True}}, = 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'enp3s0f0', = 'tx': '1091404888', 'txDropped': '0', 'rx': '11914970667', 'rxErrors': = '0', 'speed': '100', 'rxDropped': '1455'}, 'ovirtmgmt': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': = 'ovirtmgmt', 'tx': '1048752406', 'txDropped': '0', 'rx': '11615298347', = 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'lo', 'tx': = '58215969455', 'txDropped': '0', 'rx': '58215969455', 'rxErrors': '0', = 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'enp3s0f1', = 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': = '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': = 'down', 'sampleTime': 1521621990.363726, 'name': ';vdsmdummy;', 'tx': = '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621990.363726, 'name': 'enp4s0f0', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521621990.363726, 'name': 'enp4s0f1', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '206', = 'ksmPages': 100, 'elapsedTime': '76562.02', 'cpuLoad': '0.18', 'cpuSys': = '0.45', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': = {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.27', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1455', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:46:43 GMT', 'cpuUser': '0.87', = 'memFree': 31336, 'cpuIdle': '98.68', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '1.20'}} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29885:2018-03-21 10:46:43,190+0200 INFO (jsonrpc/1) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573)<br class=3D"">29886:2018-03-21 10:46:45,288+0200 = INFO (jsonrpc/5) [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3Dffe57349-2532-4b64-b32c-8a558273d8e5 (api:46)<br = class=3D"">29887:2018-03-21 10:46:45,292+0200 INFO (jsonrpc/5) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3Dffe57349-2532-4b64-b32c-8a558273d8e5 (api:52)<br = class=3D"">29888:2018-03-21 10:46:45,293+0200 INFO (jsonrpc/5) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.00 seconds (__init__:573)<br class=3D"">29889:2018-03-21 = 10:46:45,298+0200 INFO (jsonrpc/3) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3Db2ea28f0-abf0-4258-8152-6035d93aa5dc (api:46)<br = class=3D"">29890:2018-03-21 10:46:45,305+0200 INFO (jsonrpc/3) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3Db2ea28f0-abf0-4258-8152-6035d93aa5dc (api:52)<br = class=3D"">29891:2018-03-21 10:46:45,306+0200 INFO (jsonrpc/3) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29892:2018-03-21 10:46:45,395+0200 = INFO (periodic/2) [vdsm.api] START repoStats(domains=3D()) = from=3Dinternal, task_id=3D4010bf72-5c86-4daf-a8f9-12ec229a86e4 = (api:46)<br class=3D"">29893:2018-03-21 10:46:45,395+0200 INFO = (periodic/2) [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000486101', = 'lastCheck': '5.8', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': = '5.7', 'valid': True}} from=3Dinternal, = task_id=3D4010bf72-5c86-4daf-a8f9-12ec229a86e4 (api:52)<br = class=3D"">29894:2018-03-21 10:46:45,396+0200 INFO (periodic/2) = [vdsm.api] START multipath_health() from=3Dinternal, = task_id=3D37a1132f-452b-4805-9cc0-b8e74e8d6b02 (api:46)<br = class=3D"">29895:2018-03-21 10:46:45,396+0200 INFO (periodic/2) = [vdsm.api] FINISH multipath_health return=3D{} from=3Dinternal, = task_id=3D37a1132f-452b-4805-9cc0-b8e74e8d6b02 (api:52)<br = class=3D"">29896:2018-03-21 10:46:50,467+0200 INFO (jsonrpc/0) = [api.host] START getAllVmStats() from=3D::ffff:192.168.0.4,49914 = (api:46)<br class=3D"">29897:2018-03-21 10:46:50,468+0200 INFO = (jsonrpc/0) [throttled] Current getAllVmStats: {} (throttledlog:103)<br = class=3D"">29898:2018-03-21 10:46:50,468+0200 INFO (jsonrpc/0) = [api.host] FINISH getAllVmStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'statsList': (suppressed)} from=3D::ffff:192.168.0.4,49914 = (api:52)<br class=3D"">29899:2018-03-21 10:46:50,468+0200 INFO = (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats = succeeded in 0.01 seconds (__init__:573)<br class=3D"">29900:2018-03-21 = 10:46:51,967+0200 INFO (jsonrpc/6) [api.host] START = getAllVmStats() from=3D::1,36114 (api:46)<br class=3D"">29901:2018-03-21 = 10:46:51,968+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats = return=3D{'status': {'message': 'Done', 'code': 0}, 'statsList': = (suppressed)} from=3D::1,36114 (api:52)<br class=3D"">29902:2018-03-21 = 10:46:51,968+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br = class=3D"">29903:2018-03-21 10:46:55,648+0200 INFO (jsonrpc/4) = [vdsm.api] START = getSpmStatus(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49914, = task_id=3Da975635f-59d7-4f40-b9a7-a5ea7150368e (api:46)<br = class=3D"">29904:2018-03-21 10:46:55,653+0200 INFO (jsonrpc/4) = [vdsm.api] FINISH getSpmStatus return=3D{'spm_st': {'spmId': 1, = 'spmStatus': 'SPM', 'spmLver': 6L}} from=3D::ffff:192.168.0.4,49914, = task_id=3Da975635f-59d7-4f40-b9a7-a5ea7150368e (api:52)<br = class=3D"">29905:2018-03-21 10:46:55,653+0200 INFO (jsonrpc/4) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in = 0.00 seconds (__init__:573)<br class=3D"">29906:2018-03-21 = 10:46:55,659+0200 INFO (jsonrpc/2) [vdsm.api] START = getStoragePoolInfo(spUUID=3Du'80cc922f-8dea-4fed-b951-1060ba116ad5', = options=3DNone) from=3D::ffff:192.168.0.4,49920, = task_id=3D97d4ae6c-3719-4149-85fd-28e924ad8883 (api:46)<br = class=3D"">29907:2018-03-21 10:46:55,666+0200 INFO (jsonrpc/2) = [vdsm.api] FINISH getStoragePoolInfo return=3D{'info': {'name': 'No = Description', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'pool_status': 'connected', 'lver': 6L, 'domains': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc87= 97ea149f:Active', 'master_uuid': = u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, = 'type': 'NFS', 'master_ver': 3}, 'dominfo': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', = 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': = '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': = {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': = u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_nfs_iso/8a2c304b-c8ae-4= 38b-af54-fc8797ea149f/images/11111111-1111-1111-1111-111111111111', = 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} = from=3D::ffff:192.168.0.4,49920, = task_id=3D97d4ae6c-3719-4149-85fd-28e924ad8883 (api:52)<br = class=3D"">29908:2018-03-21 10:46:55,667+0200 INFO (jsonrpc/2) = [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 = seconds (__init__:573)<br class=3D"">29909:2018-03-21 10:46:59,473+0200 = INFO (jsonrpc/7) [api.host] START getStats() = from=3D::ffff:192.168.0.4,49914 (api:46)<br class=3D"">29910:2018-03-21 = 10:46:59,474+0200 INFO (jsonrpc/7) [vdsm.api] START = repoStats(domains=3D()) from=3D::ffff:192.168.0.4,49914, = task_id=3D177b551b-9bd3-45a4-84d0-5244630c08e5 (api:46)<br = class=3D"">29911:2018-03-21 10:46:59,474+0200 INFO (jsonrpc/7) = [vdsm.api] FINISH repoStats = return=3D{u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': = True, 'version': 4, 'acquired': True, 'delay': '0.000470906', = 'lastCheck': '9.9', 'valid': True}, = u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000334928', 'lastCheck': = '9.7', 'valid': True}} from=3D::ffff:192.168.0.4,49914, = task_id=3D177b551b-9bd3-45a4-84d0-5244630c08e5 (api:52)<br = class=3D"">29912:2018-03-21 10:46:59,475+0200 INFO (jsonrpc/7) = [vdsm.api] START multipath_health() from=3D::ffff:192.168.0.4,49914, = task_id=3D30753d5c-9bfd-4b0f-ac83-e32278d82b63 (api:46)<br = class=3D"">29913:2018-03-21 10:46:59,475+0200 INFO (jsonrpc/7) = [vdsm.api] FINISH multipath_health return=3D{} = from=3D::ffff:192.168.0.4,49914, = task_id=3D30753d5c-9bfd-4b0f-ac83-e32278d82b63 (api:52)<br = class=3D"">29914:2018-03-21 10:46:59,482+0200 INFO (jsonrpc/7) = [api.host] FINISH getStats return=3D{'status': {'message': 'Done', = 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', = 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '10': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '13': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.86'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', = 'cpuIdle': '99.93'}, '15': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.87'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, = 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '1': {'cpuUser': '1.07', = 'nodeIndex': 1, 'cpuSys': '1.07', 'cpuIdle': '97.86'}, '0': {'cpuUser': = '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '3': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': = '99.86'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', = 'cpuIdle': '99.87'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': = '0.13', 'cpuIdle': '99.80'}, '4': {'cpuUser': '0.47', 'nodeIndex': 0, = 'cpuSys': '0.27', 'cpuIdle': '99.26'}, '7': {'cpuUser': '0.00', = 'nodeIndex': 1, 'cpuSys': '0.47', 'cpuIdle': '99.53'}, '6': {'cpuUser': = '0.20', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.73'}, '9': = {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': = '99.80'}, '8': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', = 'cpuIdle': '99.93'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, = 'memFree': '15125'}, '0': {'memPercent': 5, 'memFree': '15364'}}, = 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, = 'vmCount': 0, 'memUsed': '3', 'storageDomains': = {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, = 'version': 4, 'acquired': True, 'delay': '0.000470906', 'lastCheck': = '9.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.000334928', 'lastCheck': '9.7', 'valid': True}}, = 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'enp3s0f0', = 'tx': '1091412332', 'txDropped': '0', 'rx': '11914976557', 'rxErrors': = '0', 'speed': '100', 'rxDropped': '1456'}, 'ovirtmgmt': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': = 'ovirtmgmt', 'tx': '1048761326', 'txDropped': '0', 'rx': '11615304242', = 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': = '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'lo', 'tx': = '58228460774', 'txDropped': '0', 'rx': '58228460774', 'rxErrors': '0', = 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', = 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'enp3s0f1', = 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': = '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': = 'down', 'sampleTime': 1521622005.383289, 'name': ';vdsmdummy;', 'tx': = '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521622005.383289, 'name': 'enp4s0f0', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', = 'sampleTime': 1521622005.383289, 'name': 'enp4s0f1', 'tx': '0', = 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', = 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '208', = 'ksmPages': 100, 'elapsedTime': '76578.31', 'cpuLoad': '0.17', 'cpuSys': = '0.20', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': = {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.13', = 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, = 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31078, 'bootTime': = '1521545382', 'haStats': {'active': False, 'configured': False, 'score': = 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'multipathHealth': {}, 'rxDropped': '1456', = 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, = 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': = 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, = 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, = 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, = 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': = 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': = 0}}), 'dateTime': '2018-03-21T08:46:59 GMT', 'cpuUser': '0.15', = 'memFree': 31334, 'cpuIdle': '99.65', 'vmActive': 0, 'v2vJobs': {}, = 'cpuSysVdsmd': '0.60'}} from=3D::ffff:192.168.0.4,49914 (api:52)<br = class=3D"">29915:2018-03-21 10:46:59,484+0200 INFO (jsonrpc/7) = [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds = (__init__:573)</div><div class=3D""><br class=3D""><div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D"">On 21 = Mar 2018, at 11:33, Fred Rolland <<a = href=3D"mailto:frolland@redhat.com" class=3D"">frolland@redhat.com</a>>= wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = dir=3D"ltr" class=3D""><div class=3D"">Can you provide the vdsm logs = from the host.<br class=3D""></div>It looks the vdsm failed to connect = to the server.<br class=3D""></div><div class=3D"gmail_extra"><br = class=3D""><div class=3D"gmail_quote">On Wed, Mar 21, 2018 at 11:21 AM, = Andrei Verovski <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:andreil1@starlett.lv" target=3D"_blank" = class=3D"">andreil1@starlett.lv</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div = style=3D"word-wrap:break-word;line-break:after-white-space" = class=3D"">Hi,<div class=3D""><br class=3D""></div><div class=3D"">I = have 2-host oVirt setup with 2 Data Centers, one with local storage = domain (DC #1) for VMs + Export domain on NFS, another with all NFS = shared (DC #2).</div><div class=3D"">Trying to export VMs from DC = #1 to DC #2.</div><div class=3D"">VMs are exported to DC #1 export = domain (NFS), then domain put into maintenance mode and detached from DC = #1.</div><div class=3D""><br class=3D""></div><div = class=3D"">Unfortunately, attaching it to DC #2 failed. Logs attached. = Tried to run this command twice.</div><div class=3D"">Workaround are = possible in order to accomplish this task, yet it would be better to do = in a way as it was designed.<br class=3D""> Thanks.<br class=3D""><br class=3D""><br class=3D""><div = class=3D"">2018-03-21 10:43:24,024+02 ERROR = [org.ovirt.engine.core.dal.<wbr class=3D"">dbbroker.auditloghandling.<wbr = class=3D"">AuditLogDirector] (EE-ManagedThreadFactory-<wbr = class=3D"">engine-Thread-128904) [<font color=3D"#942192" = class=3D"">570ec5d9-fff5-4656-afbd-<wbr class=3D"">90b3207a616e</font>] = EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_<wbr class=3D"">POOL_FAILED(963), = Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center = StrDataCenter11. (User: admin@internal-authz)<br class=3D"">2018-03-21 = 10:46:16,512+02 ERROR [org.ovirt.engine.core.dal.<wbr = class=3D"">dbbroker.auditloghandling.<wbr class=3D"">AuditLogDirector] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128956) = [1435fc81] EVENT_ID: VDS_STORAGES_CONNECTION_<wbr class=3D"">FAILED(188), = Failed to connect Host node11 to the Storage Domains = node10-NFS-EXPORTS.<br class=3D"">2018-03-21 10:46:16,651+02 ERROR = [org.ovirt.engine.core.dal.<wbr class=3D"">dbbroker.auditloghandling.<wbr = class=3D"">AuditLogDirector] (EE-ManagedThreadFactory-<wbr = class=3D"">engine-Thread-128955) [<font color=3D"#942192" = class=3D"">a81ffa4a-5a58-41a0-888a-<wbr class=3D"">f0edc321609b</font>] = EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_<wbr class=3D"">POOL_FAILED(963), = Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center = StrDataCenter11. (User: admin@internal-authz)<br class=3D""><br = class=3D""><br class=3D""><font color=3D"#aa7942" class=3D""><b = class=3D"">tail -n 1000 engine.log | grep 570ec5d9-fff5-4656-afbd-<wbr = class=3D"">90b3207a616e</b></font><br class=3D"">2018-03-21 = 10:41:14,643+02 INFO [org.ovirt.engine.core.bll.<wbr = class=3D"">storage.domain.<wbr = class=3D"">AttachStorageDomainToPoolComma<wbr class=3D"">nd] (default = task-2) [570ec5d9-fff5-4656-afbd-<wbr class=3D"">90b3207a616e] Lock = Acquired to object 'EngineLock:{exclusiveLocks=3D'[<wbr = class=3D"">1d7208ce-d3a1-4406-9638-<wbr class=3D"">fe7051562994=3DSTORAGE]= ', sharedLocks=3D''}'<br class=3D"">2018-03-21 10:41:16,129+02 = INFO [org.ovirt.engine.core.bll.<wbr class=3D"">storage.domain.<wbr = class=3D"">AttachStorageDomainToPoolComma<wbr class=3D"">nd] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-<wbr class=3D"">90b3207a616e] Running command: = AttachStorageDomainToPoolComma<wbr class=3D"">nd internal: false. = Entities affected : ID: 1d7208ce-d3a1-4406-9638-<wbr = class=3D"">fe7051562994 Type: StorageAction group = MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: = 80cc922f-8dea-4fed-b951-<wbr class=3D"">1060ba116ad5 Type: = StoragePoolAction group MANIPULATE_STORAGE_DOMAIN with role type = ADMIN<br class=3D"">2018-03-21 10:43:23,564+02 ERROR = [org.ovirt.engine.core.bll.<wbr class=3D"">storage.domain.<wbr = class=3D"">AttachStorageDomainToPoolComma<wbr class=3D"">nd] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-<wbr class=3D"">90b3207a616e] Cannot connect = storage connection server, aborting attach storage domain operation.<br = class=3D"">2018-03-21 10:43:23,567+02 INFO = [org.ovirt.engine.core.bll.<wbr class=3D"">CommandCompensator] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-<wbr class=3D"">90b3207a616e] Command = [id=3D921ca7cd-4f93-46aa-8de2-<wbr class=3D"">91b13b8f96cb]: = Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.<wbr = class=3D"">businessentities.<wbr class=3D"">StoragePoolIsoMap; snapshot: = StoragePoolIsoMapId:{<wbr class=3D"">storagePoolId=3D'80cc922f-8dea-<wbr = class=3D"">4fed-b951-1060ba116ad5', storageId=3D'1d7208ce-d3a1-4406-<wbr = class=3D"">9638-fe7051562994'}.<br class=3D"">2018-03-21 10:43:24,024+02 = ERROR [org.ovirt.engine.core.dal.<wbr = class=3D"">dbbroker.auditloghandling.<wbr class=3D"">AuditLogDirector] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-<wbr class=3D"">90b3207a616e] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_<wbr class=3D"">POOL_FAILED(963), Failed = to attach Storage Domain node10-NFS-EXPORTS to Data Center = StrDataCenter11. (User: admin@internal-authz)<br class=3D"">2018-03-21 = 10:43:24,114+02 INFO [org.ovirt.engine.core.bll.<wbr = class=3D"">storage.domain.<wbr = class=3D"">AttachStorageDomainToPoolComma<wbr class=3D"">nd] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128904) = [570ec5d9-fff5-4656-afbd-<wbr class=3D"">90b3207a616e] Lock freed to = object 'EngineLock:{exclusiveLocks=3D'[<wbr = class=3D"">1d7208ce-d3a1-4406-9638-<wbr class=3D"">fe7051562994=3DSTORAGE]= ', sharedLocks=3D''}'<br class=3D""><br class=3D""><br class=3D""><br = class=3D""><font color=3D"#aa7942" class=3D""><b class=3D"">[root@node00 = ovirt-engine]# tail -n 1000 engine.log | grep = a81ffa4a-5a58-41a0-888a-<wbr class=3D"">f0edc321609b</b></font><br = class=3D"">2018-03-21 10:44:11,025+02 INFO = [org.ovirt.engine.core.bll.<wbr class=3D"">storage.domain.<wbr = class=3D"">AttachStorageDomainToPoolComma<wbr class=3D"">nd] (default = task-16) [a81ffa4a-5a58-41a0-888a-<wbr class=3D"">f0edc321609b] Lock = Acquired to object 'EngineLock:{exclusiveLocks=3D'[<wbr = class=3D"">1d7208ce-d3a1-4406-9638-<wbr class=3D"">fe7051562994=3DSTORAGE]= ', sharedLocks=3D''}'<br class=3D"">2018-03-21 10:44:11,236+02 = INFO [org.ovirt.engine.core.bll.<wbr class=3D"">storage.domain.<wbr = class=3D"">AttachStorageDomainToPoolComma<wbr class=3D"">nd] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-<wbr class=3D"">f0edc321609b] Running command: = AttachStorageDomainToPoolComma<wbr class=3D"">nd internal: false. = Entities affected : ID: 1d7208ce-d3a1-4406-9638-<wbr = class=3D"">fe7051562994 Type: StorageAction group = MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: = 80cc922f-8dea-4fed-b951-<wbr class=3D"">1060ba116ad5 Type: = StoragePoolAction group MANIPULATE_STORAGE_DOMAIN with role type = ADMIN<br class=3D"">2018-03-21 10:46:16,567+02 ERROR = [org.ovirt.engine.core.bll.<wbr class=3D"">storage.domain.<wbr = class=3D"">AttachStorageDomainToPoolComma<wbr class=3D"">nd] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-<wbr class=3D"">f0edc321609b] Cannot connect = storage connection server, aborting attach storage domain operation.<br = class=3D"">2018-03-21 10:46:16,568+02 INFO = [org.ovirt.engine.core.bll.<wbr class=3D"">CommandCompensator] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-<wbr class=3D"">f0edc321609b] Command = [id=3Db5c25100-1a8a-4db0-9509-<wbr class=3D"">99cfa60995b1]: = Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.<wbr = class=3D"">businessentities.<wbr class=3D"">StoragePoolIsoMap; snapshot: = StoragePoolIsoMapId:{<wbr class=3D"">storagePoolId=3D'80cc922f-8dea-<wbr = class=3D"">4fed-b951-1060ba116ad5', storageId=3D'1d7208ce-d3a1-4406-<wbr = class=3D"">9638-fe7051562994'}.<br class=3D"">2018-03-21 10:46:16,651+02 = ERROR [org.ovirt.engine.core.dal.<wbr = class=3D"">dbbroker.auditloghandling.<wbr class=3D"">AuditLogDirector] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-<wbr class=3D"">f0edc321609b] EVENT_ID: = USER_ATTACH_STORAGE_DOMAIN_TO_<wbr class=3D"">POOL_FAILED(963), Failed = to attach Storage Domain node10-NFS-EXPORTS to Data Center = StrDataCenter11. (User: admin@internal-authz)<br class=3D"">2018-03-21 = 10:46:16,681+02 INFO [org.ovirt.engine.core.bll.<wbr = class=3D"">storage.domain.<wbr = class=3D"">AttachStorageDomainToPoolComma<wbr class=3D"">nd] = (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-128955) = [a81ffa4a-5a58-41a0-888a-<wbr class=3D"">f0edc321609b] Lock freed to = object 'EngineLock:{exclusiveLocks=3D'[<wbr = class=3D"">1d7208ce-d3a1-4406-9638-<wbr class=3D"">fe7051562994=3DSTORAGE]= ', sharedLocks=3D'=E2=80=99}'</div><div class=3D""><br = class=3D""></div></div></div><br = class=3D"">______________________________<wbr = class=3D"">_________________<br class=3D""> Users mailing list<br class=3D""> <a href=3D"mailto:Users@ovirt.org" class=3D"">Users@ovirt.org</a><br = class=3D""> <a href=3D"http://lists.ovirt.org/mailman/listinfo/users" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/users</a><br class=3D""> <br class=3D""></blockquote></div><br class=3D""></div> </div></blockquote></div><br class=3D""></div></body></html>= --Apple-Mail=_C5052CF0-37B9-4927-AE48-DF656F09C5B6--
node11 fails to mount the export domain from node 10. You can try manually to see if you have access from node 11. Look at this page to debug the NFS connection: https://www.ovirt.org/documentation/how-to/troubleshooting/troubleshooting-n... On Wed, Mar 21, 2018 at 12:04 PM, Andrei Verovski <andreil1@starlett.lv> wrote:
Hi,
Errors occurred at 10.43 AM and 10.46 AM node00.starlett.lv - 192.168.0.4 - oVirt host engine (separate PC, not hosted) node10.starlett.lv - 192.168.0.5 - host #1 of DC #1, export domain from which was detached from DC #1 node11.starlett.lv - 192.168.0.6 - host #1 of DC #2,
Logs from DC#2 node11, to which I’m trying to attach export domain located at NFS share node10.
*grep 570ec5d9-fff5-4656-afbd-90b3207a616e*
within vdsm.log returned nothing, so I did
*grep -n 10:43 vdsm.log | tail -1000* 1011:2018-03-21 06:10:43,077+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,36114 (api:46) 1012:2018-03-21 06:10:43,077+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52) 1013:2018-03-21 06:10:43,077+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573) 1014:2018-03-21 06:10:43,868+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46) 1015:2018-03-21 06:10:43,868+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52) 1016:2018-03-21 06:10:43,868+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573) 19729:2018-03-21 09:10:43,641+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,36114 (api:46) 19730:2018-03-21 09:10:43,641+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52) 19731:2018-03-21 09:10:43,642+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573) 29302:2018-03-21 10:43:00,085+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=7b690c8a-5470-44da-a8f3-7e7e9b018e88 (api:46) 29303:2018-03-21 10:43:00,085+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000477038', 'lastCheck': '0.5', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': '0.4', 'valid': True}} from=internal, task_id=7b690c8a-5470-44da-a8f3-7e7e9b018e88 (api:52) 29304:2018-03-21 10:43:00,086+0200 INFO (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=9a625d43-a03c-429c-856f-9aa8e5ff65b5 (api:46) 29305:2018-03-21 10:43:00,086+0200 INFO (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=9a625d43-a03c-429c-856f-9aa8e5ff65b5 (api:52) 29306:2018-03-21 10:43:05,382+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46) 29307:2018-03-21 10:43:05,383+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52) 29308:2018-03-21 10:43:05,383+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573) 29309:2018-03-21 10:43:06,248+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46) 29310:2018-03-21 10:43:06,249+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=f80f9bb8-2e84-4f45-ab58-09a88e808cf3 (api:46) 29311:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000477038', 'lastCheck': '6.7', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': '6.6', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=f80f9bb8-2e84-4f45-ab58-09a88e808cf3 (api:52) 29312:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=ddd7064d-cd12-43bf-a177-b2a93cc9035e (api:46) 29313:2018-03-21 10:43:06,250+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=ddd7064d-cd12-43bf-a177-b2a93cc9035e (api:52) 29314:2018-03-21 10:43:06,256+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '10': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.47'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '12': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '15': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.47', 'cpuIdle': '99.33'}, '14': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '1': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': '1.60', 'cpuIdle': '97.27'}, '0': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.26'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '2': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.54'}, '5': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.74'}, '4': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '7': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.60', 'cpuIdle': '99.27'}, '6': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '8': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15142'}, '0': {'memPercent': 5, 'memFree': '15349'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000477038', 'lastCheck': '6.7', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000498252', 'lastCheck': '6.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'enp3s0f0', 'tx': '1091287978', 'txDropped': '0', 'rx': '11914868563', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1439'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'ovirtmgmt', 'tx': '1048639112', 'txDropped': '0', 'rx': '11615209092', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'lo', 'tx': '58057075945', 'txDropped': '0', 'rx': '58057075945', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621780.072418, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621780.072418, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621780.072418, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621780.072418, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76345.09', 'cpuLoad': '0.16', 'cpuSys': '0.36', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31081, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1439', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:43:06 GMT', 'cpuUser': '0.21', 'memFree': 31337, 'cpuIdle': '99.43', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.93'}} from=::ffff:192.168.0.4,49914 (api:52) 29315:2018-03-21 10:43:06,258+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573) 29316:2018-03-21 10:43:06,632+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,36114 (api:46) 29317:2018-03-21 10:43:06,632+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52) 29318:2018-03-21 10:43:06,633+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573) 29319:2018-03-21 10:43:07,415+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=276c13c7-61ec-4782-a0d9-5cd0f0b0a729 (api:46) 29320:2018-03-21 10:43:07,420+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=276c13c7-61ec-4782-a0d9-5cd0f0b0a729 (api:52) 29321:2018-03-21 10:43:07,420+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573) 29322:2018-03-21 10:43:07,427+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=344b18df-a428-429a-b49f-381f901d1618 (api:46) 29323:2018-03-21 10:43:07,434+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3- 85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=344b18df-a428-429a-b49f-381f901d1618 (api:52) 29324:2018-03-21 10:43:07,435+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573) 29325:2018-03-21 10:43:13,839+0200 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/1 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524', u'connection': u'node10.starlett.lv:/vmdata/nfs/exports', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'protocol_version': u'auto', u'password': '********', u'port': u''}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': u'5f98dfd5-998f-44a6-8594-e5f40144b0ed'} at 0x4076150> timeout=60, duration=120 at 0x402fe50> task#=3754 at 0x353e690>, traceback: 29376:2018-03-21 10:43:15,104+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=672a883e-1d60-4981-9eee-bdcac2ec30c1 (api:46) 29377:2018-03-21 10:43:15,105+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000488777', 'lastCheck': '5.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000481962', 'lastCheck': '5.4', 'valid': True}} from=internal, task_id=672a883e-1d60-4981-9eee-bdcac2ec30c1 (api:52) 29378:2018-03-21 10:43:15,105+0200 INFO (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=55f1b797-da2a-4b07-affe-5b9e41a74307 (api:46) 29379:2018-03-21 10:43:15,106+0200 INFO (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=55f1b797-da2a-4b07-affe-5b9e41a74307 (api:52) 29380:2018-03-21 10:43:17,645+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=52a6ddeb-3c20-4f3f-b27d-94d19580b473 (api:46) 29381:2018-03-21 10:43:17,649+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=52a6ddeb-3c20-4f3f-b27d-94d19580b473 (api:52) 29382:2018-03-21 10:43:17,650+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573) 29383:2018-03-21 10:43:17,656+0200 INFO (jsonrpc/2) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=1a103883-6a6a-4d61-ade6-96559591ddb0 (api:46) 29384:2018-03-21 10:43:17,663+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3- 85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=1a103883-6a6a-4d61-ade6-96559591ddb0 (api:52) 29385:2018-03-21 10:43:17,663+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573) 29386:2018-03-21 10:43:18,882+0200 ERROR (jsonrpc/1) [storage.HSM] Could not connect to storageServer (hsm:2407) 29406:2018-03-21 10:43:18,883+0200 INFO (jsonrpc/1) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 477, 'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524'}]} from=::ffff:192.168.0.4,49914, flow_id=4a53b512, task_id=e598cbe0-cde8-4c73-b526-4398df05e67f (api:52) 29407:2018-03-21 10:43:18,883+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 125.04 seconds (__init__:573) 29408:2018-03-21 10:43:20,387+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46) 29409:2018-03-21 10:43:20,388+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52) 29410:2018-03-21 10:43:20,388+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573) 29411:2018-03-21 10:43:21,497+0200 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46) 29412:2018-03-21 10:43:21,498+0200 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=a7469c1f-3d6d-42e7-8798-5c5335da4a9b (api:46) 29413:2018-03-21 10:43:21,498+0200 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000445096', 'lastCheck': '1.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000296586', 'lastCheck': '1.8', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=a7469c1f-3d6d-42e7-8798-5c5335da4a9b (api:52) 29414:2018-03-21 10:43:21,499+0200 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=9fa5dae3-e2bd-4186-9a1d-afb0aa6e2511 (api:46) 29415:2018-03-21 10:43:21,499+0200 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=9fa5dae3-e2bd-4186-9a1d-afb0aa6e2511 (api:52) 29416:2018-03-21 10:43:21,505+0200 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '10': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '12': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '15': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.60'}, '1': {'cpuUser': '1.00', 'nodeIndex': 1, 'cpuSys': '1.00', 'cpuIdle': '98.00'}, '0': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '3': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '2': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.66'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '7': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '6': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '8': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15142'}, '0': {'memPercent': 5, 'memFree': '15351'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000445096', 'lastCheck': '1.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000296586', 'lastCheck': '1.8', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'enp3s0f0', 'tx': '1091295628', 'txDropped': '0', 'rx': '11914875210', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1442'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'ovirtmgmt', 'tx': '1048646480', 'txDropped': '0', 'rx': '11615214750', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'lo', 'tx': '58066397668', 'txDropped': '0', 'rx': '58066397668', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621795.091263, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621795.091263, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621795.091263, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621795.091263, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76360.34', 'cpuLoad': '0.15', 'cpuSys': '0.18', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31031, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1442', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:43:21 GMT', 'cpuUser': '0.15', 'memFree': 31287, 'cpuIdle': '99.67', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.67'}} from=::ffff:192.168.0.4,49914 (api:52) 29417:2018-03-21 10:43:21,508+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.02 seconds (__init__:573) 29418:2018-03-21 10:43:21,651+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,36114 (api:46) 29419:2018-03-21 10:43:21,651+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52) 29420:2018-03-21 10:43:21,652+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573) 29421:2018-03-21 10:43:27,764+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=46d19cab-8903-4c52-9bb2-4dd8f370997e (api:46) 29422:2018-03-21 10:43:27,769+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=46d19cab-8903-4c52-9bb2-4dd8f370997e (api:52) 29423:2018-03-21 10:43:27,770+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573) 29424:2018-03-21 10:43:27,815+0200 INFO (jsonrpc/6) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=290acc73-8f25-4fd7-9263-7bd05d46430a (api:46) 29425:2018-03-21 10:43:27,822+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3- 85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=290acc73-8f25-4fd7-9263-7bd05d46430a (api:52) 29426:2018-03-21 10:43:27,823+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573) 29427:2018-03-21 10:43:30,121+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=0abdebcf-2b0b-4d33-b552-c0f2ed3e567b (api:46) 29428:2018-03-21 10:43:30,122+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000402077', 'lastCheck': '0.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': '0.5', 'valid': True}} from=internal, task_id=0abdebcf-2b0b-4d33-b552-c0f2ed3e567b (api:52) 29429:2018-03-21 10:43:30,122+0200 INFO (periodic/3) [vdsm.api] START multipath_health() from=internal, task_id=acde7e7d-ab4a-4811-934f-4d312da33208 (api:46) 29430:2018-03-21 10:43:30,123+0200 INFO (periodic/3) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=acde7e7d-ab4a-4811-934f-4d312da33208 (api:52) 29431:2018-03-21 10:43:35,393+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46) 29432:2018-03-21 10:43:35,394+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52) 29433:2018-03-21 10:43:35,394+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573) 29434:2018-03-21 10:43:36,671+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,36114 (api:46) 29435:2018-03-21 10:43:36,671+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52) 29436:2018-03-21 10:43:36,672+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573) 29437:2018-03-21 10:43:37,356+0200 INFO (jsonrpc/1) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46) 29438:2018-03-21 10:43:37,357+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=bc14e253-d3d4-4cc3-9ae1-c0e6ac6e37f7 (api:46) 29439:2018-03-21 10:43:37,357+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000402077', 'lastCheck': '7.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': '7.7', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=bc14e253-d3d4-4cc3-9ae1-c0e6ac6e37f7 (api:52) 29440:2018-03-21 10:43:37,358+0200 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=23c698b8-d5bd-4746-b738-97b10f3e4bfb (api:46) 29441:2018-03-21 10:43:37,358+0200 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=23c698b8-d5bd-4746-b738-97b10f3e4bfb (api:52) 29442:2018-03-21 10:43:37,365+0200 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '10': {'cpuUser': '3.06', 'nodeIndex': 0, 'cpuSys': '0.80', 'cpuIdle': '96.14'}, '13': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '12': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '15': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '14': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.33'}, '1': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': '1.33', 'cpuIdle': '97.54'}, '0': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.20'}, '3': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '2': {'cpuUser': '0.67', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '98.93'}, '5': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.74'}, '4': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '7': {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': '0.73', 'cpuIdle': '98.94'}, '6': {'cpuUser': '5.39', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '94.14'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '8': {'cpuUser': '0.53', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.07'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15143'}, '0': {'memPercent': 5, 'memFree': '15348'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000402077', 'lastCheck': '7.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000344445', 'lastCheck': '7.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'enp3s0f0', 'tx': '1091305295', 'txDropped': '0', 'rx': '11914880953', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1442'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'ovirtmgmt', 'tx': '1048655897', 'txDropped': '0', 'rx': '11615220341', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'lo', 'tx': '58078891247', 'txDropped': '0', 'rx': '58078891247', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621810.108511, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621810.108511, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621810.108511, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621810.108511, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76376.19', 'cpuLoad': '0.16', 'cpuSys': '0.39', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.00', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1442', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:43:37 GMT', 'cpuUser': '0.79', 'memFree': 31336, 'cpuIdle': '98.82', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.73'}} from=::ffff:192.168.0.4,49914 (api:52) 29443:2018-03-21 10:43:37,367+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573) 29444:2018-03-21 10:43:38,124+0200 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=9d2a82dd-c76c-4eb5-a1e1-b49bf004384e (api:46) 29445:2018-03-21 10:43:38,128+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=9d2a82dd-c76c-4eb5-a1e1-b49bf004384e (api:52) 29446:2018-03-21 10:43:38,129+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573) 29447:2018-03-21 10:43:38,171+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=62d44b0b-20b2-47ae-bfb8-d4717e301b69 (api:46) 29448:2018-03-21 10:43:38,177+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3- 85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=62d44b0b-20b2-47ae-bfb8-d4717e301b69 (api:52) 29449:2018-03-21 10:43:38,178+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573) 29450:2018-03-21 10:43:45,147+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=49c33285-a222-464d-8984-4e75a5c6354a (api:46) 29451:2018-03-21 10:43:45,148+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000454102', 'lastCheck': '5.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000347381', 'lastCheck': '5.5', 'valid': True}} from=internal, task_id=49c33285-a222-464d-8984-4e75a5c6354a (api:52) 29452:2018-03-21 10:43:45,148+0200 INFO (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=eed6b564-76b1-4f9b-80a2-1f0f6d9bbe32 (api:46) 29453:2018-03-21 10:43:45,149+0200 INFO (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=eed6b564-76b1-4f9b-80a2-1f0f6d9bbe32 (api:52) 29454:2018-03-21 10:43:48,405+0200 INFO (jsonrpc/0) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=84651a37-7d13-41a9-9dcf-2169a9c46fc5 (api:46) 29455:2018-03-21 10:43:48,410+0200 INFO (jsonrpc/0) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=84651a37-7d13-41a9-9dcf-2169a9c46fc5 (api:52) 29456:2018-03-21 10:43:48,411+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573) 29457:2018-03-21 10:43:48,417+0200 INFO (jsonrpc/4) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=4ebd1d1a-11ac-41ee-b459-7e45f008776b (api:46) 29458:2018-03-21 10:43:48,423+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3- 85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=4ebd1d1a-11ac-41ee-b459-7e45f008776b (api:52) 29459:2018-03-21 10:43:48,424+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573) 29460:2018-03-21 10:43:50,399+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46) 29461:2018-03-21 10:43:50,400+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52) 29462:2018-03-21 10:43:50,400+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573) 29463:2018-03-21 10:43:51,694+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,36114 (api:46) 29464:2018-03-21 10:43:51,694+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52) 29465:2018-03-21 10:43:51,695+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573) 29466:2018-03-21 10:43:52,516+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46) 29467:2018-03-21 10:43:52,517+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=0a88f0dc-eccf-4d24-9ac2-22e079a1480a (api:46) 29468:2018-03-21 10:43:52,517+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000487937', 'lastCheck': '3.0', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000455956', 'lastCheck': '2.9', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=0a88f0dc-eccf-4d24-9ac2-22e079a1480a (api:52) 29469:2018-03-21 10:43:52,518+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=a7a44ff7-a137-41b6-9ad5-2c0b32ab79ec (api:46) 29470:2018-03-21 10:43:52,518+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=a7a44ff7-a137-41b6-9ad5-2c0b32ab79ec (api:52) 29471:2018-03-21 10:43:52,524+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '10': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '13': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '15': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '1': {'cpuUser': '1.13', 'nodeIndex': 1, 'cpuSys': '1.13', 'cpuIdle': '97.74'}, '0': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '5': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '4': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.60'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '99.67'}, '6': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '8': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15143'}, '0': {'memPercent': 5, 'memFree': '15349'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000487937', 'lastCheck': '3.0', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000455956', 'lastCheck': '2.9', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'enp3s0f0', 'tx': '1091312739', 'txDropped': '0', 'rx': '11914899051', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1443'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'ovirtmgmt', 'tx': '1048663107', 'txDropped': '0', 'rx': '11615236520', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'lo', 'tx': '58090295494', 'txDropped': '0', 'rx': '58090295494', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621825.13588, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621825.13588, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621825.13588, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621825.13588, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76391.35', 'cpuLoad': '0.15', 'cpuSys': '0.20', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.13', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1443', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:43:52 GMT', 'cpuUser': '0.14', 'memFree': 31336, 'cpuIdle': '99.66', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.73'}} from=::ffff:192.168.0.4,49914 (api:52) 29472:2018-03-21 10:43:52,526+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573) 29473:2018-03-21 10:43:58,686+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=d6d29cf6-c083-4639-91a2-f062eae4e629 (api:46) 29474:2018-03-21 10:43:58,690+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=d6d29cf6-c083-4639-91a2-f062eae4e629 (api:52) 29475:2018-03-21 10:43:58,690+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573) 29476:2018-03-21 10:43:58,697+0200 INFO (jsonrpc/5) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=9dbd91df-dd9c-469c-8308-bfbfc228937c (api:46) 29477:2018-03-21 10:43:58,703+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3- 85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=9dbd91df-dd9c-469c-8308-bfbfc228937c (api:52) 29478:2018-03-21 10:43:58,704+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573)
*grep -n 10:46 vdsm.log | tail -1000* 1017:2018-03-21 06:10:46,882+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46) 1018:2018-03-21 06:10:46,883+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=6e045b00-88b2-469b-80d6-af38303f8c32 (api:46) 1019:2018-03-21 06:10:46,883+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000332785', 'lastCheck': '7.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000203914', 'lastCheck': '5.6', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=6e045b00-88b2-469b-80d6-af38303f8c32 (api:52) 1020:2018-03-21 06:10:46,884+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=475a5b40-4a43-42ee-9524-2bdd220aed83 (api:46) 1021:2018-03-21 06:10:46,884+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=475a5b40-4a43-42ee-9524-2bdd220aed83 (api:52) 1022:2018-03-21 06:10:46,889+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '10': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '12': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.47'}, '15': {'cpuUser': '0.33', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.40'}, '14': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '1': {'cpuUser': '3.00', 'nodeIndex': 1, 'cpuSys': '1.40', 'cpuIdle': '95.60'}, '0': {'cpuUser': '0.73', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.07'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '2': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.66'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '4': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '7': {'cpuUser': '0.80', 'nodeIndex': 1, 'cpuSys': '0.80', 'cpuIdle': '98.40'}, '6': {'cpuUser': '5.46', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '94.27'}, '9': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '8': {'cpuUser': '1.27', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '98.26'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15128'}, '0': {'memPercent': 5, 'memFree': '15366'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000332785', 'lastCheck': '7.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000203914', 'lastCheck': '5.6', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'enp3s0f0', 'tx': '1082264809', 'txDropped': '0', 'rx': '11907188594', 'rxErrors': '0', 'speed': '100', 'rxDropped': '289'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'ovirtmgmt', 'tx': '1039869554', 'txDropped': '0', 'rx': '11608697383', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'lo', 'tx': '45587320171', 'txDropped': '0', 'rx': '45587320171', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521605437.203776, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '53620', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521605437.203776, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521605437.203776, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521605437.203776, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '60005.72', 'cpuLoad': '0.06', 'cpuSys': '0.38', 'diskStats': {'/var/log': {'free': '7357'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.07', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31084, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '289', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T04:10:46 GMT', 'cpuUser': '0.83', 'memFree': 31340, 'cpuIdle': '98.79', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.80'}} from=::ffff:192.168.0.4,49914 (api:52) 1023:2018-03-21 06:10:46,891+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573) 13522:2018-03-21 08:10:46,827+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46) 13523:2018-03-21 08:10:46,828+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52) 13524:2018-03-21 08:10:46,828+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573) 29734:2018-03-21 10:46:00,338+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=af099ce6-0f44-461b-9e8e-99de63b3884f (api:46) 29735:2018-03-21 10:46:00,339+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470245', 'lastCheck': '0.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000476935', 'lastCheck': '0.7', 'valid': True}} from=internal, task_id=af099ce6-0f44-461b-9e8e-99de63b3884f (api:52) 29736:2018-03-21 10:46:00,339+0200 INFO (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=4a5709d9-e322-44eb-a8a4-bbe3f5e5e1bd (api:46) 29737:2018-03-21 10:46:00,340+0200 INFO (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=4a5709d9-e322-44eb-a8a4-bbe3f5e5e1bd (api:52) 29738:2018-03-21 10:46:04,637+0200 INFO (jsonrpc/7) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=5b9300f7-f8c4-4f63-a49c-45ca8119d3b9 (api:46) 29739:2018-03-21 10:46:04,642+0200 INFO (jsonrpc/7) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=5b9300f7-f8c4-4f63-a49c-45ca8119d3b9 (api:52) 29740:2018-03-21 10:46:04,642+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573) 29741:2018-03-21 10:46:04,648+0200 INFO (jsonrpc/1) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=f4db5bfa-e131-4f9a-baaa-e7555c996ab0 (api:46) 29742:2018-03-21 10:46:04,655+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3- 85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=f4db5bfa-e131-4f9a-baaa-e7555c996ab0 (api:52) 29743:2018-03-21 10:46:04,656+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:573) 29744:2018-03-21 10:46:05,451+0200 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46) 29745:2018-03-21 10:46:05,452+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52) 29746:2018-03-21 10:46:05,452+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573) 29747:2018-03-21 10:46:06,903+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,36114 (api:46) 29748:2018-03-21 10:46:06,904+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52) 29749:2018-03-21 10:46:06,904+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573) 29750:2018-03-21 10:46:08,791+0200 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/6 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524', u'connection': u'node10.starlett.lv:/vmdata/nfs/exports', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'protocol_version': u'auto', u'password': '********', u'port': u''}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': u'71143eef-aac6-4996-9783-3e0e3da180c3'} at 0x3604fd0> timeout=60, duration=120 at 0x3604e10> task#=3762 at 0x3541650>, traceback: 29801:2018-03-21 10:46:12,852+0200 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46) 29802:2018-03-21 10:46:12,853+0200 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=391fdab9-d3c2-4075-9c34-177a07a21ec3 (api:46) 29803:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000492277', 'lastCheck': '3.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': '3.2', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=391fdab9-d3c2-4075-9c34-177a07a21ec3 (api:52) 29804:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=7e53adbc-3974-4f51-bfc8-8fbd3ca6b749 (api:46) 29805:2018-03-21 10:46:12,854+0200 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=7e53adbc-3974-4f51-bfc8-8fbd3ca6b749 (api:52) 29806:2018-03-21 10:46:12,861+0200 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '10': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '12': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '15': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '14': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}, '1': {'cpuUser': '1.00', 'nodeIndex': 1, 'cpuSys': '1.33', 'cpuIdle': '97.67'}, '0': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.60'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '2': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.47'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '4': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '7': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.53'}, '6': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '99.13'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '8': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.54'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15142'}, '0': {'memPercent': 5, 'memFree': '15348'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000492277', 'lastCheck': '3.3', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': '3.2', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'enp3s0f0', 'tx': '1091387790', 'txDropped': '0', 'rx': '11914959714', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1451'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'ovirtmgmt', 'tx': '1048735814', 'txDropped': '0', 'rx': '11615289173', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'lo', 'tx': '58192068637', 'txDropped': '0', 'rx': '58192068637', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621960.326184, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621960.326184, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621960.326184, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621960.326184, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76531.69', 'cpuLoad': '0.17', 'cpuSys': '0.32', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.00', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31081, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1451', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:46:12 GMT', 'cpuUser': '0.21', 'memFree': 31337, 'cpuIdle': '99.46', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.80'}} from=::ffff:192.168.0.4,49914 (api:52) 29807:2018-03-21 10:46:12,863+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573) 29808:2018-03-21 10:46:13,829+0200 ERROR (jsonrpc/6) [storage.HSM] Could not connect to storageServer (hsm:2407) 29828:2018-03-21 10:46:13,829+0200 INFO (jsonrpc/6) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 477, 'id': u'461f65a9-3a81-4f3f-a46d-c5ed12520524'}]} from=::ffff:192.168.0.4,49914, flow_id=1435fc81, task_id=0a828d2c-d9f4-4f83-a9e9-7393159d5323 (api:52) 29829:2018-03-21 10:46:13,830+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 125.04 seconds (__init__:573) 29830:2018-03-21 10:46:14,767+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=49cd2ae6-7943-47c9-ad86-9e0b7e58bca3 (api:46) 29831:2018-03-21 10:46:14,772+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=49cd2ae6-7943-47c9-ad86-9e0b7e58bca3 (api:52) 29832:2018-03-21 10:46:14,773+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.01 seconds (__init__:573) 29833:2018-03-21 10:46:14,815+0200 INFO (jsonrpc/2) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=19909616-efb4-490d-982c-66eda9ca4381 (api:46) 29834:2018-03-21 10:46:14,822+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3- 85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=19909616-efb4-490d-982c-66eda9ca4381 (api:52) 29835:2018-03-21 10:46:14,823+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573) 29836:2018-03-21 10:46:15,360+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=da74ea9f-164b-450d-b836-6818caa3fdc5 (api:46) 29837:2018-03-21 10:46:15,360+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000492277', 'lastCheck': '5.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000479153', 'lastCheck': '5.7', 'valid': True}} from=internal, task_id=da74ea9f-164b-450d-b836-6818caa3fdc5 (api:52) 29838:2018-03-21 10:46:15,361+0200 INFO (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=06245e2b-e8ca-41c4-90bf-4294d0c699b8 (api:46) 29839:2018-03-21 10:46:15,361+0200 INFO (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=06245e2b-e8ca-41c4-90bf-4294d0c699b8 (api:52) 29840:2018-03-21 10:46:20,456+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46) 29841:2018-03-21 10:46:20,457+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52) 29842:2018-03-21 10:46:20,457+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573) 29843:2018-03-21 10:46:21,925+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,36114 (api:46) 29844:2018-03-21 10:46:21,926+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52) 29845:2018-03-21 10:46:21,926+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573) 29846:2018-03-21 10:46:24,930+0200 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=d964535d-e5eb-42bc-b324-432a05c364da (api:46) 29847:2018-03-21 10:46:24,935+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=d964535d-e5eb-42bc-b324-432a05c364da (api:52) 29848:2018-03-21 10:46:24,935+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:573) 29849:2018-03-21 10:46:24,987+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=167dcd23-04b5-4de3-9c81-7d25ba56407a (api:46) 29850:2018-03-21 10:46:25,022+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3- 85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=167dcd23-04b5-4de3-9c81-7d25ba56407a (api:52) 29851:2018-03-21 10:46:25,023+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.03 seconds (__init__:573) 29852:2018-03-21 10:46:28,010+0200 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46) 29853:2018-03-21 10:46:28,011+0200 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=011616ce-53c9-4cfc-8b29-247b1be03409 (api:46) 29854:2018-03-21 10:46:28,011+0200 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000218955', 'lastCheck': '8.4', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000351741', 'lastCheck': '8.4', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=011616ce-53c9-4cfc-8b29-247b1be03409 (api:52) 29855:2018-03-21 10:46:28,012+0200 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=85b42a8d-1bf1-495d-ad32-a1f9710a6468 (api:46) 29856:2018-03-21 10:46:28,012+0200 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=85b42a8d-1bf1-495d-ad32-a1f9710a6468 (api:52) 29857:2018-03-21 10:46:28,018+0200 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '10': {'cpuUser': '1.73', 'nodeIndex': 0, 'cpuSys': '0.53', 'cpuIdle': '97.74'}, '13': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.53'}, '12': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '15': {'cpuUser': '0.67', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.06'}, '14': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '99.47'}, '1': {'cpuUser': '1.26', 'nodeIndex': 1, 'cpuSys': '1.33', 'cpuIdle': '97.41'}, '0': {'cpuUser': '0.40', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.53'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.73'}, '5': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '4': {'cpuUser': '13.78', 'nodeIndex': 0, 'cpuSys': '0.80', 'cpuIdle': '85.42'}, '7': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.67', 'cpuIdle': '98.86'}, '6': {'cpuUser': '1.07', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '98.66'}, '9': {'cpuUser': '1.20', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '98.53'}, '8': {'cpuUser': '0.80', 'nodeIndex': 0, 'cpuSys': '0.33', 'cpuIdle': '98.87'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15136'}, '0': {'memPercent': 5, 'memFree': '15357'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000218955', 'lastCheck': '8.4', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000351741', 'lastCheck': '8.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'enp3s0f0', 'tx': '1091397599', 'txDropped': '0', 'rx': '11914966591', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1454'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'ovirtmgmt', 'tx': '1048745347', 'txDropped': '0', 'rx': '11615294897', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'lo', 'tx': '58205583384', 'txDropped': '0', 'rx': '58205583384', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621975.346633, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621975.346633, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621975.346633, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621975.346633, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '204', 'ksmPages': 100, 'elapsedTime': '76546.85', 'cpuLoad': '0.18', 'cpuSys': '0.39', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.20', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31031, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1454', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:46:28 GMT', 'cpuUser': '1.38', 'memFree': 31287, 'cpuIdle': '98.23', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.87'}} from=::ffff:192.168.0.4,49914 (api:52) 29858:2018-03-21 10:46:28,020+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573) 29859:2018-03-21 10:46:29,667+0200 INFO (itmap/0) [IOProcessClient] Starting client ioprocess-575 (__init__:308) 29860:2018-03-21 10:46:29,700+0200 INFO (itmap/1) [IOProcessClient] Starting client ioprocess-576 (__init__:308) 29861:2018-03-21 10:46:29,714+0200 INFO (ioprocess/1711202) [IOProcess] Starting ioprocess (__init__:437) 29862:2018-03-21 10:46:29,725+0200 INFO (ioprocess/1711208) [IOProcess] Starting ioprocess (__init__:437) 29863:2018-03-21 10:46:30,375+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=bdacf1c0-3fcb-43b1-82e2-a041a26d0d7f (api:46) 29864:2018-03-21 10:46:30,376+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00031111', 'lastCheck': '0.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00045594', 'lastCheck': '0.6', 'valid': True}} from=internal, task_id=bdacf1c0-3fcb-43b1-82e2-a041a26d0d7f (api:52) 29865:2018-03-21 10:46:30,376+0200 INFO (periodic/0) [vdsm.api] START multipath_health() from=internal, task_id=3d3cad70-e5c8-49b2-9f5d-92a249cc102d (api:46) 29866:2018-03-21 10:46:30,377+0200 INFO (periodic/0) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=3d3cad70-e5c8-49b2-9f5d-92a249cc102d (api:52) 29867:2018-03-21 10:46:35,108+0200 INFO (jsonrpc/6) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=304d7e7c-03fc-44d6-be32-d083ada09b30 (api:46) 29868:2018-03-21 10:46:35,114+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=304d7e7c-03fc-44d6-be32-d083ada09b30 (api:52) 29869:2018-03-21 10:46:35,114+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:573) 29870:2018-03-21 10:46:35,170+0200 INFO (jsonrpc/4) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=76736a93-d4c4-4b2f-9848-7b8731fe2b67 (api:46) 29871:2018-03-21 10:46:35,176+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3- 85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=76736a93-d4c4-4b2f-9848-7b8731fe2b67 (api:52) 29872:2018-03-21 10:46:35,177+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:573) 29873:2018-03-21 10:46:35,462+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46) 29874:2018-03-21 10:46:35,462+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52) 29875:2018-03-21 10:46:35,463+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573) 29876:2018-03-21 10:46:36,948+0200 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,36114 (api:46) 29877:2018-03-21 10:46:36,948+0200 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52) 29878:2018-03-21 10:46:36,949+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573) 29879:2018-03-21 10:46:43,179+0200 INFO (jsonrpc/1) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46) 29880:2018-03-21 10:46:43,180+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=83bbc95c-6d59-4eec-82b1-e8993be28759 (api:46) 29881:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000486101', 'lastCheck': '3.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': '3.4', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=83bbc95c-6d59-4eec-82b1-e8993be28759 (api:52) 29882:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=9b781324-b50b-49a9-bf31-5eca835dc75c (api:46) 29883:2018-03-21 10:46:43,181+0200 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=9b781324-b50b-49a9-bf31-5eca835dc75c (api:52) 29884:2018-03-21 10:46:43,188+0200 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.40', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.33'}, '10': {'cpuUser': '0.60', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '98.93'}, '13': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.27', 'cpuIdle': '99.73'}, '12': {'cpuUser': '1.46', 'nodeIndex': 0, 'cpuSys': '0.60', 'cpuIdle': '97.94'}, '15': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '99.47'}, '14': {'cpuUser': '0.40', 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '99.13'}, '1': {'cpuUser': '1.33', 'nodeIndex': 1, 'cpuSys': '1.93', 'cpuIdle': '96.74'}, '0': {'cpuUser': '0.60', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.13'}, '3': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '99.40'}, '2': {'cpuUser': '6.99', 'nodeIndex': 0, 'cpuSys': '0.40', 'cpuIdle': '92.61'}, '5': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '4': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.46'}, '7': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.60', 'cpuIdle': '98.93'}, '6': {'cpuUser': '0.53', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.20'}, '9': {'cpuUser': '0.47', 'nodeIndex': 1, 'cpuSys': '0.33', 'cpuIdle': '99.20'}, '8': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15131'}, '0': {'memPercent': 5, 'memFree': '15357'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000486101', 'lastCheck': '3.6', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': '3.4', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'enp3s0f0', 'tx': '1091404888', 'txDropped': '0', 'rx': '11914970667', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1455'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'ovirtmgmt', 'tx': '1048752406', 'txDropped': '0', 'rx': '11615298347', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'lo', 'tx': '58215969455', 'txDropped': '0', 'rx': '58215969455', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521621990.363726, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621990.363726, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621990.363726, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521621990.363726, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '206', 'ksmPages': 100, 'elapsedTime': '76562.02', 'cpuLoad': '0.18', 'cpuSys': '0.45', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.27', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31080, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1455', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:46:43 GMT', 'cpuUser': '0.87', 'memFree': 31336, 'cpuIdle': '98.68', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '1.20'}} from=::ffff:192.168.0.4,49914 (api:52) 29885:2018-03-21 10:46:43,190+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573) 29886:2018-03-21 10:46:45,288+0200 INFO (jsonrpc/5) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=ffe57349-2532-4b64-b32c-8a558273d8e5 (api:46) 29887:2018-03-21 10:46:45,292+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=ffe57349-2532-4b64-b32c-8a558273d8e5 (api:52) 29888:2018-03-21 10:46:45,293+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:573) 29889:2018-03-21 10:46:45,298+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=b2ea28f0-abf0-4258-8152-6035d93aa5dc (api:46) 29890:2018-03-21 10:46:45,305+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3- 85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=b2ea28f0-abf0-4258-8152-6035d93aa5dc (api:52) 29891:2018-03-21 10:46:45,306+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:573) 29892:2018-03-21 10:46:45,395+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=4010bf72-5c86-4daf-a8f9-12ec229a86e4 (api:46) 29893:2018-03-21 10:46:45,395+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000486101', 'lastCheck': '5.8', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.00043444', 'lastCheck': '5.7', 'valid': True}} from=internal, task_id=4010bf72-5c86-4daf-a8f9-12ec229a86e4 (api:52) 29894:2018-03-21 10:46:45,396+0200 INFO (periodic/2) [vdsm.api] START multipath_health() from=internal, task_id=37a1132f-452b-4805-9cc0-b8e74e8d6b02 (api:46) 29895:2018-03-21 10:46:45,396+0200 INFO (periodic/2) [vdsm.api] FINISH multipath_health return={} from=internal, task_id=37a1132f-452b-4805-9cc0-b8e74e8d6b02 (api:52) 29896:2018-03-21 10:46:50,467+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::ffff:192.168.0.4,49914 (api:46) 29897:2018-03-21 10:46:50,468+0200 INFO (jsonrpc/0) [throttled] Current getAllVmStats: {} (throttledlog:103) 29898:2018-03-21 10:46:50,468+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:192.168.0.4,49914 (api:52) 29899:2018-03-21 10:46:50,468+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573) 29900:2018-03-21 10:46:51,967+0200 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,36114 (api:46) 29901:2018-03-21 10:46:51,968+0200 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,36114 (api:52) 29902:2018-03-21 10:46:51,968+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573) 29903:2018-03-21 10:46:55,648+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49914, task_id=a975635f-59d7-4f40-b9a7-a5ea7150368e (api:46) 29904:2018-03-21 10:46:55,653+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 6L}} from=::ffff:192.168.0.4,49914, task_id=a975635f-59d7-4f40-b9a7-a5ea7150368e (api:52) 29905:2018-03-21 10:46:55,653+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:573) 29906:2018-03-21 10:46:55,659+0200 INFO (jsonrpc/2) [vdsm.api] START getStoragePoolInfo(spUUID=u'80cc922f-8dea-4fed-b951-1060ba116ad5', options=None) from=::ffff:192.168.0.4,49920, task_id=97d4ae6c-3719-4149-85fd-28e924ad8883 (api:46) 29907:2018-03-21 10:46:55,666+0200 INFO (jsonrpc/2) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 6L, 'domains': u'ef184b28-1dbc-45ed-b0b3- 85e780cce5d8:Active,8a2c304b-c8ae-438b-af54-fc8797ea149f:Active', 'master_uuid': u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8', 'version': '4', 'spm_id': 1, 'type': 'NFS', 'master_ver': 3}, 'dominfo': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': '', 'alerts': [], 'disktotal': '1968811540480', 'version': 4}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'status': u'Active', 'diskfree': '1868692455424', 'isoprefix': u'/rhev/data-center/mnt/node11.starlett.lv:_vmraid_ nfs_iso/8a2c304b-c8ae-438b-af54-fc8797ea149f/images/ 11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '1968811540480', 'version': 0}}} from=::ffff:192.168.0.4,49920, task_id=97d4ae6c-3719-4149-85fd-28e924ad8883 (api:52) 29908:2018-03-21 10:46:55,667+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.00 seconds (__init__:573) 29909:2018-03-21 10:46:59,473+0200 INFO (jsonrpc/7) [api.host] START getStats() from=::ffff:192.168.0.4,49914 (api:46) 29910:2018-03-21 10:46:59,474+0200 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.0.4,49914, task_id=177b551b-9bd3-45a4-84d0-5244630c08e5 (api:46) 29911:2018-03-21 10:46:59,474+0200 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470906', 'lastCheck': '9.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000334928', 'lastCheck': '9.7', 'valid': True}} from=::ffff:192.168.0.4,49914, task_id=177b551b-9bd3-45a4-84d0-5244630c08e5 (api:52) 29912:2018-03-21 10:46:59,475+0200 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::ffff:192.168.0.4,49914, task_id=30753d5c-9bfd-4b0f-ac83-e32278d82b63 (api:46) 29913:2018-03-21 10:46:59,475+0200 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.0.4,49914, task_id=30753d5c-9bfd-4b0f-ac83-e32278d82b63 (api:52) 29914:2018-03-21 10:46:59,482+0200 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'11': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '10': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '13': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '15': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '99.73'}, '1': {'cpuUser': '1.07', 'nodeIndex': 1, 'cpuSys': '1.07', 'cpuIdle': '97.86'}, '0': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '3': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '2': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.87'}, '5': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '4': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.26'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.47', 'cpuIdle': '99.53'}, '6': {'cpuUser': '0.20', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.73'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.80'}, '8': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}}, 'numaNodeMemFree': {'1': {'memPercent': 7, 'memFree': '15125'}, '0': {'memPercent': 5, 'memFree': '15364'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '3', 'storageDomains': {u'ef184b28-1dbc-45ed-b0b3-85e780cce5d8': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000470906', 'lastCheck': '9.9', 'valid': True}, u'8a2c304b-c8ae-438b-af54-fc8797ea149f': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000334928', 'lastCheck': '9.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp3s0f0': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'enp3s0f0', 'tx': '1091412332', 'txDropped': '0', 'rx': '11914976557', 'rxErrors': '0', 'speed': '100', 'rxDropped': '1456'}, 'ovirtmgmt': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'ovirtmgmt', 'tx': '1048761326', 'txDropped': '0', 'rx': '11615304242', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'lo': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'lo', 'tx': '58228460774', 'txDropped': '0', 'rx': '58228460774', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp3s0f1': {'txErrors': '0', 'state': 'up', 'sampleTime': 1521622005.383289, 'name': 'enp3s0f1', 'tx': '0', 'txDropped': '0', 'rx': '66839', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, ';vdsmdummy;': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521622005.383289, 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f0': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521622005.383289, 'name': 'enp4s0f0', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}, 'enp4s0f1': {'txErrors': '0', 'state': 'down', 'sampleTime': 1521622005.383289, 'name': 'enp4s0f1', 'tx': '0', 'txDropped': '0', 'rx': '0', 'rxErrors': '0', 'speed': '1000', 'rxDropped': '0'}}, 'txDropped': '0', 'anonHugePages': '208', 'ksmPages': 100, 'elapsedTime': '76578.31', 'cpuLoad': '0.17', 'cpuSys': '0.20', 'diskStats': {'/var/log': {'free': '7344'}, '/var/run/vdsm/': {'free': '16060'}, '/tmp': {'free': '906'}}, 'cpuUserVdsmd': '1.13', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 31078, 'bootTime': '1521545382', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '1456', 'outgoingVmMigrations': 0, 'swapTotal': 12287, 'swapFree': 12287, 'hugepages': defaultdict(<type 'dict'>, {2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2018-03-21T08:46:59 GMT', 'cpuUser': '0.15', 'memFree': 31334, 'cpuIdle': '99.65', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.60'}} from=::ffff:192.168.0.4,49914 (api:52) 29915:2018-03-21 10:46:59,484+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.01 seconds (__init__:573)
On 21 Mar 2018, at 11:33, Fred Rolland <frolland@redhat.com> wrote:
Can you provide the vdsm logs from the host. It looks the vdsm failed to connect to the server.
On Wed, Mar 21, 2018 at 11:21 AM, Andrei Verovski <andreil1@starlett.lv> wrote:
Hi,
I have 2-host oVirt setup with 2 Data Centers, one with local storage domain (DC #1) for VMs + Export domain on NFS, another with all NFS shared (DC #2). Trying to export VMs from DC #1 to DC #2. VMs are exported to DC #1 export domain (NFS), then domain put into maintenance mode and detached from DC #1.
Unfortunately, attaching it to DC #2 failed. Logs attached. Tried to run this command twice. Workaround are possible in order to accomplish this task, yet it would be better to do in a way as it was designed. Thanks.
2018-03-21 10:43:24,024+02 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-128904) [570ec5d9-fff5-4656-afbd-90b3207a616e] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz) 2018-03-21 10:46:16,512+02 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-128956) [1435fc81] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Failed to connect Host node11 to the Storage Domains node10-NFS-EXPORTS. 2018-03-21 10:46:16,651+02 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc321609b] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz)
*tail -n 1000 engine.log | grep 570ec5d9-fff5-4656-afbd-90b3207a616e* 2018-03-21 10:41:14,643+02 INFO [org.ovirt.engine.core.bll.sto rage.domain.AttachStorageDomainToPoolCommand] (default task-2) [570ec5d9-fff5-4656-afbd-90b3207a616e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[1d7208ce-d3a1-4406-9638-fe7051562994=STORAGE]', sharedLocks=''}' 2018-03-21 10:41:16,129+02 INFO [org.ovirt.engine.core.bll.sto rage.domain.AttachStorageDomainToPoolCommand] (EE-ManagedThreadFactory-engine-Thread-128904) [570ec5d9-fff5-4656-afbd-90b3207a616e] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 1d7208ce-d3a1-4406-9638-fe7051562994 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: 80cc922f-8dea-4fed-b951-1060ba116ad5 Type: StoragePoolAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2018-03-21 10:43:23,564+02 ERROR [org.ovirt.engine.core.bll.sto rage.domain.AttachStorageDomainToPoolCommand] (EE-ManagedThreadFactory-engine-Thread-128904) [570ec5d9-fff5-4656-afbd-90b3207a616e] Cannot connect storage connection server, aborting attach storage domain operation. 2018-03-21 10:43:23,567+02 INFO [org.ovirt.engine.core.bll.CommandCompensator] (EE-ManagedThreadFactory-engine-Thread-128904) [570ec5d9-fff5-4656-afbd-90b3207a616e] Command [id=921ca7cd-4f93-46aa-8de2-91b13b8f96cb]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: StoragePoolIsoMapId:{storagePoolId='80cc922f-8dea-4fed-b951-1060ba116ad5', storageId='1d7208ce-d3a1-4406-9638-fe7051562994'}. 2018-03-21 10:43:24,024+02 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-128904) [570ec5d9-fff5-4656-afbd-90b3207a616e] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz) 2018-03-21 10:43:24,114+02 INFO [org.ovirt.engine.core.bll.sto rage.domain.AttachStorageDomainToPoolCommand] (EE-ManagedThreadFactory-engine-Thread-128904) [570ec5d9-fff5-4656-afbd-90b3207a616e] Lock freed to object 'EngineLock:{exclusiveLocks='[1d7208ce-d3a1-4406-9638-fe7051562994=STORAGE]', sharedLocks=''}'
*[root@node00 ovirt-engine]# tail -n 1000 engine.log | grep a81ffa4a-5a58-41a0-888a-f0edc321609b* 2018-03-21 10:44:11,025+02 INFO [org.ovirt.engine.core.bll.sto rage.domain.AttachStorageDomainToPoolCommand] (default task-16) [a81ffa4a-5a58-41a0-888a-f0edc321609b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[1d7208ce-d3a1-4406-9638-fe7051562994=STORAGE]', sharedLocks=''}' 2018-03-21 10:44:11,236+02 INFO [org.ovirt.engine.core.bll.sto rage.domain.AttachStorageDomainToPoolCommand] (EE-ManagedThreadFactory-engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc321609b] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 1d7208ce-d3a1-4406-9638-fe7051562994 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: 80cc922f-8dea-4fed-b951-1060ba116ad5 Type: StoragePoolAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2018-03-21 10:46:16,567+02 ERROR [org.ovirt.engine.core.bll.sto rage.domain.AttachStorageDomainToPoolCommand] (EE-ManagedThreadFactory-engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc321609b] Cannot connect storage connection server, aborting attach storage domain operation. 2018-03-21 10:46:16,568+02 INFO [org.ovirt.engine.core.bll.CommandCompensator] (EE-ManagedThreadFactory-engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc321609b] Command [id=b5c25100-1a8a-4db0-9509-99cfa60995b1]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: StoragePoolIsoMapId:{storagePoolId='80cc922f-8dea-4fed-b951-1060ba116ad5', storageId='1d7208ce-d3a1-4406-9638-fe7051562994'}. 2018-03-21 10:46:16,651+02 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc321609b] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage Domain node10-NFS-EXPORTS to Data Center StrDataCenter11. (User: admin@internal-authz) 2018-03-21 10:46:16,681+02 INFO [org.ovirt.engine.core.bll.sto rage.domain.AttachStorageDomainToPoolCommand] (EE-ManagedThreadFactory-engine-Thread-128955) [a81ffa4a-5a58-41a0-888a-f0edc321609b] Lock freed to object 'EngineLock:{exclusiveLocks='[1d7208ce-d3a1-4406-9638-fe7051562994=STORAGE]', sharedLocks='’}'
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (2)
-
Andrei Verovski -
Fred Rolland