[ovirt-users] Host reports about one of the Active Storage Domains as Problematic

Hi All, I'm hoping someone(s) can point me in the correct direction to fix this. The Architecture +++++++++++++ 3 identical VM-Hosts (vmhost{1-3}.mydomain.local): - Rocky Linux 8.8 - 4 local gluster volumes - 4 iscsi volumes provided by 3-node Ceph Cluster (also running Rocky v8.8) - 3 NICS: - One Gluster Transfer - One Ceph Data Transfer - One General Data access / OVirt management Each node of the Ceph Cluster is running an iscsi gateway, and each vmhost connects to each iscsi gateway Everything is working AOK, except vmhost1 complains that it can't reach 3 of the 4 iscsi volumes/domains. I've looked in the engine.log and vdsm.log but can't figure out how to fix things, and I don't know where else to look. engine.log (extract) +++++++++++++++ ~~~ 2023-10-16 18:21:08,892+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-27) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Lock Acquired to object 'EngineLock:{exclusiveLocks='[8e0e99ab-c212-41c4-8318-acd34acb6289=VDS]', sharedLocks=''}' 2023-10-16 18:21:09,179+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2023-10-16 18:21:09,179+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Before acquiring lock in order to prevent monitoring for host 'vmhost1.mydomain.local' from data-center 'my_datacentre' 2023-10-16 18:21:09,179+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Lock acquired, from now a monitoring of host will be skipped for host 'vmhost1.mydomain.local' from data-center 'my_datacentre' 2023-10-16 18:21:09,181+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] START, SetVdsStatusVDSCommand(HostName = vmhost1.mydomain.local, SetVdsStatusVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 732b1116 2023-10-16 18:21:09,245+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] FINISH, SetVdsStatusVDSCommand, return: , log id: 732b1116 2023-10-16 18:21:09,248+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Activate host finished. Lock released. Monitoring can run now for host 'vmhost1.mydomain.local' from data-center 'my_datacentre' 2023-10-16 18:21:09,314+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] EVENT_ID: VDS_ACTIVATE(16), Activation of host vmhost1.mydomain.local initiated by admin@internal-authz. 2023-10-16 18:21:09,373+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Lock freed to object 'EngineLock:{exclusiveLocks='[8e0e99ab-c212-41c4-8318-acd34acb6289=VDS]', sharedLocks=''}' 2023-10-16 18:21:10,620+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [] START, GetHardwareInfoAsyncVDSCommand(HostName = vmhost1.mydomain.local, VdsIdAndVdsVDSCommandParametersBase:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', vds='Host[vmhost1.mydomain.local,8e0e99ab-c212-41c4-8318-acd34acb6289]'}), log id: 3dac178f 2023-10-16 18:21:10,620+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [] FINISH, GetHardwareInfoAsyncVDSCommand, return: , log id: 3dac178f 2023-10-16 18:21:10,688+11 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [44be9d7f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS 2023-10-16 18:21:10,690+11 INFO [org.ovirt.engine.core.bll.UpdateClusterCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] Lock Acquired to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[]'}' 2023-10-16 18:21:10,703+11 INFO [org.ovirt.engine.core.bll.UpdateClusterCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] Running command: UpdateClusterCommand internal: true. Entities affected : ID: c892710d-8611-4146-aea2-05d910f1829b Type: ClusterAction group EDIT_CLUSTER_CONFIGURATION with role type ADMIN 2023-10-16 18:21:10,761+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] EVENT_ID: SYSTEM_UPDATE_CLUSTER(835), Host cluster my_cluster was updated by system 2023-10-16 18:21:10,761+11 INFO [org.ovirt.engine.core.bll.UpdateClusterCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] Lock freed to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[]'}' 2023-10-16 18:21:10,965+11 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [5f0caae0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 7bfaa890-f892-4b3b-bf18-33dabc13e646 Type: StoragePool 2023-10-16 18:21:11,032+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [5f0caae0] START, SetHaMaintenanceModeVDSCommand(HostName = vmhost1.mydomain.local, SetHaMaintenanceModeVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289'}), log id: 630966ec 2023-10-16 18:21:11,034+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [5f0caae0] FINISH, SetHaMaintenanceModeVDSCommand, return: , log id: 630966ec 2023-10-16 18:21:11,170+11 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 7bfaa890-f892-4b3b-bf18-33dabc13e646 Type: StoragePool 2023-10-16 18:21:11,196+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] START, ConnectStorageServerVDSCommand(HostName = vmhost1.mydomain.local, StorageServerConnectionManagementVDSParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='8d7b930e-a758-4010-aaf5-a230447f2761', connection='vmhosttran1.mydomain.local:/gv0', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local:vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 30a928ed 2023-10-16 18:21:11,211+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] FINISH, ConnectStorageServerVDSCommand, return: {8d7b930e-a758-4010-aaf5-a230447f2761=0}, log id: 30a928ed 2023-10-16 18:21:11,211+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] START, ConnectStorageServerVDSCommand(HostName = vmhost1.mydomain.local, StorageServerConnectionManagementVDSParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='044a086b-e94b-4e5d-a118-283ffceb8de8', connection='vmhosttran1.mydomain.local:gv1', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local:vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='2d328788-ade0-4e25-b947-1c392d4f133e', connection='vmhosttran1.mydomain.local:/gv2', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local :vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='420b9d49-eebb-4dea-99bb-bcb89fbe2bf2', connection='vmhosttran1.mydomain.local:/gv3', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local:vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 6aaeee49 2023-10-16 18:21:11,664+11 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [] Fetched 0 VMs from VDS '8e0e99ab-c212-41c4-8318-acd34acb6289' 2023-10-16 18:21:11,917+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] FINISH, ConnectStorageServerVDSCommand, return: {044a086b-e94b-4e5d-a118-283ffceb8de8=0, 2d328788-ade0-4e25-b947-1c392d4f133e=0, 420b9d49-eebb-4dea-99bb-bcb89fbe2bf2=0}, log id: 6aaeee49 2023-10-16 18:21:11,920+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] START, ConnectStorageServerVDSCommand(HostName = vmhost1.mydomain.local, StorageServerConnectionManagementVDSParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', storageType='ISCSI', connectionList='[StorageServerConnections:{id='78f16782-9a06-45f2-92e5-d1ca84a8c67e', connection='192.168.4.24', iqn='iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='868d7c6a-c933-4475-86ce-ab1359d6f4c3', connection='192.168.4.25', iqn='iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='20a52c08-a4c5-4b07-8ecc-bec0e82e582a', connection='192.168.4.23', iqn='iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 4e953edb 2023-10-16 18:21:19,245+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='TPM'}), log id: 694529cf 2023-10-16 18:21:19,247+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@a9992e5, log id: 694529cf 2023-10-16 18:21:19,247+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='NVRAM'}), log id: 786d585b 2023-10-16 18:21:19,248+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@1cf19aa, log id: 786d585b 2023-10-16 18:21:19,248+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='TPM'}), log id: 4104430d 2023-10-16 18:21:19,250+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@7ccef72, log id: 4104430d 2023-10-16 18:21:19,250+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='NVRAM'}), log id: 15f1c050 2023-10-16 18:21:19,251+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@6c8a990d, log id: 15f1c050 2023-10-16 18:21:34,401+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='TPM'}), log id: 3f508f9e 2023-10-16 18:21:34,403+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@5690567a, log id: 3f508f9e 2023-10-16 18:21:34,403+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='NVRAM'}), log id: 270bce0b 2023-10-16 18:21:34,404+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@2a7c9b44, log id: 270bce0b 2023-10-16 18:21:34,404+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='TPM'}), log id: 11ebddea 2023-10-16 18:21:34,406+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@459c7a2b, log id: 11ebddea 2023-10-16 18:21:34,406+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='NVRAM'}), log id: 3c1e9f05 2023-10-16 18:21:34,408+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@72f69579, log id: 3c1e9f05 2023-10-16 18:21:49,594+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='TPM'}), log id: 6a366a0f 2023-10-16 18:21:49,597+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@95edf66, log id: 6a366a0f 2023-10-16 18:21:49,655+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='TPM'}), log id: 40d156 2023-10-16 18:21:49,656+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@3f11df90, log id: 40d156 2023-10-16 18:23:28,802+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] FINISH, ConnectStorageServerVDSCommand, return: {78f16782-9a06-45f2-92e5-d1ca84a8c67e=465, 868d7c6a-c933-4475-86ce-ab1359d6f4c3=465, 20a52c08-a4c5-4b07-8ecc-bec0e82e582a=0}, log id: 4e953edb 2023-10-16 18:23:28,811+11 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] Host 'vmhost1.mydomain.local' storage connection was succeeded 2023-10-16 18:23:28,846+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] START, ConnectStoragePoolVDSCommand(HostName = vmhost1.mydomain.local, ConnectStoragePoolVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', vdsId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', masterVersion='1'}), log id: 3b47a386 2023-10-16 18:23:28,846+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Executing with domain map: {e311ddf1-7f2c-49ef-a618-050d9a2b947f=active, fa8d922a-df3c-4f5b-adf6-a77c3689c397=active, 30775056-38ef-483a-889c-70453e9aa5f9=active, ee5b0268-8141-43e0-995e-8b4586217123=active, 3e33b7ce-7279-4765-a203-cca37eff8cb0=active, 4412e492-f752-4a6c-98ff-f289043618da=active, 0dde39a4-42e4-4fba-8068-e63e42dd3334=active, d44ec7f2-3629-4110-93a5-7d8d57230efc=active} 2023-10-16 18:23:32,385+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 3b47a386 2023-10-16 18:23:32,418+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '0dde39a4-42e4-4fba-8068-e63e42dd3334:iscsi-vmimages' report isn't an actual report 2023-10-16 18:23:32,419+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '3e33b7ce-7279-4765-a203-cca37eff8cb0:gluster-generaldata' report isn't an actual report 2023-10-16 18:23:32,419+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain 'e311ddf1-7f2c-49ef-a618-050d9a2b947f:gluster-vmimages' report isn't an actual report 2023-10-16 18:23:32,419+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '4412e492-f752-4a6c-98ff-f289043618da:gluster-isos' report isn't an actual report 2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain 'fa8d922a-df3c-4f5b-adf6-a77c3689c397:iscsi-databases' was reported with error code '358' 2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '30775056-38ef-483a-889c-70453e9aa5f9:iscsi-generaldata' was reported with error code '358' 2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain 'd44ec7f2-3629-4110-93a5-7d8d57230efc:iscsi-backups' was reported with error code '358' 2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Storage Domain 'iscsi-databases' of pool 'my_datacentre' is in problem in host 'vmhost1.mydomain.local' 2023-10-16 18:23:32,420+11 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Storage Domain 'iscsi-generaldata' of pool 'my_datacentre' is in problem in host 'vmhost1.mydomain.local' 2023-10-16 18:23:32,420+11 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Storage Domain 'iscsi-backups' of pool 'my_datacentre' is in problem in host 'vmhost1.mydomain.local' 2023-10-16 18:23:32,454+11 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] EVENT_ID: VDS_STORAGE_VDS_STATS_FAILED(189), Host vmhost1.mydomain.local reports about one of the Active Storage Domains as Problematic. 2023-10-16 18:23:32,564+11 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS 2023-10-16 18:23:32,566+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] START, SetVdsStatusVDSCommand(HostName = vmhost1.mydomain.local, SetVdsStatusVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 992c9e 2023-10-16 18:23:32,566+11 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] Clearing domains data for host vmhost1.mydomain.local 2023-10-16 18:23:32,619+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] FINISH, SetVdsStatusVDSCommand, return: , log id: 992c9e 2023-10-16 18:23:32,689+11 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Host vmhost1.mydomain.local cannot access the Storage Domain(s) iscsi-databases, iscsi-generaldata, iscsi-backups attached to the Data Center my_datacentre. Setting Host state to Non-Operational. 2023-10-16 18:23:32,789+11 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] EVENT_ID: VDS_ALERT_FENCE_TEST_FAILED(9,001), Power Management test failed for Host vmhost1.mydomain.local.There is no other host in the data center that can be used to test the power management settings. 2023-10-16 18:23:32,856+11 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Failed to connect Host vmhost1.mydomain.local to Storage Pool my_datacentre 2023-10-16 18:23:33,016+11 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [f7474e7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS 2023-10-16 18:23:33,018+11 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [f7474e7] Host 'vmhost1.mydomain.local'(8e0e99ab-c212-41c4-8318-acd34acb6289) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. 2023-10-16 18:23:33,019+11 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [f7474e7] Received first domain report for host vmhost1.mydomain.local 2023-10-16 18:24:02,559+11 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2023-10-16 18:24:02,561+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] START, SetVdsStatusVDSCommand(HostName = vmhost1.mydomain.local, SetVdsStatusVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', status='PreparingForMaintenance', nonOperationalReason='NONE', stopSpmFailureLogged='true', maintenanceReason='null'}), log id: 51d95bef 2023-10-16 18:24:02,592+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] FINISH, SetVdsStatusVDSCommand, return: , log id: 51d95bef 2023-10-16 18:24:02,799+11 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS 2023-10-16 18:24:02,827+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] START, SetHaMaintenanceModeVDSCommand(HostName = vmhost1.mydomain.local, SetHaMaintenanceModeVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289'}), log id: 63f61978 2023-10-16 18:24:02,832+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] FINISH, SetHaMaintenanceModeVDSCommand, return: , log id: 63f61978 2023-10-16 18:24:02,914+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] EVENT_ID: USER_VDS_MAINTENANCE_WITHOUT_REASON(620), Host vmhost1.mydomain.local was switched to Maintenance mode by admin@internal-authz. ~~~ vdsm.log (extract - from vmhost1) ++++++++++++++++++++++++++ ~~~ 2023-10-16 18:21:10,069+1100 INFO (jsonrpc/7) [api.host] START getCapabilities() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:10,410+1100 ERROR (jsonrpc/7) [root] Error getting managedvolume connector info: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc=1 out=b\'\' err=b\'Traceback (most recent call last):\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 91, in <module>\\nfrom importlib import resources as _importlib_resources\\nImportError: cannot import name \\\'resources\\\'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File "/usr/libexec/vdsm/managedvolume-helper", line 26, in <module>\\nfrom os_brick.initiator import connector\\n File "/usr/lib/python3.6/site-packages/os_brick/initiator/connector.py", line 27, in <module>\\nfrom oslo_concurrency import lockutils\\n File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 29, in <module>\\nfrom oslo_config imp ort cfg\\n File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 45, in <module>\\nfrom oslo_config import types\\n File "/usr/lib/python3.6/site-packages/oslo_config/types.py", line 29, in <module>\\nimport netaddr\\n File "/usr/lib/python3.6/site-packages/netaddr/__init__.py", line 18, in <module>\\nfrom netaddr.core import (AddrConversionError, AddrFormatError,\\n File "/usr/lib/python3.6/site-packages/netaddr/core.py", line 11, in <module>\\nfrom netaddr.compat import _callable, _iter_dict_keys\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 93, in <module>\\nimport importlib_resources as _importlib_resources\\nModuleNotFoundError: No module named \\\'importlib_resources\\\'\\n\'',) (caps:165) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/host/caps.py", line 161, in get caps["connector_info"] = managedvolume.connector_info() File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 63, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 78, in connector_info return run_helper("connector_info") File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 184, in run_helper sub_cmd, cmd_input=cmd_input) File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 38, in __call__ return callMethod() File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 36, in <lambda> **kwargs) File "<string>", line 2, in managedvolume_run_helper File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod raise convert_to_error(kind, result) vdsm.storage.exception.ManagedVolumeHelperFailed: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc=1 out=b\'\' err=b\'Traceback (most recent call last):\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 91, in <module>\\nfrom importlib import resources as _importlib_resources\\nImportError: cannot import name \\\'resources\\\'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File "/usr/libexec/vdsm/managedvolume-helper", line 26, in <module>\\nfrom os_brick.initiator import connector\\n File "/usr/lib/python3.6/site-packages/os_brick/initiator/connector.py", line 27, in <module>\\nfrom oslo_concurrency import lockutils\\n File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 29, in <module>\\nfrom oslo_config import cfg\\n File "/usr/lib/python3.6/site-packag es/oslo_config/cfg.py", line 45, in <module>\\nfrom oslo_config import types\\n File "/usr/lib/python3.6/site-packages/oslo_config/types.py", line 29, in <module>\\nimport netaddr\\n File "/usr/lib/python3.6/site-packages/netaddr/__init__.py", line 18, in <module>\\nfrom netaddr.core import (AddrConversionError, AddrFormatError,\\n File "/usr/lib/python3.6/site-packages/netaddr/core.py", line 11, in <module>\\nfrom netaddr.compat import _callable, _iter_dict_keys\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 93, in <module>\\nimport importlib_resources as _importlib_resources\\nModuleNotFoundError: No module named \\\'importlib_resources\\\'\\n\'',) 2023-10-16 18:21:10,600+1100 INFO (jsonrpc/7) [root] /usr/libexec/vdsm/hooks/after_get_caps/ovirt_provider_ovn_hook: rc=0 err=b'' (hooks:106) 2023-10-16 18:21:10,601+1100 INFO (jsonrpc/7) [api.host] FINISH getCapabilities return={'status': {'code': 0, 'message': 'Done'}, 'info': {'kvmEnabled': 'true', 'cpuCores': '16', 'cpuThreads': '32', 'cpuSockets': '2', 'onlineCpus': '0,1,2,3,4,5,6,7,16,17,18,19,20,21,22,23,8,9,10,11,12,13,14,15,24,25,26,27,28,29,30,31', 'cpuTopology': [{'cpu_id': 0, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 0}, {'cpu_id': 1, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 1}, {'cpu_id': 2, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 2}, {'cpu_id': 3, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 3}, {'cpu_id': 4, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 4}, {'cpu_id': 5, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 5}, {'cpu_id': 6, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 6}, {'cpu_id': 7, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 7}, {'cpu_id': 16, 'numa_cell_id': 0, 'socket_id ': 0, 'die_id': 0, 'core_id': 0}, {'cpu_id': 17, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 1}, {'cpu_id': 18, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 2}, {'cpu_id': 19, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 3}, {'cpu_id': 20, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 4}, {'cpu_id': 21, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 5}, {'cpu_id': 22, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 6}, {'cpu_id': 23, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 7}, {'cpu_id': 8, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 0}, {'cpu_id': 9, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 1}, {'cpu_id': 10, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 2}, {'cpu_id': 11, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 3}, {'cpu_id': 12, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 4}, {'cpu_id': 13, 'numa_cell_id' : 1, 'socket_id': 1, 'die_id': 0, 'core_id': 5}, {'cpu_id': 14, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 6}, {'cpu_id': 15, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 7}, {'cpu_id': 24, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 0}, {'cpu_id': 25, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 1}, {'cpu_id': 26, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 2}, {'cpu_id': 27, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 3}, {'cpu_id': 28, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 4}, {'cpu_id': 29, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 5}, {'cpu_id': 30, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 6}, {'cpu_id': 31, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 7}], 'cpuSpeed': '4000.000', 'cpuModel': 'Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz', 'cpuFlags': 'pku,clflushopt,avx2,nopl,pni,pln,dca,smep,pat,art,cx8,msr,pts,arch_capa bilities,xsave,ht,lm,smap,de,mce,pse36,cx16,skip-l1dfl-vmentry,cpuid,bts,lahf_lm,avx512vl,tsc_deadline_timer,bmi2,md-clear,tm2,cmov,sep,sse4_1,invtsc,md_clear,intel_pt,amd-ssbd,mpx,ssse3,rdt_a,cpuid_fault,x2apic,mmx,avx512dq,dtes64,avx512cd,vme,pclmulqdq,cqm_occup_llc,xsavec,ibpb,umip,pschange-mc-no,mtrr,avx512f,stibp,clwb,xsaves,f16c,invpcid_single,avx512bw,arat,pdpe1gb,ibrs,ept,tm,invpcid,cqm_mbm_total,ibrs-all,3dnowprefetch,popcnt,cqm,rdctl-no,xsaveopt,pdcm,aperfmperf,cat_l3,xgetbv1,xtpr,syscall,smx,pbe,sse2,amd-stibp,pebs,xtopology,tsc_adjust,mca,pae,ida,nonstop_tsc,tpr_shadow,ssbd,nx,tsc,dtherm,aes,fxsr,fma,cqm_mbm_local,movbe,erms,intel_ppin,mds-no,ds_cpl,rdseed,dts,ibrs_enhanced,constant_tsc,ept_ad,flexpriority,avx512_vnni,adx,clflush,acpi,cdp_l3,sse,rdtscp,fsgsbase,ss,bmi1,arch_perfmon,monitor,pse,ospke,pcid,rdrand,hypervisor,flush_l1d,tsx-ctrl,arch-capabilities,vmx,avx,pge,rep_good,vpid,sdbg,sse4_2,mba,cqm_llc,est,epb,fpu,vnmi,apic,abm,spec_ctrl,model_pentium3,model_Penryn, model_Cascadelake-Server-noTSX,model_Haswell-noTSX-IBRS,model_Haswell-noTSX,model_kvm32,model_Nehalem-IBRS,model_pentium,model_Opteron_G1,model_IvyBridge-IBRS,model_Westmere,model_n270,model_qemu32,model_coreduo,model_486,model_IvyBridge,model_SandyBridge,model_Nehalem,model_Skylake-Client-noTSX-IBRS,model_pentium2,model_Westmere-IBRS,model_SandyBridge-IBRS,model_Conroe,model_qemu64,model_Opteron_G2,model_kvm64,model_Broadwell-noTSX,model_Skylake-Server-noTSX-IBRS,model_Broadwell-noTSX-IBRS,model_core2duo', 'vdsmToCpusAffinity': [1], 'version_name': 'Snow Man', 'software_version': '4.50.3.4', 'software_revision': '1', 'supportedENGINEs': ['4.2', '4.3', '4.4', '4.5'], 'clusterLevels': ['4.2', '4.3', '4.4', '4.5', '4.6', '4.7'], 'networks': {'ovirtmgmt': {'ports': ['bond0.20'], 'stp': 'off', 'iface': 'ovirtmgmt', 'bridged': True, 'addr': '192.168.2.20', 'netmask': '255.255.255.0', 'ipv4addrs': ['192.168.2.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.2.1', 'ipv6 gateway': '::', 'ipv4defaultroute': True, 'mtu': '1500', 'switch': 'legacy', 'southbound': 'bond0.20', 'vlanid': 20, 'dhcpv4': False, 'dhcpv6': False}, 'DMZ': {'ports': ['bond0.10'], 'stp': 'off', 'iface': 'DMZ', 'bridged': True, 'addr': '192.168.1.20', 'netmask': '255.255.255.0', 'ipv4addrs': ['192.168.1.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.1.1', 'ipv6gateway': '::', 'ipv4defaultroute': False, 'mtu': '1500', 'switch': 'legacy', 'southbound': 'bond0.10', 'vlanid': 10, 'dhcpv4': False, 'dhcpv6': False}, 'Migration': {'interface': 'ens1', 'iface': 'ens1', 'bridged': False, 'addr': '192.168.5.20', 'netmask': '255.255.255.0', 'ipv4addrs': ['192.168.5.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'ipv4defaultroute': False, 'mtu': '9000', 'switch': 'legacy', 'southbound': 'ens1', 'dhcpv4': False, 'dhcpv6': False}, 'SAN': {'interface': 'bond0.40', 'iface': 'bond0.40', 'bridged': False, 'addr': '192.168.4.20', 'netmask': '255.255.255.0', 'ipv4addrs': ['192.168.4.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'ipv4defaultroute': False, 'mtu': '9000', 'switch': 'legacy', 'southbound': 'bond0.40', 'vlanid': 40, 'dhcpv4': False, 'dhcpv6': False}}, 'bondings': {'bond0': {'hwaddr': 'ac:1f:6b:3c:cc:e8', 'slaves': ['eno1', 'eno2'], 'active_slave': '', 'opts': {'updelay': '1000', 'mode': '4', 'miimon': '1000', 'downdelay': '1000'}, 'ad_aggregator_id': '1', 'ad_partner_mac': '10:b3:c6:ae:a8:2d', 'switch': 'legacy', 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'speed': 2000}}, 'bridges': {'ovirtmgmt': {'ports': ['bond0.20'], 'stp': 'off', 'addr': '192.168.2.20', 'ipv4addrs': ['192.168.2.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.2.1', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'ipv4defaultroute': True, 'dhcpv4': False, 'dhcpv6': False, 'opts': {'ageing_time': '30000', 'multicast_query_use_ifaddr': '0', 'gc_timer': '17569', 'hello_time': '200', 'multicast_router': '1', 'nf_call_iptables': '0', 'group_addr': '01:80:c2:00:00:00', 'group_fwd_mask': '0x0', 'multicast_querier': '0', 'hash_max': '4096', 'tcn_timer': '0', 'vlan_protocol': '0x8100', 'vlan_stats_per_port': '0', 'root_port': '0', 'vlan_filtering': '0', 'multicast_query_interval': '12500', 'bridge_id': '8000.ac1f6b3ccce8', 'max_age': '2000', 'nf_call_arptables': '0', 'multicast_startup_query_interval': '3125', 'multicast_stats_enabled': '0', 'multicast_query_response_interval': '1000', 'topology_change': '0', 'priority': '32768', 'multicast_mld_version': '1', 'hash_elasticity': '16', 'hello_timer': '0', 'default_pvid': '1', 'root_path_cost': '0', 'multicast_igmp_version': '2', 'stp_state': '0', 'multicast_startup_query_count': '2', 'topology_change_detected': '0', 'multicast_last_member_interval': '1 00', 'topology_change_timer': '0', 'root_id': '8000.ac1f6b3ccce8', 'no_linklocal_learn': '0', 'forward_delay': '1500', 'multicast_membership_interval': '26000', 'multicast_querier_interval': '25500', 'vlan_stats_enabled': '0', 'multicast_snooping': '1', 'nf_call_ip6tables': '0', 'multicast_last_member_count': '2'}}, 'DMZ': {'ports': ['bond0.10'], 'stp': 'off', 'addr': '192.168.1.20', 'ipv4addrs': ['192.168.1.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.1.1', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'opts': {'ageing_time': '30000', 'multicast_query_use_ifaddr': '0', 'gc_timer': '8995', 'hello_time': '200', 'multicast_router': '1', 'nf_call_iptables': '0', 'group_addr': '01:80:c2:00:00:00', 'group_fwd_mask': '0x0', 'multicast_querier': '0', 'hash_max': '4096', 'tcn_timer': '0', 'vlan_protocol': '0x8100', 'vlan_stats_per_port': '0', 'root_port': '0', 'vlan_filtering': '0', 'multi cast_query_interval': '12500', 'bridge_id': '8000.ac1f6b3ccce8', 'max_age': '2000', 'nf_call_arptables': '0', 'multicast_startup_query_interval': '3125', 'multicast_stats_enabled': '0', 'multicast_query_response_interval': '1000', 'topology_change': '0', 'priority': '32768', 'multicast_mld_version': '1', 'hash_elasticity': '16', 'hello_timer': '0', 'default_pvid': '1', 'root_path_cost': '0', 'multicast_igmp_version': '2', 'stp_state': '0', 'multicast_startup_query_count': '2', 'topology_change_detected': '0', 'multicast_last_member_interval': '100', 'topology_change_timer': '0', 'root_id': '8000.ac1f6b3ccce8', 'no_linklocal_learn': '0', 'forward_delay': '1500', 'multicast_membership_interval': '26000', 'multicast_querier_interval': '25500', 'vlan_stats_enabled': '0', 'multicast_snooping': '1', 'nf_call_ip6tables': '0', 'multicast_last_member_count': '2'}}, 'virbr0': {'ports': [], 'stp': 'on', 'addr': '192.168.122.1', 'ipv4addrs': ['192.168.122.1/24'], 'ipv6addrs': [], 'ipv6autoconf' : False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'opts': {'ageing_time': '30000', 'multicast_query_use_ifaddr': '0', 'gc_timer': '22103', 'hello_time': '200', 'multicast_router': '1', 'nf_call_iptables': '0', 'group_addr': '01:80:c2:00:00:00', 'group_fwd_mask': '0x0', 'multicast_querier': '0', 'hash_max': '4096', 'tcn_timer': '0', 'vlan_protocol': '0x8100', 'vlan_stats_per_port': '0', 'root_port': '0', 'vlan_filtering': '0', 'multicast_query_interval': '12500', 'bridge_id': '8000.525400a8ef1c', 'max_age': '2000', 'nf_call_arptables': '0', 'multicast_startup_query_interval': '3125', 'multicast_stats_enabled': '0', 'multicast_query_response_interval': '1000', 'topology_change': '0', 'priority': '32768', 'multicast_mld_version': '1', 'hash_elasticity': '16', 'hello_timer': '103', 'default_pvid': '1', 'root_path_cost': '0', 'multicast_igmp_version': '2', 'stp_state': '1', 'multicast_start up_query_count': '2', 'topology_change_detected': '0', 'multicast_last_member_interval': '100', 'topology_change_timer': '0', 'root_id': '8000.525400a8ef1c', 'no_linklocal_learn': '0', 'forward_delay': '200', 'multicast_membership_interval': '26000', 'multicast_querier_interval': '25500', 'vlan_stats_enabled': '0', 'multicast_snooping': '1', 'nf_call_ip6tables': '0', 'multicast_last_member_count': '2'}}}, 'nics': {'ens1': {'hwaddr': '00:1b:21:e4:a6:f4', 'addr': '192.168.5.20', 'ipv4addrs': ['192.168.5.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'speed': 1000}, 'eno1': {'hwaddr': 'ac:1f:6b:3c:cc:e8', 'ad_aggregator_id': '1', 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'ipv4defaultroute': False, 'permhwaddr': 'ac:1f:6b:3c:cc:e8', 'dhcpv4': False, 'dhcpv 6': False, 'speed': 1000}, 'eno2': {'hwaddr': 'ac:1f:6b:3c:cc:e8', 'ad_aggregator_id': '1', 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'ipv4defaultroute': False, 'permhwaddr': 'ac:1f:6b:3c:cc:e9', 'dhcpv4': False, 'dhcpv6': False, 'speed': 1000}}, 'vlans': {'bond0.20': {'iface': 'bond0', 'vlanid': 20, 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False}, 'bond0.10': {'iface': 'bond0', 'vlanid': 10, 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False}, 'bond0.40': {'iface': 'bond0', 'vlanid': 40, 'addr': '192.168.4.20', 'ipv4addrs': ['192.168.4.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6 gateway': '::', 'mtu': '9000', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False}}, 'nameservers': ['192.168.2.16', '192.168.2.17'], 'supportsIPv6': True, 'ovnConfigured': False, 'hooks': {'after_get_caps': {'ovirt_provider_ovn_hook': {'checksum': 'a2bdefca38b96c8ddab39822cc8282bf3f67d875c4879003ffc9661826c92421'}}, 'before_device_create': {'10_ovirt_provider_ovn_hook': {'checksum': 'b9d67afb41bd86a521ada2121e3505297b3b6dfd478275ce0bd9501fcda6dafc'}}, 'before_nic_hotplug': {'10_ovirt_provider_ovn_hook': {'checksum': 'b9d67afb41bd86a521ada2121e3505297b3b6dfd478275ce0bd9501fcda6dafc'}}, 'before_vm_start': {'50_hostedengine': {'checksum': 'e5f5262cf22e06cd34e227afb27647e479351266876019a64210dbcbd2a43830'}}}, 'operatingSystem': {'release': '1.8.el8', 'version': '8.8', 'name': 'RHEL', 'pretty_name': 'Rocky Linux 8.8 (Green Obsidian)'}, 'uuid': '18e93400-808a-11ea-8000-ac1f6b3ccce8', 'packages2': {'kernel': {'version': '4.18.0', 'release': '477.27.1.e l8_8.x86_64'}, 'glusterfs-cli': {'version': '10.4', 'release': '1.el8s'}, 'librbd1': {'version': '16.2.13', 'release': '1.el8s'}, 'libvirt': {'version': '8.0.0', 'release': '19.2.module+el8.8.0+1321+70d049b2'}, 'mom': {'version': '0.6.4', 'release': '1.el8'}, 'ovirt-hosted-engine-ha': {'version': '2.5.0', 'release': '1.el8'}, 'openvswitch': {'version': '2.15', 'release': '4.el8'}, 'nmstate': {'version': '1.4.4', 'release': '2.el8_8'}, 'qemu-img': {'version': '6.2.0', 'release': '33.module+el8.8.0+1454+0b2cbfb8'}, 'qemu-kvm': {'version': '6.2.0', 'release': '33.module+el8.8.0+1454+0b2cbfb8'}, 'spice-server': {'version': '0.14.3', 'release': '4.el8'}, 'vdsm': {'version': '4.50.3.4', 'release': '1.el8'}, 'glusterfs': {'version': '10.4', 'release': '1.el8s'}, 'glusterfs-fuse': {'version': '10.4', 'release': '1.el8s'}, 'glusterfs-geo-replication': {'version': '10.4', 'release': '1.el8s'}, 'glusterfs-server': {'version': '10.4', 'release': '1.el8s'}}, 'realtimeKernel': False, 'kernelArgs' : 'BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-477.27.1.el8_8.x86_64 root=UUID=9acf17b0-558a-46d3-b21c-74c208680fcb ro crashkernel=auto resume=UUID=9c5f897c-8e34-48a7-9f06-786af3c079e9', 'nestedVirtualization': True, 'emulatedMachines': ['pc-q35-rhel8.0.0', 'pc-q35-rhel7.4.0', 'pc-i440fx-rhel7.3.0', 'pc-q35-rhel8.2.0', 'pc-q35-rhel7.3.0', 'pc-i440fx-rhel7.4.0', 'pc-q35-rhel8.4.0', 'pc-q35-rhel7.6.0', 'pc-i440fx-rhel7.5.0', 'pc-q35-rhel8.1.0', 'pc-q35-rhel8.6.0', 'pc-q35-rhel8.3.0', 'pc-q35-rhel7.5.0', 'pc-q35-rhel8.5.0', 'pc-i440fx-rhel7.1.0', 'q35', 'pc-i440fx-rhel7.2.0', 'pc', 'pc-i440fx-rhel7.6.0', 'pc-i440fx-rhel7.0.0'], 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:dc85f9619f1d', 'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:dc85f9619f1d'}], 'FC': []}, 'vmTypes': ['kvm'], 'memSize': '63805', 'reservedMem': '321', 'guestOverhead': '65', 'rngSources': ['random', 'hwrng'], 'numaNodes': {'0': {'totalMemory': '31553', 'hugepages': {'4': {'totalPages': '8077814'}, ' 2048': {'totalPages': '0'}, '1048576': {'totalPages': '0'}}, 'cpus': [0, 1, 2, 3, 4, 5, 6, 7, 16, 17, 18, 19, 20, 21, 22, 23]}, '1': {'totalMemory': '32251', 'hugepages': {'4': {'totalPages': '8256270'}, '2048': {'totalPages': '0'}, '1048576': {'totalPages': '0'}}, 'cpus': [8, 9, 10, 11, 12, 13, 14, 15, 24, 25, 26, 27, 28, 29, 30, 31]}}, 'numaNodeDistance': {'0': [10, 21], '1': [21, 10]}, 'autoNumaBalancing': 2, 'selinux': {'mode': '1'}, 'liveSnapshot': 'true', 'liveMerge': 'true', 'kdumpStatus': 1, 'deferred_preallocation': True, 'hostdevPassthrough': 'false', 'additionalFeatures': ['libgfapi_supported', 'GLUSTER_SNAPSHOT', 'GLUSTER_GEO_REPLICATION', 'GLUSTER_BRICK_MANAGEMENT'], 'hostedEngineDeployed': True, 'hugepages': [2048, 1048576], 'kernelFeatures': {'SPECTRE_V2': '(Mitigation: Enhanced IBRS, IBPB: conditional, RSB filling, PBRSB-eIBRS: SW sequence)', 'ITLB_MULTIHIT': '(KVM: Mitigation: VMX disabled)', 'MMIO_STALE_DATA': '(Mitigation: Clear CPU buffers; SMT vulnerable)', 'MDS ': '(Not affected)', 'L1TF': '(Not affected)', 'SPEC_STORE_BYPASS': '(Mitigation: Speculative Store Bypass disabled via prctl)', 'TSX_ASYNC_ABORT': '(Mitigation: TSX disabled)', 'SPECTRE_V1': '(Mitigation: usercopy/swapgs barriers and __user pointer sanitization)', 'RETBLEED': '(Mitigation: Enhanced IBRS)', 'SRBDS': '(Not affected)', 'MELTDOWN': '(Not affected)'}, 'vncEncrypted': True, 'backupEnabled': True, 'coldBackupEnabled': True, 'clearBitmapsEnabled': True, 'fipsEnabled': False, 'boot_uuid': '68e5a5ca-3dd5-4103-9613-6065a5fc74da', 'tscFrequency': '3199999000', 'tscScaling': True, 'domain_versions': [0, 2, 3, 4, 5], 'supported_block_size': {'FCP': [512], 'GLUSTERFS': [0, 512, 4096], 'ISCSI': [512], 'LOCALFS': [0, 512, 4096], 'NFS': [512], 'POSIXFS': [512]}, 'cd_change_pdiv': True, 'refresh_disk_supported': True, 'replicate_extend': True, 'measure_subchain': True, 'measure_active': True, 'mailbox_events': True, 'zerocopy_migrations': True, 'netConfigDirty': 'False', 'openstack_b inding_host_ids': {'OVIRT_PROVIDER_OVN': 'vmhost1.mydoamin.local'}, 'lastClientIface': 'ovirtmgmt'}} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:10,620+1100 INFO (jsonrpc/1) [api.host] START getHardwareInfo() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:10,621+1100 INFO (jsonrpc/1) [api.host] FINISH getHardwareInfo return={'status': {'code': 0, 'message': 'Done'}, 'info': {'systemManufacturer': 'Supermicro', 'systemProductName': 'X11DPi-N(T)', 'systemVersion': '123456789', 'systemSerialNumber': '', 'systemUUID': '0034e918-8a80-ea11-8000-ac1f6b3ccce8', 'systemFamily': 'SMC X11'}} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:10,767+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.2.28,43698, flow_id=c4e6178 (api:31) 2023-10-16 18:21:10,778+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=bab905a7-0651-41a3-a9f6-49707358a0eb (api:31) 2023-10-16 18:21:10,778+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={} from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=bab905a7-0651-41a3-a9f6-49707358a0eb (api:37) 2023-10-16 18:21:10,779+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=0f382ce1-8a94-4764-94b7-394b65b3db99 (api:31) 2023-10-16 18:21:10,779+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=0f382ce1-8a94-4764-94b7-394b65b3db99 (api:37) 2023-10-16 18:21:10,786+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698, flow_id=c4e6178 (api:37) 2023-10-16 18:21:11,032+1100 INFO (jsonrpc/6) [vds] Setting Hosted Engine HA local maintenance to False (API:1697) 2023-10-16 18:21:11,198+1100 INFO (jsonrpc/4) [vdsm.api] START connectStorageServer(domType=7, spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', conList=[{'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:/gv0', 'ipv6_enabled': 'false', 'id': '8d7b930e-a758-4010-aaf5-a230447f2761', 'user': '', 'tpgt': '1'}]) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=be7fffbe-f4ca-48f3-b1af-064268ad9eac (api:31) 2023-10-16 18:21:11,199+1100 INFO (tmap-752/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Closing client (__init__:641) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77159) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Shutdown requested (__init__:126) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77159) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,200+1100 INFO (tmap-752/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Closing client (__init__:641) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77153) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Shutdown requested (__init__:126) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77153) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,200+1100 INFO (tmap-752/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Closing client (__init__:641) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77159) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,201+1100 INFO (ioprocess/77153) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,201+1100 INFO (ioprocess/77148) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Shutdown requested (__init__:126) 2023-10-16 18:21:11,201+1100 INFO (ioprocess/77148) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,202+1100 INFO (ioprocess/77148) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194) 2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) Closing client (__init__:641) 2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57) 2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'id': '8d7b930e-a758-4010-aaf5-a230447f2761', 'status': 0}]} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=be7fffbe-f4ca-48f3-b1af-064268ad9eac (api:37) 2023-10-16 18:21:11,210+1100 INFO (ioprocess/76327) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,210+1100 INFO (ioprocess/76327) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,216+1100 INFO (jsonrpc/2) [vdsm.api] START connectStorageServer(domType=7, spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', conList=[{'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:gv1', 'ipv6_enabled': 'false', 'id': '044a086b-e94b-4e5d-a118-283ffceb8de8', 'user': '', 'tpgt': '1'}, {'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:/gv2', 'ipv6_enabled': 'false', 'id': '2d328788-ade0-4e25-b947-1c392d4f133e', 'user': '', 'tpgt': '1'}, {'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:/gv3', 'i pv6_enabled': 'false', 'id': '420b9d49-eebb-4dea-99bb-bcb89fbe2bf2', 'user': '', 'tpgt': '1'}]) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=92bf27ed-8388-4485-abc3-cd79f0d4163f (api:31) 2023-10-16 18:21:11,293+1100 INFO (jsonrpc/2) [storage.storageServer] Creating directory '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1' (storageServer:217) 2023-10-16 18:21:11,293+1100 INFO (jsonrpc/2) [storage.fileutils] Creating directory: /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1 mode: None (fileUtils:214) 2023-10-16 18:21:11,293+1100 WARN (jsonrpc/2) [storage.storageServer] Using user specified backup-volfile-servers option (storageServer:326) 2023-10-16 18:21:11,293+1100 INFO (jsonrpc/2) [storage.mount] mounting vmhosttran1.mydoamin.local:gv1 at /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1 (mount:190) 2023-10-16 18:21:11,415+1100 INFO (jsonrpc/2) [IOProcessClient] (Global) Starting client (__init__:340) 2023-10-16 18:21:11,424+1100 INFO (ioprocess/86760) [IOProcess] (Global) Starting ioprocess (__init__:465) 2023-10-16 18:21:11,518+1100 INFO (jsonrpc/2) [storage.storageServer] Creating directory '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2' (storageServer:217) 2023-10-16 18:21:11,518+1100 INFO (jsonrpc/2) [storage.fileutils] Creating directory: /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2 mode: None (fileUtils:214) 2023-10-16 18:21:11,518+1100 WARN (jsonrpc/2) [storage.storageServer] Using user specified backup-volfile-servers option (storageServer:326) 2023-10-16 18:21:11,518+1100 INFO (jsonrpc/2) [storage.mount] mounting vmhosttran1.mydoamin.local:/gv2 at /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2 (mount:190) 2023-10-16 18:21:11,662+1100 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:11,662+1100 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:11,730+1100 INFO (jsonrpc/2) [storage.storageServer] Creating directory '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3' (storageServer:217) 2023-10-16 18:21:11,730+1100 INFO (jsonrpc/2) [storage.fileutils] Creating directory: /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3 mode: None (fileUtils:214) 2023-10-16 18:21:11,730+1100 WARN (jsonrpc/2) [storage.storageServer] Using user specified backup-volfile-servers option (storageServer:326) 2023-10-16 18:21:11,730+1100 INFO (jsonrpc/2) [storage.mount] mounting vmhosttran1.mydoamin.local:/gv3 at /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3 (mount:190) 2023-10-16 18:21:11,875+1100 INFO (tmap-753/1) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting client (__init__:340) 2023-10-16 18:21:11,883+1100 INFO (tmap-753/2) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting client (__init__:340) 2023-10-16 18:21:11,885+1100 INFO (ioprocess/87016) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting ioprocess (__init__:465) 2023-10-16 18:21:11,889+1100 INFO (tmap-753/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting client (__init__:340) 2023-10-16 18:21:11,892+1100 INFO (ioprocess/87022) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting ioprocess (__init__:465) 2023-10-16 18:21:11,898+1100 INFO (ioprocess/87027) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting ioprocess (__init__:465) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain e311ddf1-7f2c-49ef-a618-050d9a2b947f from storage domain cache (sdc:194) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Closing client (__init__:641) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 from storage domain cache (sdc:194) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Closing client (__init__:641) 2023-10-16 18:21:11,904+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 4412e492-f752-4a6c-98ff-f289043618da from storage domain cache (sdc:194) 2023-10-16 18:21:11,904+1100 INFO (jsonrpc/2) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) Closing client (__init__:641) 2023-10-16 18:21:11,904+1100 INFO (ioprocess/77213) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,904+1100 INFO (ioprocess/77197) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,904+1100 INFO (ioprocess/77222) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,905+1100 INFO (ioprocess/77197) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,905+1100 INFO (ioprocess/77213) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,906+1100 INFO (ioprocess/77222) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194) 2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain e311ddf1-7f2c-49ef-a618-050d9a2b947f from storage domain cache (sdc:194) 2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 from storage domain cache (sdc:194) 2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 4412e492-f752-4a6c-98ff-f289043618da from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain e311ddf1-7f2c-49ef-a618-050d9a2b947f from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 4412e492-f752-4a6c-98ff-f289043618da from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57) 2023-10-16 18:21:11,915+1100 INFO (jsonrpc/2) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'id': '044a086b-e94b-4e5d-a118-283ffceb8de8', 'status': 0}, {'id': '2d328788-ade0-4e25-b947-1c392d4f133e', 'status': 0}, {'id': '420b9d49-eebb-4dea-99bb-bcb89fbe2bf2', 'status': 0}]} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=92bf27ed-8388-4485-abc3-cd79f0d4163f (api:37) 2023-10-16 18:21:11,924+1100 INFO (jsonrpc/3) [vdsm.api] START connectStorageServer(domType=3, spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', conList=[{'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.24', 'ipv6_enabled': 'false', 'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'user': 'cephiscsichapinitiator', 'tpgt': '3'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.25', 'ipv6_enabled': 'false', 'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'user': 'cephiscsichapinitiator', 'tpgt': '2'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.23', 'ipv6_enabled': 'false', 'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'user': 'cephiscsichapinitiator', 'tpgt': '1'}]) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=32063392-d746-458c-b143-919fbf9df35b (api:31) 2023-10-16 18:21:11,925+1100 INFO (jsonrpc/3) [storage.iscsi] Adding iscsi node for target 192.168.4.24:3260,3 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:192) 2023-10-16 18:21:12,053+1100 INFO (jsonrpc/3) [storage.iscsi] Adding iscsi node for target 192.168.4.25:3260,2 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:192) 2023-10-16 18:21:12,174+1100 INFO (jsonrpc/3) [storage.iscsi] Adding iscsi node for target 192.168.4.23:3260,1 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:192) 2023-10-16 18:21:12,293+1100 INFO (jsonrpc/3) [storage.storageServer] Log in to 3 targets using 3 workers (storageServer:600) 2023-10-16 18:21:12,294+1100 INFO (iscsi-login/0) [storage.iscsi] Logging in to iscsi target 192.168.4.24:3260,3 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw via iface default (iscsi:214) 2023-10-16 18:21:12,297+1100 INFO (iscsi-login/1) [storage.iscsi] Logging in to iscsi target 192.168.4.25:3260,2 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw via iface default (iscsi:214) 2023-10-16 18:21:12,303+1100 INFO (iscsi-login/2) [storage.iscsi] Logging in to iscsi target 192.168.4.23:3260,1 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw via iface default (iscsi:214) 2023-10-16 18:21:13,667+1100 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=a9c9c716-740c-424c-8a49-62f84192416a (api:31) 2023-10-16 18:21:13,667+1100 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=a9c9c716-740c-424c-8a49-62f84192416a (api:37) 2023-10-16 18:21:13,676+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=093dfcd6-6f98-49bf-9723-0aca86957167 (api:31) 2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=093dfcd6-6f98-49bf-9723-0aca86957167 (api:37) 2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=69a16b74-67a6-4e74-9fb5-1c738e8bf029 (api:31) 2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=69a16b74-67a6-4e74-9fb5-1c738e8bf029 (api:37) 2023-10-16 18:21:13,693+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:14,410+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=1c78fec1-ad8d-4c4a-9396-bd8bb30248e4 (api:31) 2023-10-16 18:21:14,410+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=1c78fec1-ad8d-4c4a-9396-bd8bb30248e4 (api:37) 2023-10-16 18:21:15,485+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=0f779632-8240-4d02-b43e-ed4737312c36 (api:31) 2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=0f779632-8240-4d02-b43e-ed4737312c36 (api:37) 2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=2375aa06-ec3c-4658-b775-59683a0d25de (api:31) 2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=2375aa06-ec3c-4658-b775-59683a0d25de (api:37) 2023-10-16 18:21:15,503+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:16,692+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:21:16,693+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:17,362+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=bd2bdfd6-4d5e-4e60-8cab-3d25a2f31745 (api:31) 2023-10-16 18:21:17,362+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=bd2bdfd6-4d5e-4e60-8cab-3d25a2f31745 (api:37) 2023-10-16 18:21:17,366+1100 INFO (jsonrpc/1) [vdsm.api] START startMonitoringDomain(sdUUID='ee5b0268-8141-43e0-995e-8b4586217123', hostID=1) from=::1,50158, task_id=fca171ce-9c55-4300-affa-5caa1227495e (api:31) 2023-10-16 18:21:17,367+1100 INFO (jsonrpc/1) [storage.monitor] Start monitoring ee5b0268-8141-43e0-995e-8b4586217123 (monitor:176) 2023-10-16 18:21:17,368+1100 INFO (jsonrpc/1) [vdsm.api] FINISH startMonitoringDomain return=None from=::1,50158, task_id=fca171ce-9c55-4300-affa-5caa1227495e (api:37) 2023-10-16 18:21:17,368+1100 INFO (monitor/ee5b026) [storage.storagedomaincache] Refreshing storage domain cache (resize=True) (sdc:63) 2023-10-16 18:21:17,368+1100 INFO (monitor/ee5b026) [storage.iscsi] Scanning iSCSI devices (iscsi:445) 2023-10-16 18:21:17,377+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=750f996b-757e-4ccf-b08f-e82b91a88678 (api:31) 2023-10-16 18:21:17,378+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.0', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}} from=::1,50158, task_id=750f996b-757e-4ccf-b08f-e82b91a88678 (api:37) 2023-10-16 18:21:17,489+1100 INFO (monitor/ee5b026) [storage.iscsi] Scanning iSCSI devices: 0.12 seconds (utils:373) 2023-10-16 18:21:17,490+1100 INFO (monitor/ee5b026) [storage.hba] Scanning FC devices (hba:42) 2023-10-16 18:21:17,544+1100 INFO (monitor/ee5b026) [storage.hba] Scanning FC devices: 0.05 seconds (utils:373) 2023-10-16 18:21:17,544+1100 INFO (monitor/ee5b026) [storage.multipath] Waiting until multipathd is ready (multipath:95) 2023-10-16 18:21:19,586+1100 INFO (monitor/ee5b026) [storage.multipath] Waited 2.04 seconds for multipathd (tries=2, ready=2) (multipath:122) 2023-10-16 18:21:19,586+1100 INFO (monitor/ee5b026) [storage.multipath] Resizing multipath devices (multipath:223) 2023-10-16 18:21:19,588+1100 INFO (monitor/ee5b026) [storage.multipath] Resizing multipath devices: 0.00 seconds (utils:373) 2023-10-16 18:21:19,588+1100 INFO (monitor/ee5b026) [storage.storagedomaincache] Refreshing storage domain cache: 2.22 seconds (utils:373) 2023-10-16 18:21:19,594+1100 INFO (monitor/ee5b026) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) Starting client (__init__:340) 2023-10-16 18:21:19,603+1100 INFO (ioprocess/87289) [IOProcess] (ee5b0268-8141-43e0-995e-8b4586217123) Starting ioprocess (__init__:465) 2023-10-16 18:21:20,734+1100 INFO (monitor/ee5b026) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:21:20,734+1100 INFO (monitor/ee5b026) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123/dom_md/metadata' (check:104) 2023-10-16 18:21:20,777+1100 INFO (check/loop) [storage.monitor] Domain ee5b0268-8141-43e0-995e-8b4586217123 became VALID (monitor:465) 2023-10-16 18:21:20,779+1100 INFO (event/70) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:21:22,387+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=cb5a305c-dc34-4a94-bf20-fbec9643d784 (api:31) 2023-10-16 18:21:22,387+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=cb5a305c-dc34-4a94-bf20-fbec9643d784 (api:37) 2023-10-16 18:21:23,720+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=c4472b6f-95ab-4803-a5fd-b91b006f0e59 (api:31) 2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=c4472b6f-95ab-4803-a5fd-b91b006f0e59 (api:37) 2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=74b73927-5e56-4483-9f00-d69e5798b4c8 (api:31) 2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=74b73927-5e56-4483-9f00-d69e5798b4c8 (api:37) 2023-10-16 18:21:24,421+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ce22cf89-56ab-4757-97fb-c8e10692223b (api:31) 2023-10-16 18:21:24,422+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.6', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=ce22cf89-56ab-4757-97fb-c8e10692223b (api:37) 2023-10-16 18:21:25,529+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:25,540+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=9465754e-6bf1-4328-9a45-e3b521725225 (api:31) 2023-10-16 18:21:25,540+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.8', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=9465754e-6bf1-4328-9a45-e3b521725225 (api:37) 2023-10-16 18:21:25,541+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=dab8366e-1533-4155-9635-911729326a1a (api:31) 2023-10-16 18:21:25,541+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=dab8366e-1533-4155-9635-911729326a1a (api:37) 2023-10-16 18:21:26,665+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:26,665+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:27,399+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=e69c93c1-639c-4640-b7de-23724deb52f2 (api:31) 2023-10-16 18:21:27,399+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.6', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=e69c93c1-639c-4640-b7de-23724deb52f2 (api:37) 2023-10-16 18:21:28,689+1100 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=fc8a5b61-e928-425a-bd60-36a14d9334e2 (api:31) 2023-10-16 18:21:28,689+1100 INFO (periodic/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=internal, task_id=fc8a5b61-e928-425a-bd60-36a14d9334e2 (api:37) 2023-10-16 18:21:28,741+1100 WARN (jsonrpc/1) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:28,742+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:28,744+1100 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:30,548+1100 WARN (jsonrpc/5) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:30,549+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:30,550+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:30,763+1100 INFO (monitor/ee5b026) [storage.sanlock] Acquiring host id for domain ee5b0268-8141-43e0-995e-8b4586217123 (id=1, wait=False) (clusterlock:359) 2023-10-16 18:21:31,712+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:21:31,712+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:21:32,411+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=b0a218d5-91b0-4dc8-9f4c-45825e601a3c (api:31) 2023-10-16 18:21:32,412+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=b0a218d5-91b0-4dc8-9f4c-45825e601a3c (api:37) 2023-10-16 18:21:33,697+1100 WARN (periodic/0) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:33,753+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=39f0b3ae-475f-4951-90ff-581637710dc9 (api:31) 2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=39f0b3ae-475f-4951-90ff-581637710dc9 (api:37) 2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=2fd4952e-b521-4c0a-93e0-d1186a376d20 (api:31) 2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=2fd4952e-b521-4c0a-93e0-d1186a376d20 (api:37) 2023-10-16 18:21:34,435+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=fad247c4-b608-4df8-88e6-51bc025f21cd (api:31) 2023-10-16 18:21:34,435+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.6', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=fad247c4-b608-4df8-88e6-51bc025f21cd (api:37) 2023-10-16 18:21:35,562+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:35,573+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=37f91533-90ae-4c76-a3e2-c1b8362a9494 (api:31) 2023-10-16 18:21:35,574+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.8', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=37f91533-90ae-4c76-a3e2-c1b8362a9494 (api:37) 2023-10-16 18:21:35,574+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=a25a55a8-10df-48fc-82c2-f603096e58cd (api:31) 2023-10-16 18:21:35,574+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=a25a55a8-10df-48fc-82c2-f603096e58cd (api:37) 2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:37,420+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=8270e681-279e-44a0-a9b0-b7a0110803d7 (api:31) 2023-10-16 18:21:37,420+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.6', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=8270e681-279e-44a0-a9b0-b7a0110803d7 (api:37) 2023-10-16 18:21:38,773+1100 WARN (jsonrpc/4) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:38,774+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:38,775+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:40,581+1100 WARN (jsonrpc/7) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:40,582+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:40,583+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:41,668+1100 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:41,668+1100 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:42,430+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=3be41433-ac1d-4d72-963d-2a69eef1fb26 (api:31) 2023-10-16 18:21:42,431+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=3be41433-ac1d-4d72-963d-2a69eef1fb26 (api:37) 2023-10-16 18:21:43,788+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=5ab069ab-377b-47a4-a19c-e8b2606c7453 (api:31) 2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=5ab069ab-377b-47a4-a19c-e8b2606c7453 (api:37) 2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=14c99af0-527b-444a-92df-d84724c0d215 (api:31) 2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=14c99af0-527b-444a-92df-d84724c0d215 (api:37) 2023-10-16 18:21:44,450+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=88a44308-aac3-441f-a05b-bb0809854292 (api:31) 2023-10-16 18:21:44,450+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=88a44308-aac3-441f-a05b-bb0809854292 (api:37) 2023-10-16 18:21:45,595+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=a871b766-4ae8-4c0e-bd44-2d88034cd1b7 (api:31) 2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.8', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=a871b766-4ae8-4c0e-bd44-2d88034cd1b7 (api:37) 2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=5db4e011-a522-4c96-8a33-6eba3170787f (api:31) 2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=5db4e011-a522-4c96-8a33-6eba3170787f (api:37) 2023-10-16 18:21:45,610+1100 WARN (jsonrpc/6) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155) 2023-10-16 18:21:45,610+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:46,726+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:21:46,726+1100 INFO (jsonrpc/1) [throttled] Current getAllVmStats: {} (throttledlog:87) 2023-10-16 18:21:46,726+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:47,440+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=7e1eac36-3a04-4e4f-8a20-7cb84b703d1a (api:31) 2023-10-16 18:21:47,441+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.7', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=7e1eac36-3a04-4e4f-8a20-7cb84b703d1a (api:37) 2023-10-16 18:21:48,712+1100 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=e6378dc2-3d21-4cd5-851d-5db5bc5c18d4 (api:31) 2023-10-16 18:21:48,712+1100 INFO (periodic/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=internal, task_id=e6378dc2-3d21-4cd5-851d-5db5bc5c18d4 (api:37) 2023-10-16 18:21:48,715+1100 WARN (periodic/3) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155) 2023-10-16 18:21:48,808+1100 WARN (jsonrpc/4) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:48,809+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:48,810+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:52,453+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=10828206-04bc-4cd4-b5a4-38bd0cf9dc8d (api:31) 2023-10-16 18:21:52,453+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.7', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=10828206-04bc-4cd4-b5a4-38bd0cf9dc8d (api:37) 2023-10-16 18:21:53,821+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:53,833+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=6c48b607-2157-4404-9927-fbfca271c525 (api:31) 2023-10-16 18:21:53,833+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=6c48b607-2157-4404-9927-fbfca271c525 (api:37) 2023-10-16 18:21:53,834+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=4e007b1a-3480-4a60-91c9-ccc4a2b8a552 (api:31) 2023-10-16 18:21:53,834+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=4e007b1a-3480-4a60-91c9-ccc4a2b8a552 (api:37) 2023-10-16 18:21:53,837+1100 WARN (jsonrpc/7) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155) 2023-10-16 18:21:53,837+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:54,463+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=91a5858c-775d-4172-8bd7-085a03865e4b (api:31) 2023-10-16 18:21:54,463+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=91a5858c-775d-4172-8bd7-085a03865e4b (api:37) 2023-10-16 18:21:55,639+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=20121b0e-9d04-4a74-a7b8-3bbf48d1fd09 (api:31) 2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.9', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=20121b0e-9d04-4a74-a7b8-3bbf48d1fd09 (api:37) 2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=efa8dcb2-b1cf-4ab5-ae22-7953bdf379e2 (api:31) 2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=efa8dcb2-b1cf-4ab5-ae22-7953bdf379e2 (api:37) 2023-10-16 18:21:55,653+1100 WARN (jsonrpc/6) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155) 2023-10-16 18:21:55,653+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:56,671+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:56,671+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:57,462+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=0a931307-57e8-43ed-bffa-d832f421327a (api:31) 2023-10-16 18:21:57,462+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.7', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=0a931307-57e8-43ed-bffa-d832f421327a (api:37) 2023-10-16 18:22:00,800+1100 INFO (monitor/ee5b026) [storage.sanlock] Host id 1 for domain ee5b0268-8141-43e0-995e-8b4586217123 acquired in 30 seconds (clusterlock:427) 2023-10-16 18:22:01,744+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:22:01,745+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:22:02,471+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=4813e4e9-f151-4727-9cf7-fcf87d1ffdcd (api:31) 2023-10-16 18:22:02,471+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.7', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=4813e4e9-f151-4727-9cf7-fcf87d1ffdcd (api:37) 2023-10-16 18:22:02,569+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=859cacca-c5a4-4eb9-b5a8-c28ef96b464d (api:31) 2023-10-16 18:22:02,569+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.8', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=859cacca-c5a4-4eb9-b5a8-c28ef96b464d (api:37) 2023-10-16 18:22:03,663+1100 INFO (health) [health] LVM cache hit ratio: 39.35% (hits: 170 misses: 262) (health:114) 2023-10-16 18:22:03,729+1100 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=bb148f84-5eab-4cac-b563-b89d690db0bb (api:31) 2023-10-16 18:22:03,729+1100 INFO (periodic/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=bb148f84-5eab-4cac-b563-b89d690db0bb (api:37) 2023-10-16 18:22:03,862+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:03,873+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=305776e1-ad6e-4530-ae48-0f58899b0f6e (api:31) 2023-10-16 18:22:03,873+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.1', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=305776e1-ad6e-4530-ae48-0f58899b0f6e (api:37) 2023-10-16 18:22:03,874+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=f1300145-b974-4f32-90a7-553a6cefe028 (api:31) 2023-10-16 18:22:03,874+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=f1300145-b974-4f32-90a7-553a6cefe028 (api:37) 2023-10-16 18:22:03,879+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:04,475+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ee277021-41dc-414c-9886-c01c807e5658 (api:31) 2023-10-16 18:22:04,475+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=ee277021-41dc-414c-9886-c01c807e5658 (api:37) 2023-10-16 18:22:05,682+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=d1d6d329-10cb-4439-9218-e00b797220ab (api:31) 2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.9', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=d1d6d329-10cb-4439-9218-e00b797220ab (api:37) 2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::1,50158, task_id=1e8c24fa-9d0c-49ea-8f85-f06527068e73 (api:31) 2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=1e8c24fa-9d0c-49ea-8f85-f06527068e73 (api:37) 2023-10-16 18:22:05,699+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:11,675+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:22:11,675+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:22:11,925+1100 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'jsonrpc': '2.0', 'method': 'StoragePool.connectStorageServer', 'params': {'storagepoolID': '7bfaa890-f892-4b3b-bf18-33dabc13e646', 'domainType': 3, 'connectionParams': [{'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.24', 'ipv6_enabled': 'false', 'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'user': 'cephiscsichapinitiator', 'tpgt': '3'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.25', 'ipv6_enabled': 'false', 'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'user': 'cephiscsichapinitiator', 'tpgt': '2'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.23', 'ipv6_enabled': 'false', 'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'user': 'cephiscsich apinitiator', 'tpgt': '1'}]}, 'id': 'd19073fd-9420-4dee-ac37-509a5b321b58'} at 0x7f877a5fa128> timeout=60, duration=60.00 at 0x7f877a69d550> task#=1179 at 0x7f8778933240>, traceback: File: "/usr/lib64/python3.6/threading.py", line 905, in _bootstrap self._bootstrap_inner() File: "/usr/lib64/python3.6/threading.py", line 937, in _bootstrap_inner self.run() File: "/usr/lib64/python3.6/threading.py", line 885, in run self._target(*self._args, **self._kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 243, in run ret = func(*args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 284, in _run self._execute_task() File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 298, in _execute_task task() File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 374, in __call__ self._callable() File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 253, in __call__ self._handler(self._ctx, self._req) File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 296, in _serveRequest response = self._handle_request(req, ctx) File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 338, in _handle_request res = method(**params) File: "/usr/lib/python3.6/site-packages/vdsm/rpc/Bridge.py", line 182, in _dynamicMethod result = fn(*methodArgs) File: "/usr/lib/python3.6/site-packages/vdsm/API.py", line 1117, in connectStorageServer connectionParams) File: "/usr/lib/python3.6/site-packages/vdsm/storage/dispatcher.py", line 57, in wrapper result = ctask.prepare(func, *args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 93, in wrapper return m(self, *a, **kw) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 1163, in prepare result = self._run(func, *args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 867, in _run return fn(*args, **kargs) File: "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-117>", line 2, in connectStorageServer File: "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 33, in method ret = func(*args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2164, in connectStorageServer results = storageServer.connect(domType, conList) File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 914, in connect return con_class.connect_all(connections) File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 617, in connect_all for res in login_results: File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 207, in tmap yield results.get() File: "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File: "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() (executor:346) 2023-10-16 18:22:12,654+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=837a670b-015f-44ad-9f05-d3ca615f3981 (api:31) 2023-10-16 18:22:12,654+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.8', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=837a670b-015f-44ad-9f05-d3ca615f3981 (api:37) 2023-10-16 18:22:13,908+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=399b71ba-f296-4c9c-9ead-870d40d751fe (api:31) 2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.1', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=399b71ba-f296-4c9c-9ead-870d40d751fe (api:37) 2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=0b89ae1d-b45c-4274-a63d-f5dfa0f9392e (api:31) 2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=0b89ae1d-b45c-4274-a63d-f5dfa0f9392e (api:37) 2023-10-16 18:22:13,927+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:14,488+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=8f652e74-faae-4ac9-a6cb-1c114c23cf48 (api:31) 2023-10-16 18:22:14,488+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=8f652e74-faae-4ac9-a6cb-1c114c23cf48 (api:37) 2023-10-16 18:22:15,726+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=f85241ec-5b32-4efa-b1f9-8592a9fd57d5 (api:31) 2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.9', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=f85241ec-5b32-4efa-b1f9-8592a9fd57d5 (api:37) 2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::1,50158, task_id=01d80faa-e134-42d1-bce2-28f91748a9e3 (api:31) 2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=01d80faa-e134-42d1-bce2-28f91748a9e3 (api:37) 2023-10-16 18:22:15,743+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:16,763+1100 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:22:16,763+1100 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:18,750+1100 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=468af5ce-aa49-48f1-ac7c-762cc713710a (api:31) 2023-10-16 18:22:18,750+1100 INFO (periodic/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=468af5ce-aa49-48f1-ac7c-762cc713710a (api:37) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Closing client (__init__:641) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Closing client (__init__:641) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Closing client (__init__:641) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (Global) Closing client (__init__:641) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Closing client (__init__:641) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/87027) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/87022) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/87016) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/86760) [IOProcessClient] (Global) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/77141) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,815+1100 INFO (ioprocess/87022) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:20,815+1100 INFO (ioprocess/86760) [IOProcessClient] (Global) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:20,815+1100 INFO (ioprocess/87016) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:20,816+1100 INFO (ioprocess/87027) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:20,816+1100 INFO (ioprocess/77141) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:22,741+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=62257d75-2f2b-432d-a752-e0ee3e8fd29b (api:31) 2023-10-16 18:22:22,741+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.9', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=62257d75-2f2b-432d-a752-e0ee3e8fd29b (api:37) 2023-10-16 18:22:23,954+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=78b74721-7d6c-4ed2-88f0-4fff08c4b407 (api:31) 2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.1', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=78b74721-7d6c-4ed2-88f0-4fff08c4b407 (api:37) 2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=78ff047f-a712-4dbb-99e3-25909b977799 (api:31) 2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=78ff047f-a712-4dbb-99e3-25909b977799 (api:37) 2023-10-16 18:22:23,974+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:24,503+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=0a2888a2-c8ae-49e5-89fc-a6b5bebfd77f (api:31) 2023-10-16 18:22:24,503+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=0a2888a2-c8ae-49e5-89fc-a6b5bebfd77f (api:37) 2023-10-16 18:22:25,769+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=59f0f76f-e6d4-47cf-8ce0-6b3f9a7e850d (api:31) 2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.0', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=59f0f76f-e6d4-47cf-8ce0-6b3f9a7e850d (api:37) 2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=c5576936-9a21-4b28-886e-b6cd73c90cd4 (api:31) 2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=c5576936-9a21-4b28-886e-b6cd73c90cd4 (api:37) 2023-10-16 18:22:25,786+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:26,678+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:22:26,678+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:31,782+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:22:31,783+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:22:32,825+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=66cc2698-9c5a-4c9d-803b-d89bd549f4f0 (api:31) 2023-10-16 18:22:32,826+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.0', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=66cc2698-9c5a-4c9d-803b-d89bd549f4f0 (api:37) 2023-10-16 18:22:33,771+1100 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=c267fd6f-633d-4c6d-94ad-b5cb5bb8d02d (api:31) 2023-10-16 18:22:33,772+1100 INFO (periodic/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=c267fd6f-633d-4c6d-94ad-b5cb5bb8d02d (api:37) 2023-10-16 18:22:33,998+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=2654a62d-b4f9-425c-8442-b4c418c4f542 (api:31) 2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.2', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=2654a62d-b4f9-425c-8442-b4c418c4f542 (api:37) 2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=c02547e7-20f6-437a-84b4-4fdf98cfd003 (api:31) 2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=c02547e7-20f6-437a-84b4-4fdf98cfd003 (api:37) 2023-10-16 18:22:34,014+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:34,515+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=667f9b48-b2ab-4113-ae86-1b586bbdf5d6 (api:31) 2023-10-16 18:22:34,516+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=667f9b48-b2ab-4113-ae86-1b586bbdf5d6 (api:37) 2023-10-16 18:22:35,812+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=c04a2aa5-1f9d-4c87-b51f-34eaeeedb8e2 (api:31) 2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.0', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=c04a2aa5-1f9d-4c87-b51f-34eaeeedb8e2 (api:37) 2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=f6ac211c-50fa-481a-aeaa-01779cc54412 (api:31) 2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=f6ac211c-50fa-481a-aeaa-01779cc54412 (api:37) 2023-10-16 18:22:35,830+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:37,110+1100 INFO (jsonrpc/2) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:37,111+1100 INFO (jsonrpc/2) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:37,111+1100 INFO (jsonrpc/2) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:37,111+1100 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:41,681+1100 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:22:41,681+1100 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:22:42,911+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=c1a8bf5a-4aa0-4c85-a9fa-8c240a07daa9 (api:31) 2023-10-16 18:22:42,911+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=c1a8bf5a-4aa0-4c85-a9fa-8c240a07daa9 (api:37) 2023-10-16 18:22:44,040+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=d7eb5985-0470-4aea-bd23-937c1a03da72 (api:31) 2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.2', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=d7eb5985-0470-4aea-bd23-937c1a03da72 (api:37) 2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::1,50158, task_id=a32bf2e0-f35c-46f4-b582-a28de56e1960 (api:31) 2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=a32bf2e0-f35c-46f4-b582-a28de56e1960 (api:37) 2023-10-16 18:22:44,059+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:44,528+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=68c2e209-d655-437d-add5-769c0a4932ba (api:31) 2023-10-16 18:22:44,528+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=68c2e209-d655-437d-add5-769c0a4932ba (api:37) 2023-10-16 18:22:45,855+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=b93990c2-b263-4c89-ba4b-ab0e962f6ce8 (api:31) 2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.0', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=b93990c2-b263-4c89-ba4b-ab0e962f6ce8 (api:37) 2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=444bef31-e827-478f-a2fc-16e18dc0b9a6 (api:31) 2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=444bef31-e827-478f-a2fc-16e18dc0b9a6 (api:37) 2023-10-16 18:22:45,873+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:46,786+1100 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:22:46,787+1100 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:48,792+1100 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=50b7c32e-202c-40ec-b484-5eb3f45f7f28 (api:31) 2023-10-16 18:22:48,792+1100 INFO (periodic/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '8.0', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=50b7c32e-202c-40ec-b484-5eb3f45f7f28 (api:37) 2023-10-16 18:22:52,996+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=28687ab3-2149-42d2-92aa-d28cc5ad8d96 (api:31) 2023-10-16 18:22:52,996+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=28687ab3-2149-42d2-92aa-d28cc5ad8d96 (api:37) 2023-10-16 18:22:54,086+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=96f37ad8-6f60-4c9b-8986-cad8c4a5bdf3 (api:31) 2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.2', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=96f37ad8-6f60-4c9b-8986-cad8c4a5bdf3 (api:37) 2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=aa6dc50d-c1c9-4a8c-bf9d-b7ca456f73a6 (api:31) 2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=aa6dc50d-c1c9-4a8c-bf9d-b7ca456f73a6 (api:37) 2023-10-16 18:22:54,106+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:54,533+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ada1e11f-23e9-4704-a8c4-27bfecfd9974 (api:31) 2023-10-16 18:22:54,533+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=ada1e11f-23e9-4704-a8c4-27bfecfd9974 (api:37) 2023-10-16 18:22:55,899+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=a0dd0ff4-5a0a-43cf-8612-ac5ff8e953a8 (api:31) 2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=a0dd0ff4-5a0a-43cf-8612-ac5ff8e953a8 (api:37) 2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=9211755e-e9f9-4c7b-b0dc-8a7689a7884a (api:31) 2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=9211755e-e9f9-4c7b-b0dc-8a7689a7884a (api:37) 2023-10-16 18:22:55,916+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:56,684+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:22:56,684+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:22:57,141+1100 INFO (jsonrpc/5) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:57,141+1100 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:57,141+1100 INFO (jsonrpc/5) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:57,142+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:01,805+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:23:01,805+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:23:03,079+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=95612d72-c18c-4dfe-87f2-f8501f713aca (api:31) 2023-10-16 18:23:03,079+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.2', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=95612d72-c18c-4dfe-87f2-f8501f713aca (api:37) 2023-10-16 18:23:03,813+1100 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=c193e1ec-73f7-4858-aa76-fb9f7d977517 (api:31) 2023-10-16 18:23:03,814+1100 INFO (periodic/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=c193e1ec-73f7-4858-aa76-fb9f7d977517 (api:37) 2023-10-16 18:23:04,130+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:04,141+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=1df7dacf-ebad-4493-ad11-a7ce10fbf851 (api:31) 2023-10-16 18:23:04,141+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.3', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=1df7dacf-ebad-4493-ad11-a7ce10fbf851 (api:37) 2023-10-16 18:23:04,142+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=cbc0b187-25a2-43aa-bfa5-9cf54f675e08 (api:31) 2023-10-16 18:23:04,142+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=cbc0b187-25a2-43aa-bfa5-9cf54f675e08 (api:37) 2023-10-16 18:23:04,147+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:04,541+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=77176abe-b0e4-4f22-bce7-2f26f571743d (api:31) 2023-10-16 18:23:04,541+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=77176abe-b0e4-4f22-bce7-2f26f571743d (api:37) 2023-10-16 18:23:05,942+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:05,954+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=99736bd8-b5b6-4e29-a345-cec8db30f976 (api:31) 2023-10-16 18:23:05,954+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=99736bd8-b5b6-4e29-a345-cec8db30f976 (api:37) 2023-10-16 18:23:05,955+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=286f88d9-373c-42c6-b990-b70d2eb73137 (api:31) 2023-10-16 18:23:05,955+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=286f88d9-373c-42c6-b990-b70d2eb73137 (api:37) 2023-10-16 18:23:05,960+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:11,687+1100 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:11,687+1100 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:11,925+1100 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'jsonrpc': '2.0', 'method': 'StoragePool.connectStorageServer', 'params': {'storagepoolID': '7bfaa890-f892-4b3b-bf18-33dabc13e646', 'domainType': 3, 'connectionParams': [{'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.24', 'ipv6_enabled': 'false', 'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'user': 'cephiscsichapinitiator', 'tpgt': '3'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.25', 'ipv6_enabled': 'false', 'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'user': 'cephiscsichapinitiator', 'tpgt': '2'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.23', 'ipv6_enabled': 'false', 'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'user': 'cephiscsich apinitiator', 'tpgt': '1'}]}, 'id': 'd19073fd-9420-4dee-ac37-509a5b321b58'} at 0x7f877a5fa128> timeout=60, duration=120.00 at 0x7f877a69d550> task#=1179 at 0x7f8778933240>, traceback: File: "/usr/lib64/python3.6/threading.py", line 905, in _bootstrap self._bootstrap_inner() File: "/usr/lib64/python3.6/threading.py", line 937, in _bootstrap_inner self.run() File: "/usr/lib64/python3.6/threading.py", line 885, in run self._target(*self._args, **self._kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 243, in run ret = func(*args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 284, in _run self._execute_task() File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 298, in _execute_task task() File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 374, in __call__ self._callable() File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 253, in __call__ self._handler(self._ctx, self._req) File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 296, in _serveRequest response = self._handle_request(req, ctx) File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 338, in _handle_request res = method(**params) File: "/usr/lib/python3.6/site-packages/vdsm/rpc/Bridge.py", line 182, in _dynamicMethod result = fn(*methodArgs) File: "/usr/lib/python3.6/site-packages/vdsm/API.py", line 1117, in connectStorageServer connectionParams) File: "/usr/lib/python3.6/site-packages/vdsm/storage/dispatcher.py", line 57, in wrapper result = ctask.prepare(func, *args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 93, in wrapper return m(self, *a, **kw) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 1163, in prepare result = self._run(func, *args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 867, in _run return fn(*args, **kargs) File: "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-117>", line 2, in connectStorageServer File: "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 33, in method ret = func(*args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2164, in connectStorageServer results = storageServer.connect(domType, conList) File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 914, in connect return con_class.connect_all(connections) File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 617, in connect_all for res in login_results: File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 207, in tmap yield results.get() File: "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File: "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() (executor:346) 2023-10-16 18:23:13,165+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=0cb1a91c-f096-4daa-bb08-ec407a86433a (api:31) 2023-10-16 18:23:13,165+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.3', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=0cb1a91c-f096-4daa-bb08-ec407a86433a (api:37) 2023-10-16 18:23:14,173+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:14,185+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=fbf38d2b-bc3e-49ad-afd9-de55861129db (api:31) 2023-10-16 18:23:14,185+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.3', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=fbf38d2b-bc3e-49ad-afd9-de55861129db (api:37) 2023-10-16 18:23:14,185+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=3e936d0b-7879-45c8-89d4-ae6bbae466a2 (api:31) 2023-10-16 18:23:14,186+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=3e936d0b-7879-45c8-89d4-ae6bbae466a2 (api:37) 2023-10-16 18:23:14,192+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:14,554+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=316fa299-920a-4c26-a829-9e93647ea804 (api:31) 2023-10-16 18:23:14,554+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=316fa299-920a-4c26-a829-9e93647ea804 (api:37) 2023-10-16 18:23:15,986+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=c7094132-9870-44f0-8274-54527984da18 (api:31) 2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=c7094132-9870-44f0-8274-54527984da18 (api:37) 2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=61b7e2c2-106d-42ff-b6cd-e91e557af60d (api:31) 2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=61b7e2c2-106d-42ff-b6cd-e91e557af60d (api:37) 2023-10-16 18:23:16,003+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:16,823+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:23:16,823+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:18,834+1100 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=5086291d-9172-41f1-9db9-7011a6a34abb (api:31) 2023-10-16 18:23:18,834+1100 INFO (periodic/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=5086291d-9172-41f1-9db9-7011a6a34abb (api:37) 2023-10-16 18:23:23,249+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=bf9f76c3-bead-4087-947b-ebd063ebab6d (api:31) 2023-10-16 18:23:23,249+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.3', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=bf9f76c3-bead-4087-947b-ebd063ebab6d (api:37) 2023-10-16 18:23:24,217+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=3296546b-846b-48f8-b2f3-9fa13e06864e (api:31) 2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.3', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=3296546b-846b-48f8-b2f3-9fa13e06864e (api:37) 2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=77d3abd5-6cf4-4a3f-874e-0a0a1adcf3d5 (api:31) 2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=77d3abd5-6cf4-4a3f-874e-0a0a1adcf3d5 (api:37) 2023-10-16 18:23:24,236+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:24,566+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=12ad1f12-b845-4107-9bc0-326453b99721 (api:31) 2023-10-16 18:23:24,566+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=12ad1f12-b845-4107-9bc0-326453b99721 (api:37) 2023-10-16 18:23:26,030+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=cf9b3b2a-8182-440a-82b8-0c2dd0e33cee (api:31) 2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=cf9b3b2a-8182-440a-82b8-0c2dd0e33cee (api:37) 2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=20a2b03c-eee3-4d1d-a47e-cc6e581df61f (api:31) 2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=20a2b03c-eee3-4d1d-a47e-cc6e581df61f (api:37) 2023-10-16 18:23:26,047+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:26,373+1100 INFO (iscsi-login/0) [storage.iscsi] Removing iscsi node for target 192.168.4.24:3260,3 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:227) 2023-10-16 18:23:26,388+1100 INFO (iscsi-login/1) [storage.iscsi] Removing iscsi node for target 192.168.4.25:3260,2 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:227) 2023-10-16 18:23:26,436+1100 ERROR (iscsi-login/0) [storage.storageServer] Could not login to target (storageServer:607) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 301, in node_login "-p", portal, "-l"]) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 104, in run_cmd out = commands.run(cmd, sudo=True) File "/usr/lib/python3.6/site-packages/vdsm/common/commands.py", line 85, in run raise cmdutils.Error(args, p.returncode, out, err) vdsm.common.cmdutils.Error: Command ['/sbin/iscsiadm', '-m', 'node', '-T', 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', '-I', 'default', '-p', '192.168.4.24:3260,3', '-l'] failed with rc=11 out=b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260]\n' err=b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 604, in iscsi_login iscsi.loginToIscsiNode(con.iface, con.target) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsi.py", line 216, in loginToIscsiNode iscsiadm.node_login(iface.name, target.address, target.iqn) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 321, in node_login raise IscsiNodeError(e.rc, e.out, e.err) vdsm.storage.iscsiadm.IscsiNodeError: (11, b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260]\n', b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n') 2023-10-16 18:23:26,497+1100 ERROR (iscsi-login/1) [storage.storageServer] Could not login to target (storageServer:607) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 301, in node_login "-p", portal, "-l"]) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 104, in run_cmd out = commands.run(cmd, sudo=True) File "/usr/lib/python3.6/site-packages/vdsm/common/commands.py", line 85, in run raise cmdutils.Error(args, p.returncode, out, err) vdsm.common.cmdutils.Error: Command ['/sbin/iscsiadm', '-m', 'node', '-T', 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', '-I', 'default', '-p', '192.168.4.25:3260,2', '-l'] failed with rc=11 out=b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260]\n' err=b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 604, in iscsi_login iscsi.loginToIscsiNode(con.iface, con.target) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsi.py", line 216, in loginToIscsiNode iscsiadm.node_login(iface.name, target.address, target.iqn) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 321, in node_login raise IscsiNodeError(e.rc, e.out, e.err) vdsm.storage.iscsiadm.IscsiNodeError: (11, b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260]\n', b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n') 2023-10-16 18:23:26,514+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Refreshing storage domain cache (resize=True) (sdc:63) 2023-10-16 18:23:26,514+1100 INFO (jsonrpc/3) [storage.iscsi] Scanning iSCSI devices (iscsi:445) 2023-10-16 18:23:26,609+1100 INFO (jsonrpc/3) [storage.iscsi] Scanning iSCSI devices: 0.09 seconds (utils:373) 2023-10-16 18:23:26,609+1100 INFO (jsonrpc/3) [storage.hba] Scanning FC devices (hba:42) 2023-10-16 18:23:26,663+1100 INFO (jsonrpc/3) [storage.hba] Scanning FC devices: 0.05 seconds (utils:373) 2023-10-16 18:23:26,663+1100 INFO (jsonrpc/3) [storage.multipath] Waiting until multipathd is ready (multipath:95) 2023-10-16 18:23:26,690+1100 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:26,690+1100 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:28,705+1100 INFO (jsonrpc/3) [storage.multipath] Waited 2.04 seconds for multipathd (tries=2, ready=2) (multipath:122) 2023-10-16 18:23:28,705+1100 INFO (jsonrpc/3) [storage.multipath] Resizing multipath devices (multipath:223) 2023-10-16 18:23:28,706+1100 INFO (jsonrpc/3) [storage.multipath] Resizing multipath devices: 0.00 seconds (utils:373) 2023-10-16 18:23:28,706+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Refreshing storage domain cache: 2.19 seconds (utils:373) 2023-10-16 18:23:28,796+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Removing domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 from storage domain cache (sdc:194) 2023-10-16 18:23:28,796+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57) 2023-10-16 18:23:28,796+1100 INFO (jsonrpc/3) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'status': 0}, {'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'status': 465}, {'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'status': 465}]} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=32063392-d746-458c-b143-919fbf9df35b (api:37) 2023-10-16 18:23:28,797+1100 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer took more than 1.00 seconds to succeed: 136.87 (__init__:305) 2023-10-16 18:23:28,847+1100 INFO (jsonrpc/5) [vdsm.api] START connectStoragePool(spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', hostID=1, msdUUID='ee5b0268-8141-43e0-995e-8b4586217123', masterVersion=1, domainsMap={'e311ddf1-7f2c-49ef-a618-050d9a2b947f': 'active', 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': 'active', '30775056-38ef-483a-889c-70453e9aa5f9': 'active', 'ee5b0268-8141-43e0-995e-8b4586217123': 'active', '3e33b7ce-7279-4765-a203-cca37eff8cb0': 'active', '4412e492-f752-4a6c-98ff-f289043618da': 'active', '0dde39a4-42e4-4fba-8068-e63e42dd3334': 'active', 'd44ec7f2-3629-4110-93a5-7d8d57230efc': 'active'}) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=22550292-d106-45d1-b5b7-17ff11f6f8a7 (api:31) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagepoolmemorybackend] new storage pool master version 1 and domains map {'e311ddf1-7f2c-49ef-a618-050d9a2b947f': 'Active', 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': 'Active', '30775056-38ef-483a-889c-70453e9aa5f9': 'Active', 'ee5b0268-8141-43e0-995e-8b4586217123': 'Active', '3e33b7ce-7279-4765-a203-cca37eff8cb0': 'Active', '4412e492-f752-4a6c-98ff-f289043618da': 'Active', '0dde39a4-42e4-4fba-8068-e63e42dd3334': 'Active', 'd44ec7f2-3629-4110-93a5-7d8d57230efc': 'Active'} (spbackends:417) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagepool] updating pool 7bfaa890-f892-4b3b-bf18-33dabc13e646 backend from type NoneType instance 0x7f87cc236b70 to type StoragePoolMemoryBackend instance 0x7f877a7cdcc8 (sp:149) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagepool] Connect host #1 to the storage pool 7bfaa890-f892-4b3b-bf18-33dabc13e646 with master domain: ee5b0268-8141-43e0-995e-8b4586217123 (ver = 1) (sp:699) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Clearing storage domain cache (sdc:182) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Refreshing storage domain cache (resize=True) (sdc:63) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.iscsi] Scanning iSCSI devices (iscsi:445) 2023-10-16 18:23:28,946+1100 INFO (jsonrpc/5) [storage.iscsi] Scanning iSCSI devices: 0.10 seconds (utils:373) 2023-10-16 18:23:28,946+1100 INFO (jsonrpc/5) [storage.hba] Scanning FC devices (hba:42) 2023-10-16 18:23:28,996+1100 INFO (jsonrpc/5) [storage.hba] Scanning FC devices: 0.05 seconds (utils:373) 2023-10-16 18:23:28,996+1100 INFO (jsonrpc/5) [storage.multipath] Waiting until multipathd is ready (multipath:95) 2023-10-16 18:23:31,033+1100 INFO (jsonrpc/5) [storage.multipath] Waited 2.04 seconds for multipathd (tries=2, ready=2) (multipath:122) 2023-10-16 18:23:31,033+1100 INFO (jsonrpc/5) [storage.multipath] Resizing multipath devices (multipath:223) 2023-10-16 18:23:31,034+1100 INFO (jsonrpc/5) [storage.multipath] Resizing multipath devices: 0.00 seconds (utils:373) 2023-10-16 18:23:31,034+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Refreshing storage domain cache: 2.19 seconds (utils:373) 2023-10-16 18:23:31,035+1100 INFO (tmap-757/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Starting client (__init__:340) 2023-10-16 18:23:31,043+1100 INFO (tmap-757/1) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting client (__init__:340) 2023-10-16 18:23:31,045+1100 INFO (ioprocess/87928) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Starting ioprocess (__init__:465) 2023-10-16 18:23:31,052+1100 INFO (tmap-757/2) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting client (__init__:340) 2023-10-16 18:23:31,054+1100 INFO (ioprocess/87936) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting ioprocess (__init__:465) 2023-10-16 18:23:31,059+1100 INFO (tmap-757/3) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting client (__init__:340) 2023-10-16 18:23:31,062+1100 INFO (ioprocess/87940) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting ioprocess (__init__:465) 2023-10-16 18:23:31,068+1100 INFO (ioprocess/87948) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting ioprocess (__init__:465) 2023-10-16 18:23:31,842+1100 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:23:31,843+1100 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:23:32,194+1100 INFO (jsonrpc/5) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:23:32,195+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 0dde39a4-42e4-4fba-8068-e63e42dd3334 (monitor:176) 2023-10-16 18:23:32,195+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 3e33b7ce-7279-4765-a203-cca37eff8cb0 (monitor:176) 2023-10-16 18:23:32,201+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring e311ddf1-7f2c-49ef-a618-050d9a2b947f (monitor:176) 2023-10-16 18:23:32,206+1100 INFO (jsonrpc/5) [storage.monitor] Attaching monitor for ee5b0268-8141-43e0-995e-8b4586217123 to the pool (monitor:172) 2023-10-16 18:23:32,207+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 4412e492-f752-4a6c-98ff-f289043618da (monitor:176) 2023-10-16 18:23:32,208+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring fa8d922a-df3c-4f5b-adf6-a77c3689c397 (monitor:176) 2023-10-16 18:23:32,210+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 30775056-38ef-483a-889c-70453e9aa5f9 (monitor:176) 2023-10-16 18:23:32,211+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154) 2023-10-16 18:23:32,218+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring d44ec7f2-3629-4110-93a5-7d8d57230efc (monitor:176) 2023-10-16 18:23:32,219+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154) 2023-10-16 18:23:32,230+1100 INFO (monitor/3e33b7c) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Starting client (__init__:340) 2023-10-16 18:23:32,234+1100 INFO (jsonrpc/5) [storage.storagepool] Creating pool directory '/rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646' (sp:1323) 2023-10-16 18:23:32,234+1100 INFO (jsonrpc/5) [storage.fileutils] Creating directory: /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646 mode: None (fileUtils:214) 2023-10-16 18:23:32,239+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154) 2023-10-16 18:23:32,260+1100 INFO (monitor/e311ddf) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Starting client (__init__:340) 2023-10-16 18:23:32,263+1100 INFO (ioprocess/87989) [IOProcess] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Starting ioprocess (__init__:465) 2023-10-16 18:23:32,268+1100 INFO (monitor/4412e49) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) Starting client (__init__:340) 2023-10-16 18:23:32,271+1100 INFO (ioprocess/88003) [IOProcess] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Starting ioprocess (__init__:465) 2023-10-16 18:23:32,278+1100 INFO (ioprocess/88012) [IOProcess] (4412e492-f752-4a6c-98ff-f289043618da) Starting ioprocess (__init__:465) 2023-10-16 18:23:32,323+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.11 seconds (utils:373) 2023-10-16 18:23:32,323+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',) 2023-10-16 18:23:32,323+1100 INFO (monitor/fa8d922) [storage.monitor] Domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 became INVALID (monitor:465) 2023-10-16 18:23:32,334+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.11 seconds (utils:373) 2023-10-16 18:23:32,334+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',) 2023-10-16 18:23:32,334+1100 INFO (monitor/3077505) [storage.monitor] Domain 30775056-38ef-483a-889c-70453e9aa5f9 became INVALID (monitor:465) 2023-10-16 18:23:32,339+1100 WARN (jsonrpc/5) [storage.lvm] Marked vgs=[0dde39a4-42e4-4fba-8068-e63e42dd3334] as Unreadable due to reload failure (lvm:679) 2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.storagepool] Creating domain metadata directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md' (sp:1354) 2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md mode: None (fileUtils:214) 2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.storagepool] Creating domain images directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images' (sp:1357) 2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images mode: None (fileUtils:214) 2023-10-16 18:23:32,340+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.10 seconds (utils:373) 2023-10-16 18:23:32,340+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',) 2023-10-16 18:23:32,340+1100 INFO (monitor/d44ec7f) [storage.monitor] Domain d44ec7f2-3629-4110-93a5-7d8d57230efc became INVALID (monitor:465) 2023-10-16 18:23:32,345+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/0dde39a4-42e4-4fba-8068-e63e42dd3334 (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/0dde39a4-42e4-4fba-8068-e63e42dd3334 (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/e311ddf1-7f2c-49ef-a618-050d9a2b947f (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/e311ddf1-7f2c-49ef-a618-050d9a2b947f (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/3e33b7ce-7279-4765-a203-cca37eff8cb0 (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/3e33b7ce-7279-4765-a203-cca37eff8cb0 (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/4412e492-f752-4a6c-98ff-f289043618da (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/4412e492-f752-4a6c-98ff-f289043618da (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/ee5b0268-8141-43e0-995e-8b4586217123 (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/ee5b0268-8141-43e0-995e-8b4586217123 (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/mastersd (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/mastersd (sp:1295) 2023-10-16 18:23:32,383+1100 INFO (jsonrpc/5) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=22550292-d106-45d1-b5b7-17ff11f6f8a7 (api:37) 2023-10-16 18:23:32,384+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect took more than 1.00 seconds to succeed: 3.54 (__init__:305) 2023-10-16 18:23:32,385+1100 INFO (monitor/0dde39a) [storage.lvm] Activating lvs: vg=0dde39a4-42e4-4fba-8068-e63e42dd3334 lvs=['ids'] (lvm:1839) 2023-10-16 18:23:32,390+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.2.28,43698, flow_id=706708f9 (api:31) 2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=ac4e0a4f-b2d6-4d28-b765-6ec18d5ecb94 (api:31) 2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.2', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '0.1', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actua l': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '0.1', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '0.1', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=ac4e0a4f-b2d6-4d28-b765-6ec18d5ecb94 (api:37) 2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=c006784d-0604-489e-a8a8-8be4c44e3bec (api:31) 2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=c006784d-0604-489e-a8a8-8be4c44e3bec (api:37) 2023-10-16 18:23:32,413+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698, flow_id=706708f9 (api:37) 2023-10-16 18:23:32,572+1100 INFO (monitor/0dde39a) [storage.lvm] Activating lvs: vg=0dde39a4-42e4-4fba-8068-e63e42dd3334 lvs=['leases'] (lvm:1839) 2023-10-16 18:23:32,771+1100 INFO (monitor/0dde39a) [storage.lvm] Activating lvs: vg=0dde39a4-42e4-4fba-8068-e63e42dd3334 lvs=['metadata', 'inbox', 'outbox', 'xleases', 'master'] (lvm:1839) 2023-10-16 18:23:32,975+1100 INFO (monitor/0dde39a) [storage.blocksd] Setting up domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 (blockSD:973) 2023-10-16 18:23:33,056+1100 INFO (monitor/0dde39a) [storage.check] Start checking '/dev/0dde39a4-42e4-4fba-8068-e63e42dd3334/metadata' (check:104) 2023-10-16 18:23:33,143+1100 INFO (monitor/0dde39a) [storage.monitor] Domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 became VALID (monitor:465) 2023-10-16 18:23:33,144+1100 INFO (monitor/0dde39a) [storage.sanlock] Acquiring host id for domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 (id=1, wait=False) (clusterlock:359) 2023-10-16 18:23:33,145+1100 INFO (event/81) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:23:33,145+1100 INFO (event/82) [storage.storagedomainmanifest] Creating domain images directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images' (blockSD:766) 2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images mode: None (fileUtils:214) 2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.storagedomainmanifest] Creating domain special volumes directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md' (blockSD:771) 2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md mode: None (fileUtils:214) 2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.storagepool] Linking /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/0dde39a4-42e4-4fba-8068-e63e42dd3334 (sp:1278) 2023-10-16 18:23:33,338+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=50cf6835-08a1-4b31-8a61-6e18b85fb86e (api:31) 2023-10-16 18:23:33,338+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.1', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=50cf6835-08a1-4b31-8a61-6e18b85fb86e (api:37) 2023-10-16 18:23:33,470+1100 INFO (monitor/3e33b7c) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:23:33,471+1100 INFO (monitor/3e33b7c) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0/dom_md/metadata' (check:104) 2023-10-16 18:23:33,474+1100 INFO (monitor/4412e49) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:23:33,475+1100 INFO (monitor/4412e49) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da/dom_md/metadata' (check:104) 2023-10-16 18:23:33,519+1100 INFO (check/loop) [storage.monitor] Domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 became VALID (monitor:465) 2023-10-16 18:23:33,521+1100 INFO (event/84) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:23:33,522+1100 INFO (check/loop) [storage.monitor] Domain 4412e492-f752-4a6c-98ff-f289043618da became VALID (monitor:465) 2023-10-16 18:23:33,523+1100 INFO (event/85) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/3e33b7ce-7279-4765-a203-cca37eff8cb0 (sp:1278) 2023-10-16 18:23:33,524+1100 INFO (monitor/e311ddf) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:23:33,524+1100 INFO (monitor/e311ddf) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f/dom_md/metadata' (check:104) 2023-10-16 18:23:33,528+1100 INFO (event/87) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:23:33,534+1100 INFO (event/88) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/4412e492-f752-4a6c-98ff-f289043618da (sp:1278) 2023-10-16 18:23:33,542+1100 INFO (check/loop) [storage.monitor] Domain e311ddf1-7f2c-49ef-a618-050d9a2b947f became VALID (monitor:465) 2023-10-16 18:23:33,543+1100 INFO (event/90) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:23:33,543+1100 INFO (event/91) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/e311ddf1-7f2c-49ef-a618-050d9a2b947f (sp:1278) 2023-10-16 18:23:33,856+1100 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=67318041-086e-4992-a2c3-dc0b7e5ebd4e (api:31) 2023-10-16 18:23:33,856+1100 INFO (periodic/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '0.7', 'delay': '0.00174884', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00116006', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00604746', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00091358', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '1.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '1.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '1.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=internal, task_id=67318041-086e-4992-a2c3-dc0b7e5ebd4e (api:37) 2023-10-16 18:23:34,263+1100 INFO (jsonrpc/3) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=8cf2bf3c-b4ac-4f24-9179-29054ce22a46 (api:31) 2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '1.1', 'delay': '0.00174884', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.8', 'delay': '0.00116006', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.7', 'delay': '0.00604746', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.8', 'delay': '0.00091358', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '2.0', 'delay': '0', 'valid': False, 'version': - 1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=8cf2bf3c-b4ac-4f24-9179-29054ce22a46 (api:37) 2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::1,50158, task_id=dfd9aafe-1cbf-445f-afa9-6b19287c9955 (api:31) 2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=dfd9aafe-1cbf-445f-afa9-6b19287c9955 (api:37) 2023-10-16 18:23:34,280+1100 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:34,572+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=efc38225-ad10-4a67-97d8-1f7f2854456d (api:31) 2023-10-16 18:23:34,572+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.4', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=efc38225-ad10-4a67-97d8-1f7f2854456d (api:37) 2023-10-16 18:23:36,073+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:36,084+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=e1664ede-6677-4756-97d9-d1b033ebb9b6 (api:31) 2023-10-16 18:23:36,085+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00174884', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.6', 'delay': '0.00116006', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.5', 'delay': '0.00604746', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.6', 'delay': '0.00091358', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.8', 'delay': '0', 'valid': False, 'version': - 1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.8', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=e1664ede-6677-4756-97d9-d1b033ebb9b6 (api:37) 2023-10-16 18:23:36,085+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=592b6419-5402-4f74-83ae-b088d9b4b978 (api:31) 2023-10-16 18:23:36,085+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=592b6419-5402-4f74-83ae-b088d9b4b978 (api:37) 2023-10-16 18:23:36,090+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:41,693+1100 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:41,694+1100 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:42,324+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154) 2023-10-16 18:23:42,335+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154) 2023-10-16 18:23:42,341+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154) 2023-10-16 18:23:42,423+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.09 seconds (utils:373) 2023-10-16 18:23:42,423+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',) 2023-10-16 18:23:42,426+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.10 seconds (utils:373) 2023-10-16 18:23:42,426+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',) 2023-10-16 18:23:42,428+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=139f761e-b9f7-4451-8864-f96a498176ed (api:31) 2023-10-16 18:23:42,428+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.2', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=139f761e-b9f7-4451-8864-f96a498176ed (api:37) 2023-10-16 18:23:42,432+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.09 seconds (utils:373) 2023-10-16 18:23:42,432+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',) 2023-10-16 18:23:43,488+1100 INFO (monitor/4412e49) [storage.sanlock] Acquiring host id for domain 4412e492-f752-4a6c-98ff-f289043618da (id=1, wait=False) (clusterlock:359) 2023-10-16 18:23:43,488+1100 INFO (monitor/3e33b7c) [storage.sanlock] Acquiring host id for domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 (id=1, wait=False) (clusterlock:359) 2023-10-16 18:23:43,541+1100 INFO (monitor/e311ddf) [storage.sanlock] Acquiring host id for domain e311ddf1-7f2c-49ef-a618-050d9a2b947f (id=1, wait=False) (clusterlock:359) 2023-10-16 18:23:44,308+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=0d42d65f-0534-47c0-aaf9-7dbac3067a91 (api:31) 2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '1.2', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.8', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.7', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.8', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=0d42d65f-0534-47c0-aaf9-7dbac3067a91 (api:37) 2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=dad656d9-6a05-443b-82ca-c4b7632334dd (api:31) 2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=dad656d9-6a05-443b-82ca-c4b7632334dd (api:37) 2023-10-16 18:23:44,328+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:44,587+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ae63dc53-56a4-4d9a-8bc1-63c6f3c2f160 (api:31) 2023-10-16 18:23:44,587+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.4', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=ae63dc53-56a4-4d9a-8bc1-63c6f3c2f160 (api:37) 2023-10-16 18:23:46,096+1100 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:46,107+1100 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, task_id=9d13808b-d90d-449b-9f9c-85e91b946eeb (api:31) 2023-10-16 18:23:46,107+1100 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.5', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::ffff:192.168.2.28,43698, task_id=9d13808b-d90d-449b-9f9c-85e91b946eeb (api:37) 2023-10-16 18:23:46,108+1100 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, task_id=53f94ff6-5532-41b5-ae7a-3271108c20d2 (api:31) 2023-10-16 18:23:46,108+1100 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, task_id=53f94ff6-5532-41b5-ae7a-3271108c20d2 (api:37) 2023-10-16 18:23:46,113+1100 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:46,121+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=19f3fa69-f932-4edd-96f6-76ba56449cf2 (api:31) 2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=19f3fa69-f932-4edd-96f6-76ba56449cf2 (api:37) 2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=78db57fe-4ab9-4256-b239-2fb8fa2c6a61 (api:31) 2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=78db57fe-4ab9-4256-b239-2fb8fa2c6a61 (api:37) 2023-10-16 18:23:46,137+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:46,860+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:23:46,860+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:23:47,219+1100 INFO (jsonrpc/4) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:47,220+1100 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:47,220+1100 INFO (jsonrpc/4) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:47,220+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:48,878+1100 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=56a338bf-2d23-4dfe-8065-e8e5acc5736a (api:31) 2023-10-16 18:23:48,878+1100 INFO (periodic/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.7', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '5.7', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '5.4', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '5.3', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '5.3', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '6.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '6.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '6.4', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=internal, task_id=56a338bf-2d23-4dfe-8065-e8e5acc5736a (api:37) 2023-10-16 18:23:52,424+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154) 2023-10-16 18:23:52,428+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154) 2023-10-16 18:23:52,433+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154) 2023-10-16 18:23:52,516+1100 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=a77c47c6-03ba-40fd-9bfa-29375ea32ae4 (api:31) 2023-10-16 18:23:52,516+1100 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=a77c47c6-03ba-40fd-9bfa-29375ea32ae4 (api:37) 2023-10-16 18:23:52,542+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.12 seconds (utils:373) 2023-10-16 18:23:52,543+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',) 2023-10-16 18:23:52,546+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.11 seconds (utils:373) 2023-10-16 18:23:52,546+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',) 2023-10-16 18:23:52,551+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.12 seconds (utils:373) 2023-10-16 18:23:52,551+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',) 2023-10-16 18:23:53,353+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=72f4cb81-c54d-4153-9425-0a5766308ed4 (api:31) 2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.1', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '0.2', 'delay': '0.00305946', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '9.9', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '9.8', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '9.8', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '0.8', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '0.8', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '0.8', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=72f4cb81-c54d-4153-9425-0a5766308ed4 (api:37) 2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=65572c4a-c042-49c8-834d-dee8d879896a (api:31) 2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=65572c4a-c042-49c8-834d-dee8d879896a (api:37) 2023-10-16 18:23:53,373+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:54,605+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=1ba074d9-5e35-4527-93ca-0423ed3187f8 (api:31) 2023-10-16 18:23:54,605+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.4', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=1ba074d9-5e35-4527-93ca-0423ed3187f8 (api:37) 2023-10-16 18:23:56,164+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=cdb6b35e-ab67-48e1-a210-2f4ab737a00f (api:31) 2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00305946', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.7', 'delay': '0.00109704', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.6', 'delay': '0.00924556', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.7', 'delay': '0.000824828', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=cdb6b35e-ab67-48e1-a210-2f4ab737a00f (api:37) 2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=9099f061-8efd-4c2e-9669-ff848b5aa791 (api:31) 2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=9099f061-8efd-4c2e-9669-ff848b5aa791 (api:37) 2023-10-16 18:23:56,182+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:56,696+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:56,697+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:24:01,111+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:24:01,123+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, task_id=49598180-2b1c-493c-a4ff-f49a3e988b19 (api:31) 2023-10-16 18:24:01,123+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.4', 'delay': '0.00137321', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '8.0', 'delay': '0.00305946', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '7.6', 'delay': '0.00109704', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '7.6', 'delay': '0.00924556', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '7.6', 'delay': '0.000824828', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '8.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '8.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '8.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::ffff:192.168.2.28,43698, task_id=49598180-2b1c-493c-a4ff-f49a3e988b19 (api:37) 2023-10-16 18:24:01,124+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, task_id=ed16e0f7-5985-4608-94dd-28a587cc92d2 (api:31) 2023-10-16 18:24:01,124+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, task_id=ed16e0f7-5985-4608-94dd-28a587cc92d2 (api:37) 2023-10-16 18:24:01,130+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:24:01,879+1100 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:24:01,879+1100 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:24:02,543+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154) 2023-10-16 18:24:02,547+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154) 2023-10-16 18:24:02,552+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154) 2023-10-16 18:24:02,610+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=6b15ba3d-d2fe-4568-b235-25ded159d77c (api:31) 2023-10-16 18:24:02,610+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00137321', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=6b15ba3d-d2fe-4568-b235-25ded159d77c (api:37) 2023-10-16 18:24:02,744+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.20 seconds (utils:373) 2023-10-16 18:24:02,744+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',) 2023-10-16 18:24:02,755+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.21 seconds (utils:373) 2023-10-16 18:24:02,755+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',) 2023-10-16 18:24:02,755+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.20 seconds (utils:373) 2023-10-16 18:24:02,755+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',) 2023-10-16 18:24:02,828+1100 INFO (jsonrpc/3) [vds] Setting Hosted Engine HA local maintenance to True (API:1697) ~~~ Further info can be provided if required (I don't know what else to include) Thanks in advance for any and all help.