- Users - oVirt List Archives

Request to get get an account
by ReFea_IM[e] Original 17 Oct '23

17 Oct '23

Hello, I would like to participate in the oVirt translation project. How can this be done? Thank you.

2 1

[org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for <JsonRpcResponse id: "..."
by nicolas＠devels.es 17 Oct '23

17 Oct '23

Hi, We're running oVirt 4.5.4, recently we had a problem due to /var partition being full, and we realized the /var/log/ovirt-engine/engine.log was pretty huge, approximately 16GB in size, despite it's configured to rotate daily. Checking the log, we have a lot (about 53000 at the time of this mail) of WARN events in this form: 2023-10-17 03:34:51,347+01 WARN [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for <JsonRpcResponse id: "d6cc2209-3126-48e1-97c8-97ca84d00f3a" result: [{"statusTime":"3329996823","status":"Up","vmId":"0fad0aa4-ec91-4beb-82a3-5641f0da674f","vmName":"SYTW-Backend-aluXXXXXXXXX","vmType":"kvm","kvmEnable":"true","acpiEnable":"true","elapsedTime":"285568","monitorResponse":"0","clientIp":"","timeOffset":"2","cpuUser":"-0.04","cpuSys": "0.40","cpuUsage":"12369410000000","cpuActual":true,"network":{"vnet3":{"macAddr":"00:1a:4a:4d:cc:d5","name":"vnet3","speed":"1000","state" ... This is just a chunk of the full line, which is actually pretty big as it contains a lot of parameters. So about 53.000 times this long line makes the log huge. Why are we getting such a big amount of these WARNings and what can we do to solve them? Thanks.

1 0

Host reports about one of the Active Storage Domains as Problematic
by Matthew J Black 16 Oct '23

16 Oct '23

Hi All, I'm hoping someone(s) can point me in the correct direction to fix this. The Architecture +++++++++++++ 3 identical VM-Hosts (vmhost{1-3}.mydomain.local): - Rocky Linux 8.8 - 4 local gluster volumes - 4 iscsi volumes provided by 3-node Ceph Cluster (also running Rocky v8.8) - 3 NICS: - One Gluster Transfer - One Ceph Data Transfer - One General Data access / OVirt management Each node of the Ceph Cluster is running an iscsi gateway, and each vmhost connects to each iscsi gateway Everything is working AOK, except vmhost1 complains that it can't reach 3 of the 4 iscsi volumes/domains. I've looked in the engine.log and vdsm.log but can't figure out how to fix things, and I don't know where else to look. engine.log (extract) +++++++++++++++ ~~~ 2023-10-16 18:21:08,892+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-27) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Lock Acquired to object 'EngineLock:{exclusiveLocks='[8e0e99ab-c212-41c4-8318-acd34acb6289=VDS]', sharedLocks=''}' 2023-10-16 18:21:09,179+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2023-10-16 18:21:09,179+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Before acquiring lock in order to prevent monitoring for host 'vmhost1.mydomain.local' from data-center 'my_datacentre' 2023-10-16 18:21:09,179+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Lock acquired, from now a monitoring of host will be skipped for host 'vmhost1.mydomain.local' from data-center 'my_datacentre' 2023-10-16 18:21:09,181+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] START, SetVdsStatusVDSCommand(HostName = vmhost1.mydomain.local, SetVdsStatusVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 732b1116 2023-10-16 18:21:09,245+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] FINISH, SetVdsStatusVDSCommand, return: , log id: 732b1116 2023-10-16 18:21:09,248+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Activate host finished. Lock released. Monitoring can run now for host 'vmhost1.mydomain.local' from data-center 'my_datacentre' 2023-10-16 18:21:09,314+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] EVENT_ID: VDS_ACTIVATE(16), Activation of host vmhost1.mydomain.local initiated by admin@internal-authz. 2023-10-16 18:21:09,373+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Lock freed to object 'EngineLock:{exclusiveLocks='[8e0e99ab-c212-41c4-8318-acd34acb6289=VDS]', sharedLocks=''}' 2023-10-16 18:21:10,620+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [] START, GetHardwareInfoAsyncVDSCommand(HostName = vmhost1.mydomain.local, VdsIdAndVdsVDSCommandParametersBase:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', vds='Host[vmhost1.mydomain.local,8e0e99ab-c212-41c4-8318-acd34acb6289]'}), log id: 3dac178f 2023-10-16 18:21:10,620+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [] FINISH, GetHardwareInfoAsyncVDSCommand, return: , log id: 3dac178f 2023-10-16 18:21:10,688+11 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [44be9d7f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS 2023-10-16 18:21:10,690+11 INFO [org.ovirt.engine.core.bll.UpdateClusterCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] Lock Acquired to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[]'}' 2023-10-16 18:21:10,703+11 INFO [org.ovirt.engine.core.bll.UpdateClusterCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] Running command: UpdateClusterCommand internal: true. Entities affected : ID: c892710d-8611-4146-aea2-05d910f1829b Type: ClusterAction group EDIT_CLUSTER_CONFIGURATION with role type ADMIN 2023-10-16 18:21:10,761+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] EVENT_ID: SYSTEM_UPDATE_CLUSTER(835), Host cluster my_cluster was updated by system 2023-10-16 18:21:10,761+11 INFO [org.ovirt.engine.core.bll.UpdateClusterCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] Lock freed to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[]'}' 2023-10-16 18:21:10,965+11 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [5f0caae0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 7bfaa890-f892-4b3b-bf18-33dabc13e646 Type: StoragePool 2023-10-16 18:21:11,032+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [5f0caae0] START, SetHaMaintenanceModeVDSCommand(HostName = vmhost1.mydomain.local, SetHaMaintenanceModeVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289'}), log id: 630966ec 2023-10-16 18:21:11,034+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [5f0caae0] FINISH, SetHaMaintenanceModeVDSCommand, return: , log id: 630966ec 2023-10-16 18:21:11,170+11 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 7bfaa890-f892-4b3b-bf18-33dabc13e646 Type: StoragePool 2023-10-16 18:21:11,196+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] START, ConnectStorageServerVDSCommand(HostName = vmhost1.mydomain.local, StorageServerConnectionManagementVDSParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='8d7b930e-a758-4010-aaf5-a230447f2761', connection='vmhosttran1.mydomain.local:/gv0', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local:vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 30a928ed 2023-10-16 18:21:11,211+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] FINISH, ConnectStorageServerVDSCommand, return: {8d7b930e-a758-4010-aaf5-a230447f2761=0}, log id: 30a928ed 2023-10-16 18:21:11,211+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] START, ConnectStorageServerVDSCommand(HostName = vmhost1.mydomain.local, StorageServerConnectionManagementVDSParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='044a086b-e94b-4e5d-a118-283ffceb8de8', connection='vmhosttran1.mydomain.local:gv1', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local:vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='2d328788-ade0-4e25-b947-1c392d4f133e', connection='vmhosttran1.mydomain.local:/gv2', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local :vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='420b9d49-eebb-4dea-99bb-bcb89fbe2bf2', connection='vmhosttran1.mydomain.local:/gv3', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local:vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 6aaeee49 2023-10-16 18:21:11,664+11 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [] Fetched 0 VMs from VDS '8e0e99ab-c212-41c4-8318-acd34acb6289' 2023-10-16 18:21:11,917+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] FINISH, ConnectStorageServerVDSCommand, return: {044a086b-e94b-4e5d-a118-283ffceb8de8=0, 2d328788-ade0-4e25-b947-1c392d4f133e=0, 420b9d49-eebb-4dea-99bb-bcb89fbe2bf2=0}, log id: 6aaeee49 2023-10-16 18:21:11,920+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] START, ConnectStorageServerVDSCommand(HostName = vmhost1.mydomain.local, StorageServerConnectionManagementVDSParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', storageType='ISCSI', connectionList='[StorageServerConnections:{id='78f16782-9a06-45f2-92e5-d1ca84a8c67e', connection='192.168.4.24', iqn='iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='868d7c6a-c933-4475-86ce-ab1359d6f4c3', connection='192.168.4.25', iqn='iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='20a52c08-a4c5-4b07-8ecc-bec0e82e582a', connection='192.168.4.23', iqn='iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 4e953edb 2023-10-16 18:21:19,245+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='TPM'}), log id: 694529cf 2023-10-16 18:21:19,247+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@a9992e5, log id: 694529cf 2023-10-16 18:21:19,247+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='NVRAM'}), log id: 786d585b 2023-10-16 18:21:19,248+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@1cf19aa, log id: 786d585b 2023-10-16 18:21:19,248+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='TPM'}), log id: 4104430d 2023-10-16 18:21:19,250+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@7ccef72, log id: 4104430d 2023-10-16 18:21:19,250+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='NVRAM'}), log id: 15f1c050 2023-10-16 18:21:19,251+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@6c8a990d, log id: 15f1c050 2023-10-16 18:21:34,401+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='TPM'}), log id: 3f508f9e 2023-10-16 18:21:34,403+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@5690567a, log id: 3f508f9e 2023-10-16 18:21:34,403+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='NVRAM'}), log id: 270bce0b 2023-10-16 18:21:34,404+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@2a7c9b44, log id: 270bce0b 2023-10-16 18:21:34,404+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='TPM'}), log id: 11ebddea 2023-10-16 18:21:34,406+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@459c7a2b, log id: 11ebddea 2023-10-16 18:21:34,406+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='NVRAM'}), log id: 3c1e9f05 2023-10-16 18:21:34,408+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@72f69579, log id: 3c1e9f05 2023-10-16 18:21:49,594+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='TPM'}), log id: 6a366a0f 2023-10-16 18:21:49,597+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@95edf66, log id: 6a366a0f 2023-10-16 18:21:49,655+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='TPM'}), log id: 40d156 2023-10-16 18:21:49,656+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@3f11df90, log id: 40d156 2023-10-16 18:23:28,802+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] FINISH, ConnectStorageServerVDSCommand, return: {78f16782-9a06-45f2-92e5-d1ca84a8c67e=465, 868d7c6a-c933-4475-86ce-ab1359d6f4c3=465, 20a52c08-a4c5-4b07-8ecc-bec0e82e582a=0}, log id: 4e953edb 2023-10-16 18:23:28,811+11 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] Host 'vmhost1.mydomain.local' storage connection was succeeded 2023-10-16 18:23:28,846+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] START, ConnectStoragePoolVDSCommand(HostName = vmhost1.mydomain.local, ConnectStoragePoolVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', vdsId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', masterVersion='1'}), log id: 3b47a386 2023-10-16 18:23:28,846+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Executing with domain map: {e311ddf1-7f2c-49ef-a618-050d9a2b947f=active, fa8d922a-df3c-4f5b-adf6-a77c3689c397=active, 30775056-38ef-483a-889c-70453e9aa5f9=active, ee5b0268-8141-43e0-995e-8b4586217123=active, 3e33b7ce-7279-4765-a203-cca37eff8cb0=active, 4412e492-f752-4a6c-98ff-f289043618da=active, 0dde39a4-42e4-4fba-8068-e63e42dd3334=active, d44ec7f2-3629-4110-93a5-7d8d57230efc=active} 2023-10-16 18:23:32,385+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 3b47a386 2023-10-16 18:23:32,418+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '0dde39a4-42e4-4fba-8068-e63e42dd3334:iscsi-vmimages' report isn't an actual report 2023-10-16 18:23:32,419+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '3e33b7ce-7279-4765-a203-cca37eff8cb0:gluster-generaldata' report isn't an actual report 2023-10-16 18:23:32,419+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain 'e311ddf1-7f2c-49ef-a618-050d9a2b947f:gluster-vmimages' report isn't an actual report 2023-10-16 18:23:32,419+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '4412e492-f752-4a6c-98ff-f289043618da:gluster-isos' report isn't an actual report 2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain 'fa8d922a-df3c-4f5b-adf6-a77c3689c397:iscsi-databases' was reported with error code '358' 2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '30775056-38ef-483a-889c-70453e9aa5f9:iscsi-generaldata' was reported with error code '358' 2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain 'd44ec7f2-3629-4110-93a5-7d8d57230efc:iscsi-backups' was reported with error code '358' 2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Storage Domain 'iscsi-databases' of pool 'my_datacentre' is in problem in host 'vmhost1.mydomain.local' 2023-10-16 18:23:32,420+11 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Storage Domain 'iscsi-generaldata' of pool 'my_datacentre' is in problem in host 'vmhost1.mydomain.local' 2023-10-16 18:23:32,420+11 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Storage Domain 'iscsi-backups' of pool 'my_datacentre' is in problem in host 'vmhost1.mydomain.local' 2023-10-16 18:23:32,454+11 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] EVENT_ID: VDS_STORAGE_VDS_STATS_FAILED(189), Host vmhost1.mydomain.local reports about one of the Active Storage Domains as Problematic. 2023-10-16 18:23:32,564+11 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS 2023-10-16 18:23:32,566+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] START, SetVdsStatusVDSCommand(HostName = vmhost1.mydomain.local, SetVdsStatusVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 992c9e 2023-10-16 18:23:32,566+11 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] Clearing domains data for host vmhost1.mydomain.local 2023-10-16 18:23:32,619+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] FINISH, SetVdsStatusVDSCommand, return: , log id: 992c9e 2023-10-16 18:23:32,689+11 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Host vmhost1.mydomain.local cannot access the Storage Domain(s) iscsi-databases, iscsi-generaldata, iscsi-backups attached to the Data Center my_datacentre. Setting Host state to Non-Operational. 2023-10-16 18:23:32,789+11 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] EVENT_ID: VDS_ALERT_FENCE_TEST_FAILED(9,001), Power Management test failed for Host vmhost1.mydomain.local.There is no other host in the data center that can be used to test the power management settings. 2023-10-16 18:23:32,856+11 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Failed to connect Host vmhost1.mydomain.local to Storage Pool my_datacentre 2023-10-16 18:23:33,016+11 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [f7474e7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS 2023-10-16 18:23:33,018+11 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [f7474e7] Host 'vmhost1.mydomain.local'(8e0e99ab-c212-41c4-8318-acd34acb6289) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. 2023-10-16 18:23:33,019+11 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [f7474e7] Received first domain report for host vmhost1.mydomain.local 2023-10-16 18:24:02,559+11 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2023-10-16 18:24:02,561+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] START, SetVdsStatusVDSCommand(HostName = vmhost1.mydomain.local, SetVdsStatusVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', status='PreparingForMaintenance', nonOperationalReason='NONE', stopSpmFailureLogged='true', maintenanceReason='null'}), log id: 51d95bef 2023-10-16 18:24:02,592+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] FINISH, SetVdsStatusVDSCommand, return: , log id: 51d95bef 2023-10-16 18:24:02,799+11 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS 2023-10-16 18:24:02,827+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] START, SetHaMaintenanceModeVDSCommand(HostName = vmhost1.mydomain.local, SetHaMaintenanceModeVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289'}), log id: 63f61978 2023-10-16 18:24:02,832+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] FINISH, SetHaMaintenanceModeVDSCommand, return: , log id: 63f61978 2023-10-16 18:24:02,914+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] EVENT_ID: USER_VDS_MAINTENANCE_WITHOUT_REASON(620), Host vmhost1.mydomain.local was switched to Maintenance mode by admin@internal-authz. ~~~ vdsm.log (extract - from vmhost1) ++++++++++++++++++++++++++ ~~~ 2023-10-16 18:21:10,069+1100 INFO (jsonrpc/7) [api.host] START getCapabilities() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:10,410+1100 ERROR (jsonrpc/7) [root] Error getting managedvolume connector info: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc=1 out=b\'\' err=b\'Traceback (most recent call last):\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 91, in <module>\\nfrom importlib import resources as _importlib_resources\\nImportError: cannot import name \\\'resources\\\'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File "/usr/libexec/vdsm/managedvolume-helper", line 26, in <module>\\nfrom os_brick.initiator import connector\\n File "/usr/lib/python3.6/site-packages/os_brick/initiator/connector.py", line 27, in <module>\\nfrom oslo_concurrency import lockutils\\n File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 29, in <module>\\nfrom oslo_config imp ort cfg\\n File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 45, in <module>\\nfrom oslo_config import types\\n File "/usr/lib/python3.6/site-packages/oslo_config/types.py", line 29, in <module>\\nimport netaddr\\n File "/usr/lib/python3.6/site-packages/netaddr/__init__.py", line 18, in <module>\\nfrom netaddr.core import (AddrConversionError, AddrFormatError,\\n File "/usr/lib/python3.6/site-packages/netaddr/core.py", line 11, in <module>\\nfrom netaddr.compat import _callable, _iter_dict_keys\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 93, in <module>\\nimport importlib_resources as _importlib_resources\\nModuleNotFoundError: No module named \\\'importlib_resources\\\'\\n\'',) (caps:165) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/host/caps.py", line 161, in get caps["connector_info"] = managedvolume.connector_info() File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 63, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 78, in connector_info return run_helper("connector_info") File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 184, in run_helper sub_cmd, cmd_input=cmd_input) File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 38, in __call__ return callMethod() File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 36, in <lambda> **kwargs) File "<string>", line 2, in managedvolume_run_helper File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod raise convert_to_error(kind, result) vdsm.storage.exception.ManagedVolumeHelperFailed: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc=1 out=b\'\' err=b\'Traceback (most recent call last):\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 91, in <module>\\nfrom importlib import resources as _importlib_resources\\nImportError: cannot import name \\\'resources\\\'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File "/usr/libexec/vdsm/managedvolume-helper", line 26, in <module>\\nfrom os_brick.initiator import connector\\n File "/usr/lib/python3.6/site-packages/os_brick/initiator/connector.py", line 27, in <module>\\nfrom oslo_concurrency import lockutils\\n File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 29, in <module>\\nfrom oslo_config import cfg\\n File "/usr/lib/python3.6/site-packag es/oslo_config/cfg.py", line 45, in <module>\\nfrom oslo_config import types\\n File "/usr/lib/python3.6/site-packages/oslo_config/types.py", line 29, in <module>\\nimport netaddr\\n File "/usr/lib/python3.6/site-packages/netaddr/__init__.py", line 18, in <module>\\nfrom netaddr.core import (AddrConversionError, AddrFormatError,\\n File "/usr/lib/python3.6/site-packages/netaddr/core.py", line 11, in <module>\\nfrom netaddr.compat import _callable, _iter_dict_keys\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 93, in <module>\\nimport importlib_resources as _importlib_resources\\nModuleNotFoundError: No module named \\\'importlib_resources\\\'\\n\'',) 2023-10-16 18:21:10,600+1100 INFO (jsonrpc/7) [root] /usr/libexec/vdsm/hooks/after_get_caps/ovirt_provider_ovn_hook: rc=0 err=b'' (hooks:106) 2023-10-16 18:21:10,601+1100 INFO (jsonrpc/7) [api.host] FINISH getCapabilities return={'status': {'code': 0, 'message': 'Done'}, 'info': {'kvmEnabled': 'true', 'cpuCores': '16', 'cpuThreads': '32', 'cpuSockets': '2', 'onlineCpus': '0,1,2,3,4,5,6,7,16,17,18,19,20,21,22,23,8,9,10,11,12,13,14,15,24,25,26,27,28,29,30,31', 'cpuTopology': [{'cpu_id': 0, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 0}, {'cpu_id': 1, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 1}, {'cpu_id': 2, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 2}, {'cpu_id': 3, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 3}, {'cpu_id': 4, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 4}, {'cpu_id': 5, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 5}, {'cpu_id': 6, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 6}, {'cpu_id': 7, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 7}, {'cpu_id': 16, 'numa_cell_id': 0, 'socket_id ': 0, 'die_id': 0, 'core_id': 0}, {'cpu_id': 17, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 1}, {'cpu_id': 18, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 2}, {'cpu_id': 19, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 3}, {'cpu_id': 20, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 4}, {'cpu_id': 21, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 5}, {'cpu_id': 22, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 6}, {'cpu_id': 23, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 7}, {'cpu_id': 8, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 0}, {'cpu_id': 9, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 1}, {'cpu_id': 10, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 2}, {'cpu_id': 11, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 3}, {'cpu_id': 12, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 4}, {'cpu_id': 13, 'numa_cell_id' : 1, 'socket_id': 1, 'die_id': 0, 'core_id': 5}, {'cpu_id': 14, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 6}, {'cpu_id': 15, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 7}, {'cpu_id': 24, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 0}, {'cpu_id': 25, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 1}, {'cpu_id': 26, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 2}, {'cpu_id': 27, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 3}, {'cpu_id': 28, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 4}, {'cpu_id': 29, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 5}, {'cpu_id': 30, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 6}, {'cpu_id': 31, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 7}], 'cpuSpeed': '4000.000', 'cpuModel': 'Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz', 'cpuFlags': 'pku,clflushopt,avx2,nopl,pni,pln,dca,smep,pat,art,cx8,msr,pts,arch_capa bilities,xsave,ht,lm,smap,de,mce,pse36,cx16,skip-l1dfl-vmentry,cpuid,bts,lahf_lm,avx512vl,tsc_deadline_timer,bmi2,md-clear,tm2,cmov,sep,sse4_1,invtsc,md_clear,intel_pt,amd-ssbd,mpx,ssse3,rdt_a,cpuid_fault,x2apic,mmx,avx512dq,dtes64,avx512cd,vme,pclmulqdq,cqm_occup_llc,xsavec,ibpb,umip,pschange-mc-no,mtrr,avx512f,stibp,clwb,xsaves,f16c,invpcid_single,avx512bw,arat,pdpe1gb,ibrs,ept,tm,invpcid,cqm_mbm_total,ibrs-all,3dnowprefetch,popcnt,cqm,rdctl-no,xsaveopt,pdcm,aperfmperf,cat_l3,xgetbv1,xtpr,syscall,smx,pbe,sse2,amd-stibp,pebs,xtopology,tsc_adjust,mca,pae,ida,nonstop_tsc,tpr_shadow,ssbd,nx,tsc,dtherm,aes,fxsr,fma,cqm_mbm_local,movbe,erms,intel_ppin,mds-no,ds_cpl,rdseed,dts,ibrs_enhanced,constant_tsc,ept_ad,flexpriority,avx512_vnni,adx,clflush,acpi,cdp_l3,sse,rdtscp,fsgsbase,ss,bmi1,arch_perfmon,monitor,pse,ospke,pcid,rdrand,hypervisor,flush_l1d,tsx-ctrl,arch-capabilities,vmx,avx,pge,rep_good,vpid,sdbg,sse4_2,mba,cqm_llc,est,epb,fpu,vnmi,apic,abm,spec_ctrl,model_pentium3,model_Penryn, model_Cascadelake-Server-noTSX,model_Haswell-noTSX-IBRS,model_Haswell-noTSX,model_kvm32,model_Nehalem-IBRS,model_pentium,model_Opteron_G1,model_IvyBridge-IBRS,model_Westmere,model_n270,model_qemu32,model_coreduo,model_486,model_IvyBridge,model_SandyBridge,model_Nehalem,model_Skylake-Client-noTSX-IBRS,model_pentium2,model_Westmere-IBRS,model_SandyBridge-IBRS,model_Conroe,model_qemu64,model_Opteron_G2,model_kvm64,model_Broadwell-noTSX,model_Skylake-Server-noTSX-IBRS,model_Broadwell-noTSX-IBRS,model_core2duo', 'vdsmToCpusAffinity': [1], 'version_name': 'Snow Man', 'software_version': '4.50.3.4', 'software_revision': '1', 'supportedENGINEs': ['4.2', '4.3', '4.4', '4.5'], 'clusterLevels': ['4.2', '4.3', '4.4', '4.5', '4.6', '4.7'], 'networks': {'ovirtmgmt': {'ports': ['bond0.20'], 'stp': 'off', 'iface': 'ovirtmgmt', 'bridged': True, 'addr': '192.168.2.20', 'netmask': '255.255.255.0', 'ipv4addrs': ['192.168.2.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.2.1', 'ipv6 gateway': '::', 'ipv4defaultroute': True, 'mtu': '1500', 'switch': 'legacy', 'southbound': 'bond0.20', 'vlanid': 20, 'dhcpv4': False, 'dhcpv6': False}, 'DMZ': {'ports': ['bond0.10'], 'stp': 'off', 'iface': 'DMZ', 'bridged': True, 'addr': '192.168.1.20', 'netmask': '255.255.255.0', 'ipv4addrs': ['192.168.1.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.1.1', 'ipv6gateway': '::', 'ipv4defaultroute': False, 'mtu': '1500', 'switch': 'legacy', 'southbound': 'bond0.10', 'vlanid': 10, 'dhcpv4': False, 'dhcpv6': False}, 'Migration': {'interface': 'ens1', 'iface': 'ens1', 'bridged': False, 'addr': '192.168.5.20', 'netmask': '255.255.255.0', 'ipv4addrs': ['192.168.5.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'ipv4defaultroute': False, 'mtu': '9000', 'switch': 'legacy', 'southbound': 'ens1', 'dhcpv4': False, 'dhcpv6': False}, 'SAN': {'interface': 'bond0.40', 'iface': 'bond0.40', 'bridged': False, 'addr': '192.168.4.20', 'netmask': '255.255.255.0', 'ipv4addrs': ['192.168.4.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'ipv4defaultroute': False, 'mtu': '9000', 'switch': 'legacy', 'southbound': 'bond0.40', 'vlanid': 40, 'dhcpv4': False, 'dhcpv6': False}}, 'bondings': {'bond0': {'hwaddr': 'ac:1f:6b:3c:cc:e8', 'slaves': ['eno1', 'eno2'], 'active_slave': '', 'opts': {'updelay': '1000', 'mode': '4', 'miimon': '1000', 'downdelay': '1000'}, 'ad_aggregator_id': '1', 'ad_partner_mac': '10:b3:c6:ae:a8:2d', 'switch': 'legacy', 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'speed': 2000}}, 'bridges': {'ovirtmgmt': {'ports': ['bond0.20'], 'stp': 'off', 'addr': '192.168.2.20', 'ipv4addrs': ['192.168.2.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.2.1', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'ipv4defaultroute': True, 'dhcpv4': False, 'dhcpv6': False, 'opts': {'ageing_time': '30000', 'multicast_query_use_ifaddr': '0', 'gc_timer': '17569', 'hello_time': '200', 'multicast_router': '1', 'nf_call_iptables': '0', 'group_addr': '01:80:c2:00:00:00', 'group_fwd_mask': '0x0', 'multicast_querier': '0', 'hash_max': '4096', 'tcn_timer': '0', 'vlan_protocol': '0x8100', 'vlan_stats_per_port': '0', 'root_port': '0', 'vlan_filtering': '0', 'multicast_query_interval': '12500', 'bridge_id': '8000.ac1f6b3ccce8', 'max_age': '2000', 'nf_call_arptables': '0', 'multicast_startup_query_interval': '3125', 'multicast_stats_enabled': '0', 'multicast_query_response_interval': '1000', 'topology_change': '0', 'priority': '32768', 'multicast_mld_version': '1', 'hash_elasticity': '16', 'hello_timer': '0', 'default_pvid': '1', 'root_path_cost': '0', 'multicast_igmp_version': '2', 'stp_state': '0', 'multicast_startup_query_count': '2', 'topology_change_detected': '0', 'multicast_last_member_interval': '1 00', 'topology_change_timer': '0', 'root_id': '8000.ac1f6b3ccce8', 'no_linklocal_learn': '0', 'forward_delay': '1500', 'multicast_membership_interval': '26000', 'multicast_querier_interval': '25500', 'vlan_stats_enabled': '0', 'multicast_snooping': '1', 'nf_call_ip6tables': '0', 'multicast_last_member_count': '2'}}, 'DMZ': {'ports': ['bond0.10'], 'stp': 'off', 'addr': '192.168.1.20', 'ipv4addrs': ['192.168.1.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.1.1', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'opts': {'ageing_time': '30000', 'multicast_query_use_ifaddr': '0', 'gc_timer': '8995', 'hello_time': '200', 'multicast_router': '1', 'nf_call_iptables': '0', 'group_addr': '01:80:c2:00:00:00', 'group_fwd_mask': '0x0', 'multicast_querier': '0', 'hash_max': '4096', 'tcn_timer': '0', 'vlan_protocol': '0x8100', 'vlan_stats_per_port': '0', 'root_port': '0', 'vlan_filtering': '0', 'multi cast_query_interval': '12500', 'bridge_id': '8000.ac1f6b3ccce8', 'max_age': '2000', 'nf_call_arptables': '0', 'multicast_startup_query_interval': '3125', 'multicast_stats_enabled': '0', 'multicast_query_response_interval': '1000', 'topology_change': '0', 'priority': '32768', 'multicast_mld_version': '1', 'hash_elasticity': '16', 'hello_timer': '0', 'default_pvid': '1', 'root_path_cost': '0', 'multicast_igmp_version': '2', 'stp_state': '0', 'multicast_startup_query_count': '2', 'topology_change_detected': '0', 'multicast_last_member_interval': '100', 'topology_change_timer': '0', 'root_id': '8000.ac1f6b3ccce8', 'no_linklocal_learn': '0', 'forward_delay': '1500', 'multicast_membership_interval': '26000', 'multicast_querier_interval': '25500', 'vlan_stats_enabled': '0', 'multicast_snooping': '1', 'nf_call_ip6tables': '0', 'multicast_last_member_count': '2'}}, 'virbr0': {'ports': [], 'stp': 'on', 'addr': '192.168.122.1', 'ipv4addrs': ['192.168.122.1/24'], 'ipv6addrs': [], 'ipv6autoconf' : False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'opts': {'ageing_time': '30000', 'multicast_query_use_ifaddr': '0', 'gc_timer': '22103', 'hello_time': '200', 'multicast_router': '1', 'nf_call_iptables': '0', 'group_addr': '01:80:c2:00:00:00', 'group_fwd_mask': '0x0', 'multicast_querier': '0', 'hash_max': '4096', 'tcn_timer': '0', 'vlan_protocol': '0x8100', 'vlan_stats_per_port': '0', 'root_port': '0', 'vlan_filtering': '0', 'multicast_query_interval': '12500', 'bridge_id': '8000.525400a8ef1c', 'max_age': '2000', 'nf_call_arptables': '0', 'multicast_startup_query_interval': '3125', 'multicast_stats_enabled': '0', 'multicast_query_response_interval': '1000', 'topology_change': '0', 'priority': '32768', 'multicast_mld_version': '1', 'hash_elasticity': '16', 'hello_timer': '103', 'default_pvid': '1', 'root_path_cost': '0', 'multicast_igmp_version': '2', 'stp_state': '1', 'multicast_start up_query_count': '2', 'topology_change_detected': '0', 'multicast_last_member_interval': '100', 'topology_change_timer': '0', 'root_id': '8000.525400a8ef1c', 'no_linklocal_learn': '0', 'forward_delay': '200', 'multicast_membership_interval': '26000', 'multicast_querier_interval': '25500', 'vlan_stats_enabled': '0', 'multicast_snooping': '1', 'nf_call_ip6tables': '0', 'multicast_last_member_count': '2'}}}, 'nics': {'ens1': {'hwaddr': '00:1b:21:e4:a6:f4', 'addr': '192.168.5.20', 'ipv4addrs': ['192.168.5.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'speed': 1000}, 'eno1': {'hwaddr': 'ac:1f:6b:3c:cc:e8', 'ad_aggregator_id': '1', 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'ipv4defaultroute': False, 'permhwaddr': 'ac:1f:6b:3c:cc:e8', 'dhcpv4': False, 'dhcpv 6': False, 'speed': 1000}, 'eno2': {'hwaddr': 'ac:1f:6b:3c:cc:e8', 'ad_aggregator_id': '1', 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'ipv4defaultroute': False, 'permhwaddr': 'ac:1f:6b:3c:cc:e9', 'dhcpv4': False, 'dhcpv6': False, 'speed': 1000}}, 'vlans': {'bond0.20': {'iface': 'bond0', 'vlanid': 20, 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False}, 'bond0.10': {'iface': 'bond0', 'vlanid': 10, 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False}, 'bond0.40': {'iface': 'bond0', 'vlanid': 40, 'addr': '192.168.4.20', 'ipv4addrs': ['192.168.4.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6 gateway': '::', 'mtu': '9000', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False}}, 'nameservers': ['192.168.2.16', '192.168.2.17'], 'supportsIPv6': True, 'ovnConfigured': False, 'hooks': {'after_get_caps': {'ovirt_provider_ovn_hook': {'checksum': 'a2bdefca38b96c8ddab39822cc8282bf3f67d875c4879003ffc9661826c92421'}}, 'before_device_create': {'10_ovirt_provider_ovn_hook': {'checksum': 'b9d67afb41bd86a521ada2121e3505297b3b6dfd478275ce0bd9501fcda6dafc'}}, 'before_nic_hotplug': {'10_ovirt_provider_ovn_hook': {'checksum': 'b9d67afb41bd86a521ada2121e3505297b3b6dfd478275ce0bd9501fcda6dafc'}}, 'before_vm_start': {'50_hostedengine': {'checksum': 'e5f5262cf22e06cd34e227afb27647e479351266876019a64210dbcbd2a43830'}}}, 'operatingSystem': {'release': '1.8.el8', 'version': '8.8', 'name': 'RHEL', 'pretty_name': 'Rocky Linux 8.8 (Green Obsidian)'}, 'uuid': '18e93400-808a-11ea-8000-ac1f6b3ccce8', 'packages2': {'kernel': {'version': '4.18.0', 'release': '477.27.1.e l8_8.x86_64'}, 'glusterfs-cli': {'version': '10.4', 'release': '1.el8s'}, 'librbd1': {'version': '16.2.13', 'release': '1.el8s'}, 'libvirt': {'version': '8.0.0', 'release': '19.2.module+el8.8.0+1321+70d049b2'}, 'mom': {'version': '0.6.4', 'release': '1.el8'}, 'ovirt-hosted-engine-ha': {'version': '2.5.0', 'release': '1.el8'}, 'openvswitch': {'version': '2.15', 'release': '4.el8'}, 'nmstate': {'version': '1.4.4', 'release': '2.el8_8'}, 'qemu-img': {'version': '6.2.0', 'release': '33.module+el8.8.0+1454+0b2cbfb8'}, 'qemu-kvm': {'version': '6.2.0', 'release': '33.module+el8.8.0+1454+0b2cbfb8'}, 'spice-server': {'version': '0.14.3', 'release': '4.el8'}, 'vdsm': {'version': '4.50.3.4', 'release': '1.el8'}, 'glusterfs': {'version': '10.4', 'release': '1.el8s'}, 'glusterfs-fuse': {'version': '10.4', 'release': '1.el8s'}, 'glusterfs-geo-replication': {'version': '10.4', 'release': '1.el8s'}, 'glusterfs-server': {'version': '10.4', 'release': '1.el8s'}}, 'realtimeKernel': False, 'kernelArgs' : 'BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-477.27.1.el8_8.x86_64 root=UUID=9acf17b0-558a-46d3-b21c-74c208680fcb ro crashkernel=auto resume=UUID=9c5f897c-8e34-48a7-9f06-786af3c079e9', 'nestedVirtualization': True, 'emulatedMachines': ['pc-q35-rhel8.0.0', 'pc-q35-rhel7.4.0', 'pc-i440fx-rhel7.3.0', 'pc-q35-rhel8.2.0', 'pc-q35-rhel7.3.0', 'pc-i440fx-rhel7.4.0', 'pc-q35-rhel8.4.0', 'pc-q35-rhel7.6.0', 'pc-i440fx-rhel7.5.0', 'pc-q35-rhel8.1.0', 'pc-q35-rhel8.6.0', 'pc-q35-rhel8.3.0', 'pc-q35-rhel7.5.0', 'pc-q35-rhel8.5.0', 'pc-i440fx-rhel7.1.0', 'q35', 'pc-i440fx-rhel7.2.0', 'pc', 'pc-i440fx-rhel7.6.0', 'pc-i440fx-rhel7.0.0'], 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:dc85f9619f1d', 'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:dc85f9619f1d'}], 'FC': []}, 'vmTypes': ['kvm'], 'memSize': '63805', 'reservedMem': '321', 'guestOverhead': '65', 'rngSources': ['random', 'hwrng'], 'numaNodes': {'0': {'totalMemory': '31553', 'hugepages': {'4': {'totalPages': '8077814'}, ' 2048': {'totalPages': '0'}, '1048576': {'totalPages': '0'}}, 'cpus': [0, 1, 2, 3, 4, 5, 6, 7, 16, 17, 18, 19, 20, 21, 22, 23]}, '1': {'totalMemory': '32251', 'hugepages': {'4': {'totalPages': '8256270'}, '2048': {'totalPages': '0'}, '1048576': {'totalPages': '0'}}, 'cpus': [8, 9, 10, 11, 12, 13, 14, 15, 24, 25, 26, 27, 28, 29, 30, 31]}}, 'numaNodeDistance': {'0': [10, 21], '1': [21, 10]}, 'autoNumaBalancing': 2, 'selinux': {'mode': '1'}, 'liveSnapshot': 'true', 'liveMerge': 'true', 'kdumpStatus': 1, 'deferred_preallocation': True, 'hostdevPassthrough': 'false', 'additionalFeatures': ['libgfapi_supported', 'GLUSTER_SNAPSHOT', 'GLUSTER_GEO_REPLICATION', 'GLUSTER_BRICK_MANAGEMENT'], 'hostedEngineDeployed': True, 'hugepages': [2048, 1048576], 'kernelFeatures': {'SPECTRE_V2': '(Mitigation: Enhanced IBRS, IBPB: conditional, RSB filling, PBRSB-eIBRS: SW sequence)', 'ITLB_MULTIHIT': '(KVM: Mitigation: VMX disabled)', 'MMIO_STALE_DATA': '(Mitigation: Clear CPU buffers; SMT vulnerable)', 'MDS ': '(Not affected)', 'L1TF': '(Not affected)', 'SPEC_STORE_BYPASS': '(Mitigation: Speculative Store Bypass disabled via prctl)', 'TSX_ASYNC_ABORT': '(Mitigation: TSX disabled)', 'SPECTRE_V1': '(Mitigation: usercopy/swapgs barriers and __user pointer sanitization)', 'RETBLEED': '(Mitigation: Enhanced IBRS)', 'SRBDS': '(Not affected)', 'MELTDOWN': '(Not affected)'}, 'vncEncrypted': True, 'backupEnabled': True, 'coldBackupEnabled': True, 'clearBitmapsEnabled': True, 'fipsEnabled': False, 'boot_uuid': '68e5a5ca-3dd5-4103-9613-6065a5fc74da', 'tscFrequency': '3199999000', 'tscScaling': True, 'domain_versions': [0, 2, 3, 4, 5], 'supported_block_size': {'FCP': [512], 'GLUSTERFS': [0, 512, 4096], 'ISCSI': [512], 'LOCALFS': [0, 512, 4096], 'NFS': [512], 'POSIXFS': [512]}, 'cd_change_pdiv': True, 'refresh_disk_supported': True, 'replicate_extend': True, 'measure_subchain': True, 'measure_active': True, 'mailbox_events': True, 'zerocopy_migrations': True, 'netConfigDirty': 'False', 'openstack_b inding_host_ids': {'OVIRT_PROVIDER_OVN': 'vmhost1.mydoamin.local'}, 'lastClientIface': 'ovirtmgmt'}} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:10,620+1100 INFO (jsonrpc/1) [api.host] START getHardwareInfo() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:10,621+1100 INFO (jsonrpc/1) [api.host] FINISH getHardwareInfo return={'status': {'code': 0, 'message': 'Done'}, 'info': {'systemManufacturer': 'Supermicro', 'systemProductName': 'X11DPi-N(T)', 'systemVersion': '123456789', 'systemSerialNumber': '', 'systemUUID': '0034e918-8a80-ea11-8000-ac1f6b3ccce8', 'systemFamily': 'SMC X11'}} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:10,767+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.2.28,43698, flow_id=c4e6178 (api:31) 2023-10-16 18:21:10,778+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=bab905a7-0651-41a3-a9f6-49707358a0eb (api:31) 2023-10-16 18:21:10,778+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={} from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=bab905a7-0651-41a3-a9f6-49707358a0eb (api:37) 2023-10-16 18:21:10,779+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=0f382ce1-8a94-4764-94b7-394b65b3db99 (api:31) 2023-10-16 18:21:10,779+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=0f382ce1-8a94-4764-94b7-394b65b3db99 (api:37) 2023-10-16 18:21:10,786+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698, flow_id=c4e6178 (api:37) 2023-10-16 18:21:11,032+1100 INFO (jsonrpc/6) [vds] Setting Hosted Engine HA local maintenance to False (API:1697) 2023-10-16 18:21:11,198+1100 INFO (jsonrpc/4) [vdsm.api] START connectStorageServer(domType=7, spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', conList=[{'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:/gv0', 'ipv6_enabled': 'false', 'id': '8d7b930e-a758-4010-aaf5-a230447f2761', 'user': '', 'tpgt': '1'}]) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=be7fffbe-f4ca-48f3-b1af-064268ad9eac (api:31) 2023-10-16 18:21:11,199+1100 INFO (tmap-752/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Closing client (__init__:641) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77159) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Shutdown requested (__init__:126) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77159) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,200+1100 INFO (tmap-752/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Closing client (__init__:641) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77153) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Shutdown requested (__init__:126) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77153) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,200+1100 INFO (tmap-752/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Closing client (__init__:641) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77159) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,201+1100 INFO (ioprocess/77153) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,201+1100 INFO (ioprocess/77148) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Shutdown requested (__init__:126) 2023-10-16 18:21:11,201+1100 INFO (ioprocess/77148) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,202+1100 INFO (ioprocess/77148) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194) 2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) Closing client (__init__:641) 2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57) 2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'id': '8d7b930e-a758-4010-aaf5-a230447f2761', 'status': 0}]} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=be7fffbe-f4ca-48f3-b1af-064268ad9eac (api:37) 2023-10-16 18:21:11,210+1100 INFO (ioprocess/76327) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,210+1100 INFO (ioprocess/76327) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,216+1100 INFO (jsonrpc/2) [vdsm.api] START connectStorageServer(domType=7, spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', conList=[{'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:gv1', 'ipv6_enabled': 'false', 'id': '044a086b-e94b-4e5d-a118-283ffceb8de8', 'user': '', 'tpgt': '1'}, {'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:/gv2', 'ipv6_enabled': 'false', 'id': '2d328788-ade0-4e25-b947-1c392d4f133e', 'user': '', 'tpgt': '1'}, {'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:/gv3', 'i pv6_enabled': 'false', 'id': '420b9d49-eebb-4dea-99bb-bcb89fbe2bf2', 'user': '', 'tpgt': '1'}]) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=92bf27ed-8388-4485-abc3-cd79f0d4163f (api:31) 2023-10-16 18:21:11,293+1100 INFO (jsonrpc/2) [storage.storageServer] Creating directory '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1' (storageServer:217) 2023-10-16 18:21:11,293+1100 INFO (jsonrpc/2) [storage.fileutils] Creating directory: /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1 mode: None (fileUtils:214) 2023-10-16 18:21:11,293+1100 WARN (jsonrpc/2) [storage.storageServer] Using user specified backup-volfile-servers option (storageServer:326) 2023-10-16 18:21:11,293+1100 INFO (jsonrpc/2) [storage.mount] mounting vmhosttran1.mydoamin.local:gv1 at /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1 (mount:190) 2023-10-16 18:21:11,415+1100 INFO (jsonrpc/2) [IOProcessClient] (Global) Starting client (__init__:340) 2023-10-16 18:21:11,424+1100 INFO (ioprocess/86760) [IOProcess] (Global) Starting ioprocess (__init__:465) 2023-10-16 18:21:11,518+1100 INFO (jsonrpc/2) [storage.storageServer] Creating directory '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2' (storageServer:217) 2023-10-16 18:21:11,518+1100 INFO (jsonrpc/2) [storage.fileutils] Creating directory: /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2 mode: None (fileUtils:214) 2023-10-16 18:21:11,518+1100 WARN (jsonrpc/2) [storage.storageServer] Using user specified backup-volfile-servers option (storageServer:326) 2023-10-16 18:21:11,518+1100 INFO (jsonrpc/2) [storage.mount] mounting vmhosttran1.mydoamin.local:/gv2 at /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2 (mount:190) 2023-10-16 18:21:11,662+1100 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:11,662+1100 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:11,730+1100 INFO (jsonrpc/2) [storage.storageServer] Creating directory '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3' (storageServer:217) 2023-10-16 18:21:11,730+1100 INFO (jsonrpc/2) [storage.fileutils] Creating directory: /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3 mode: None (fileUtils:214) 2023-10-16 18:21:11,730+1100 WARN (jsonrpc/2) [storage.storageServer] Using user specified backup-volfile-servers option (storageServer:326) 2023-10-16 18:21:11,730+1100 INFO (jsonrpc/2) [storage.mount] mounting vmhosttran1.mydoamin.local:/gv3 at /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3 (mount:190) 2023-10-16 18:21:11,875+1100 INFO (tmap-753/1) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting client (__init__:340) 2023-10-16 18:21:11,883+1100 INFO (tmap-753/2) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting client (__init__:340) 2023-10-16 18:21:11,885+1100 INFO (ioprocess/87016) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting ioprocess (__init__:465) 2023-10-16 18:21:11,889+1100 INFO (tmap-753/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting client (__init__:340) 2023-10-16 18:21:11,892+1100 INFO (ioprocess/87022) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting ioprocess (__init__:465) 2023-10-16 18:21:11,898+1100 INFO (ioprocess/87027) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting ioprocess (__init__:465) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain e311ddf1-7f2c-49ef-a618-050d9a2b947f from storage domain cache (sdc:194) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Closing client (__init__:641) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 from storage domain cache (sdc:194) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Closing client (__init__:641) 2023-10-16 18:21:11,904+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 4412e492-f752-4a6c-98ff-f289043618da from storage domain cache (sdc:194) 2023-10-16 18:21:11,904+1100 INFO (jsonrpc/2) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) Closing client (__init__:641) 2023-10-16 18:21:11,904+1100 INFO (ioprocess/77213) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,904+1100 INFO (ioprocess/77197) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,904+1100 INFO (ioprocess/77222) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,905+1100 INFO (ioprocess/77197) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,905+1100 INFO (ioprocess/77213) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,906+1100 INFO (ioprocess/77222) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194) 2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain e311ddf1-7f2c-49ef-a618-050d9a2b947f from storage domain cache (sdc:194) 2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 from storage domain cache (sdc:194) 2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 4412e492-f752-4a6c-98ff-f289043618da from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain e311ddf1-7f2c-49ef-a618-050d9a2b947f from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 4412e492-f752-4a6c-98ff-f289043618da from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57) 2023-10-16 18:21:11,915+1100 INFO (jsonrpc/2) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'id': '044a086b-e94b-4e5d-a118-283ffceb8de8', 'status': 0}, {'id': '2d328788-ade0-4e25-b947-1c392d4f133e', 'status': 0}, {'id': '420b9d49-eebb-4dea-99bb-bcb89fbe2bf2', 'status': 0}]} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=92bf27ed-8388-4485-abc3-cd79f0d4163f (api:37) 2023-10-16 18:21:11,924+1100 INFO (jsonrpc/3) [vdsm.api] START connectStorageServer(domType=3, spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', conList=[{'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.24', 'ipv6_enabled': 'false', 'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'user': 'cephiscsichapinitiator', 'tpgt': '3'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.25', 'ipv6_enabled': 'false', 'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'user': 'cephiscsichapinitiator', 'tpgt': '2'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.23', 'ipv6_enabled': 'false', 'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'user': 'cephiscsichapinitiator', 'tpgt': '1'}]) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=32063392-d746-458c-b143-919fbf9df35b (api:31) 2023-10-16 18:21:11,925+1100 INFO (jsonrpc/3) [storage.iscsi] Adding iscsi node for target 192.168.4.24:3260,3 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:192) 2023-10-16 18:21:12,053+1100 INFO (jsonrpc/3) [storage.iscsi] Adding iscsi node for target 192.168.4.25:3260,2 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:192) 2023-10-16 18:21:12,174+1100 INFO (jsonrpc/3) [storage.iscsi] Adding iscsi node for target 192.168.4.23:3260,1 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:192) 2023-10-16 18:21:12,293+1100 INFO (jsonrpc/3) [storage.storageServer] Log in to 3 targets using 3 workers (storageServer:600) 2023-10-16 18:21:12,294+1100 INFO (iscsi-login/0) [storage.iscsi] Logging in to iscsi target 192.168.4.24:3260,3 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw via iface default (iscsi:214) 2023-10-16 18:21:12,297+1100 INFO (iscsi-login/1) [storage.iscsi] Logging in to iscsi target 192.168.4.25:3260,2 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw via iface default (iscsi:214) 2023-10-16 18:21:12,303+1100 INFO (iscsi-login/2) [storage.iscsi] Logging in to iscsi target 192.168.4.23:3260,1 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw via iface default (iscsi:214) 2023-10-16 18:21:13,667+1100 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=a9c9c716-740c-424c-8a49-62f84192416a (api:31) 2023-10-16 18:21:13,667+1100 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=a9c9c716-740c-424c-8a49-62f84192416a (api:37) 2023-10-16 18:21:13,676+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=093dfcd6-6f98-49bf-9723-0aca86957167 (api:31) 2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=093dfcd6-6f98-49bf-9723-0aca86957167 (api:37) 2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=69a16b74-67a6-4e74-9fb5-1c738e8bf029 (api:31) 2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=69a16b74-67a6-4e74-9fb5-1c738e8bf029 (api:37) 2023-10-16 18:21:13,693+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:14,410+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=1c78fec1-ad8d-4c4a-9396-bd8bb30248e4 (api:31) 2023-10-16 18:21:14,410+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=1c78fec1-ad8d-4c4a-9396-bd8bb30248e4 (api:37) 2023-10-16 18:21:15,485+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=0f779632-8240-4d02-b43e-ed4737312c36 (api:31) 2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=0f779632-8240-4d02-b43e-ed4737312c36 (api:37) 2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=2375aa06-ec3c-4658-b775-59683a0d25de (api:31) 2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=2375aa06-ec3c-4658-b775-59683a0d25de (api:37) 2023-10-16 18:21:15,503+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:16,692+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:21:16,693+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:17,362+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=bd2bdfd6-4d5e-4e60-8cab-3d25a2f31745 (api:31) 2023-10-16 18:21:17,362+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=bd2bdfd6-4d5e-4e60-8cab-3d25a2f31745 (api:37) 2023-10-16 18:21:17,366+1100 INFO (jsonrpc/1) [vdsm.api] START startMonitoringDomain(sdUUID='ee5b0268-8141-43e0-995e-8b4586217123', hostID=1) from=::1,50158, task_id=fca171ce-9c55-4300-affa-5caa1227495e (api:31) 2023-10-16 18:21:17,367+1100 INFO (jsonrpc/1) [storage.monitor] Start monitoring ee5b0268-8141-43e0-995e-8b4586217123 (monitor:176) 2023-10-16 18:21:17,368+1100 INFO (jsonrpc/1) [vdsm.api] FINISH startMonitoringDomain return=None from=::1,50158, task_id=fca171ce-9c55-4300-affa-5caa1227495e (api:37) 2023-10-16 18:21:17,368+1100 INFO (monitor/ee5b026) [storage.storagedomaincache] Refreshing storage domain cache (resize=True) (sdc:63) 2023-10-16 18:21:17,368+1100 INFO (monitor/ee5b026) [storage.iscsi] Scanning iSCSI devices (iscsi:445) 2023-10-16 18:21:17,377+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=750f996b-757e-4ccf-b08f-e82b91a88678 (api:31) 2023-10-16 18:21:17,378+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.0', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}} from=::1,50158, task_id=750f996b-757e-4ccf-b08f-e82b91a88678 (api:37) 2023-10-16 18:21:17,489+1100 INFO (monitor/ee5b026) [storage.iscsi] Scanning iSCSI devices: 0.12 seconds (utils:373) 2023-10-16 18:21:17,490+1100 INFO (monitor/ee5b026) [storage.hba] Scanning FC devices (hba:42) 2023-10-16 18:21:17,544+1100 INFO (monitor/ee5b026) [storage.hba] Scanning FC devices: 0.05 seconds (utils:373) 2023-10-16 18:21:17,544+1100 INFO (monitor/ee5b026) [storage.multipath] Waiting until multipathd is ready (multipath:95) 2023-10-16 18:21:19,586+1100 INFO (monitor/ee5b026) [storage.multipath] Waited 2.04 seconds for multipathd (tries=2, ready=2) (multipath:122) 2023-10-16 18:21:19,586+1100 INFO (monitor/ee5b026) [storage.multipath] Resizing multipath devices (multipath:223) 2023-10-16 18:21:19,588+1100 INFO (monitor/ee5b026) [storage.multipath] Resizing multipath devices: 0.00 seconds (utils:373) 2023-10-16 18:21:19,588+1100 INFO (monitor/ee5b026) [storage.storagedomaincache] Refreshing storage domain cache: 2.22 seconds (utils:373) 2023-10-16 18:21:19,594+1100 INFO (monitor/ee5b026) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) Starting client (__init__:340) 2023-10-16 18:21:19,603+1100 INFO (ioprocess/87289) [IOProcess] (ee5b0268-8141-43e0-995e-8b4586217123) Starting ioprocess (__init__:465) 2023-10-16 18:21:20,734+1100 INFO (monitor/ee5b026) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:21:20,734+1100 INFO (monitor/ee5b026) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123/dom_md/metadata' (check:104) 2023-10-16 18:21:20,777+1100 INFO (check/loop) [storage.monitor] Domain ee5b0268-8141-43e0-995e-8b4586217123 became VALID (monitor:465) 2023-10-16 18:21:20,779+1100 INFO (event/70) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:21:22,387+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=cb5a305c-dc34-4a94-bf20-fbec9643d784 (api:31) 2023-10-16 18:21:22,387+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=cb5a305c-dc34-4a94-bf20-fbec9643d784 (api:37) 2023-10-16 18:21:23,720+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=c4472b6f-95ab-4803-a5fd-b91b006f0e59 (api:31) 2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=c4472b6f-95ab-4803-a5fd-b91b006f0e59 (api:37) 2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=74b73927-5e56-4483-9f00-d69e5798b4c8 (api:31) 2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=74b73927-5e56-4483-9f00-d69e5798b4c8 (api:37) 2023-10-16 18:21:24,421+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ce22cf89-56ab-4757-97fb-c8e10692223b (api:31) 2023-10-16 18:21:24,422+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.6', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=ce22cf89-56ab-4757-97fb-c8e10692223b (api:37) 2023-10-16 18:21:25,529+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:25,540+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=9465754e-6bf1-4328-9a45-e3b521725225 (api:31) 2023-10-16 18:21:25,540+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.8', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=9465754e-6bf1-4328-9a45-e3b521725225 (api:37) 2023-10-16 18:21:25,541+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=dab8366e-1533-4155-9635-911729326a1a (api:31) 2023-10-16 18:21:25,541+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=dab8366e-1533-4155-9635-911729326a1a (api:37) 2023-10-16 18:21:26,665+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:26,665+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:27,399+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=e69c93c1-639c-4640-b7de-23724deb52f2 (api:31) 2023-10-16 18:21:27,399+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.6', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=e69c93c1-639c-4640-b7de-23724deb52f2 (api:37) 2023-10-16 18:21:28,689+1100 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=fc8a5b61-e928-425a-bd60-36a14d9334e2 (api:31) 2023-10-16 18:21:28,689+1100 INFO (periodic/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=internal, task_id=fc8a5b61-e928-425a-bd60-36a14d9334e2 (api:37) 2023-10-16 18:21:28,741+1100 WARN (jsonrpc/1) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:28,742+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:28,744+1100 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:30,548+1100 WARN (jsonrpc/5) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:30,549+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:30,550+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:30,763+1100 INFO (monitor/ee5b026) [storage.sanlock] Acquiring host id for domain ee5b0268-8141-43e0-995e-8b4586217123 (id=1, wait=False) (clusterlock:359) 2023-10-16 18:21:31,712+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:21:31,712+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:21:32,411+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=b0a218d5-91b0-4dc8-9f4c-45825e601a3c (api:31) 2023-10-16 18:21:32,412+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=b0a218d5-91b0-4dc8-9f4c-45825e601a3c (api:37) 2023-10-16 18:21:33,697+1100 WARN (periodic/0) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:33,753+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=39f0b3ae-475f-4951-90ff-581637710dc9 (api:31) 2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=39f0b3ae-475f-4951-90ff-581637710dc9 (api:37) 2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=2fd4952e-b521-4c0a-93e0-d1186a376d20 (api:31) 2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=2fd4952e-b521-4c0a-93e0-d1186a376d20 (api:37) 2023-10-16 18:21:34,435+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=fad247c4-b608-4df8-88e6-51bc025f21cd (api:31) 2023-10-16 18:21:34,435+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.6', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=fad247c4-b608-4df8-88e6-51bc025f21cd (api:37) 2023-10-16 18:21:35,562+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:35,573+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=37f91533-90ae-4c76-a3e2-c1b8362a9494 (api:31) 2023-10-16 18:21:35,574+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.8', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=37f91533-90ae-4c76-a3e2-c1b8362a9494 (api:37) 2023-10-16 18:21:35,574+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=a25a55a8-10df-48fc-82c2-f603096e58cd (api:31) 2023-10-16 18:21:35,574+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=a25a55a8-10df-48fc-82c2-f603096e58cd (api:37) 2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:37,420+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=8270e681-279e-44a0-a9b0-b7a0110803d7 (api:31) 2023-10-16 18:21:37,420+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.6', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=8270e681-279e-44a0-a9b0-b7a0110803d7 (api:37) 2023-10-16 18:21:38,773+1100 WARN (jsonrpc/4) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:38,774+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:38,775+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:40,581+1100 WARN (jsonrpc/7) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:40,582+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:40,583+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:41,668+1100 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:41,668+1100 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:42,430+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=3be41433-ac1d-4d72-963d-2a69eef1fb26 (api:31) 2023-10-16 18:21:42,431+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=3be41433-ac1d-4d72-963d-2a69eef1fb26 (api:37) 2023-10-16 18:21:43,788+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=5ab069ab-377b-47a4-a19c-e8b2606c7453 (api:31) 2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=5ab069ab-377b-47a4-a19c-e8b2606c7453 (api:37) 2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=14c99af0-527b-444a-92df-d84724c0d215 (api:31) 2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=14c99af0-527b-444a-92df-d84724c0d215 (api:37) 2023-10-16 18:21:44,450+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=88a44308-aac3-441f-a05b-bb0809854292 (api:31) 2023-10-16 18:21:44,450+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=88a44308-aac3-441f-a05b-bb0809854292 (api:37) 2023-10-16 18:21:45,595+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=a871b766-4ae8-4c0e-bd44-2d88034cd1b7 (api:31) 2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.8', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=a871b766-4ae8-4c0e-bd44-2d88034cd1b7 (api:37) 2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=5db4e011-a522-4c96-8a33-6eba3170787f (api:31) 2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=5db4e011-a522-4c96-8a33-6eba3170787f (api:37) 2023-10-16 18:21:45,610+1100 WARN (jsonrpc/6) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155) 2023-10-16 18:21:45,610+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:46,726+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:21:46,726+1100 INFO (jsonrpc/1) [throttled] Current getAllVmStats: {} (throttledlog:87) 2023-10-16 18:21:46,726+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:47,440+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=7e1eac36-3a04-4e4f-8a20-7cb84b703d1a (api:31) 2023-10-16 18:21:47,441+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.7', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=7e1eac36-3a04-4e4f-8a20-7cb84b703d1a (api:37) 2023-10-16 18:21:48,712+1100 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=e6378dc2-3d21-4cd5-851d-5db5bc5c18d4 (api:31) 2023-10-16 18:21:48,712+1100 INFO (periodic/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=internal, task_id=e6378dc2-3d21-4cd5-851d-5db5bc5c18d4 (api:37) 2023-10-16 18:21:48,715+1100 WARN (periodic/3) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155) 2023-10-16 18:21:48,808+1100 WARN (jsonrpc/4) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:48,809+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:48,810+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:52,453+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=10828206-04bc-4cd4-b5a4-38bd0cf9dc8d (api:31) 2023-10-16 18:21:52,453+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.7', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=10828206-04bc-4cd4-b5a4-38bd0cf9dc8d (api:37) 2023-10-16 18:21:53,821+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:53,833+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=6c48b607-2157-4404-9927-fbfca271c525 (api:31) 2023-10-16 18:21:53,833+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=6c48b607-2157-4404-9927-fbfca271c525 (api:37) 2023-10-16 18:21:53,834+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=4e007b1a-3480-4a60-91c9-ccc4a2b8a552 (api:31) 2023-10-16 18:21:53,834+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=4e007b1a-3480-4a60-91c9-ccc4a2b8a552 (api:37) 2023-10-16 18:21:53,837+1100 WARN (jsonrpc/7) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155) 2023-10-16 18:21:53,837+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:54,463+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=91a5858c-775d-4172-8bd7-085a03865e4b (api:31) 2023-10-16 18:21:54,463+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=91a5858c-775d-4172-8bd7-085a03865e4b (api:37) 2023-10-16 18:21:55,639+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=20121b0e-9d04-4a74-a7b8-3bbf48d1fd09 (api:31) 2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.9', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=20121b0e-9d04-4a74-a7b8-3bbf48d1fd09 (api:37) 2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=efa8dcb2-b1cf-4ab5-ae22-7953bdf379e2 (api:31) 2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=efa8dcb2-b1cf-4ab5-ae22-7953bdf379e2 (api:37) 2023-10-16 18:21:55,653+1100 WARN (jsonrpc/6) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155) 2023-10-16 18:21:55,653+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:56,671+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:56,671+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:57,462+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=0a931307-57e8-43ed-bffa-d832f421327a (api:31) 2023-10-16 18:21:57,462+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.7', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=0a931307-57e8-43ed-bffa-d832f421327a (api:37) 2023-10-16 18:22:00,800+1100 INFO (monitor/ee5b026) [storage.sanlock] Host id 1 for domain ee5b0268-8141-43e0-995e-8b4586217123 acquired in 30 seconds (clusterlock:427) 2023-10-16 18:22:01,744+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:22:01,745+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:22:02,471+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=4813e4e9-f151-4727-9cf7-fcf87d1ffdcd (api:31) 2023-10-16 18:22:02,471+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.7', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=4813e4e9-f151-4727-9cf7-fcf87d1ffdcd (api:37) 2023-10-16 18:22:02,569+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=859cacca-c5a4-4eb9-b5a8-c28ef96b464d (api:31) 2023-10-16 18:22:02,569+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.8', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=859cacca-c5a4-4eb9-b5a8-c28ef96b464d (api:37) 2023-10-16 18:22:03,663+1100 INFO (health) [health] LVM cache hit ratio: 39.35% (hits: 170 misses: 262) (health:114) 2023-10-16 18:22:03,729+1100 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=bb148f84-5eab-4cac-b563-b89d690db0bb (api:31) 2023-10-16 18:22:03,729+1100 INFO (periodic/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=bb148f84-5eab-4cac-b563-b89d690db0bb (api:37) 2023-10-16 18:22:03,862+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:03,873+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=305776e1-ad6e-4530-ae48-0f58899b0f6e (api:31) 2023-10-16 18:22:03,873+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.1', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=305776e1-ad6e-4530-ae48-0f58899b0f6e (api:37) 2023-10-16 18:22:03,874+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=f1300145-b974-4f32-90a7-553a6cefe028 (api:31) 2023-10-16 18:22:03,874+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=f1300145-b974-4f32-90a7-553a6cefe028 (api:37) 2023-10-16 18:22:03,879+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:04,475+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ee277021-41dc-414c-9886-c01c807e5658 (api:31) 2023-10-16 18:22:04,475+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=ee277021-41dc-414c-9886-c01c807e5658 (api:37) 2023-10-16 18:22:05,682+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=d1d6d329-10cb-4439-9218-e00b797220ab (api:31) 2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.9', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=d1d6d329-10cb-4439-9218-e00b797220ab (api:37) 2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::1,50158, task_id=1e8c24fa-9d0c-49ea-8f85-f06527068e73 (api:31) 2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=1e8c24fa-9d0c-49ea-8f85-f06527068e73 (api:37) 2023-10-16 18:22:05,699+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:11,675+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:22:11,675+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:22:11,925+1100 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'jsonrpc': '2.0', 'method': 'StoragePool.connectStorageServer', 'params': {'storagepoolID': '7bfaa890-f892-4b3b-bf18-33dabc13e646', 'domainType': 3, 'connectionParams': [{'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.24', 'ipv6_enabled': 'false', 'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'user': 'cephiscsichapinitiator', 'tpgt': '3'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.25', 'ipv6_enabled': 'false', 'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'user': 'cephiscsichapinitiator', 'tpgt': '2'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.23', 'ipv6_enabled': 'false', 'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'user': 'cephiscsich apinitiator', 'tpgt': '1'}]}, 'id': 'd19073fd-9420-4dee-ac37-509a5b321b58'} at 0x7f877a5fa128> timeout=60, duration=60.00 at 0x7f877a69d550> task#=1179 at 0x7f8778933240>, traceback: File: "/usr/lib64/python3.6/threading.py", line 905, in _bootstrap self._bootstrap_inner() File: "/usr/lib64/python3.6/threading.py", line 937, in _bootstrap_inner self.run() File: "/usr/lib64/python3.6/threading.py", line 885, in run self._target(*self._args, **self._kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 243, in run ret = func(*args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 284, in _run self._execute_task() File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 298, in _execute_task task() File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 374, in __call__ self._callable() File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 253, in __call__ self._handler(self._ctx, self._req) File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 296, in _serveRequest response = self._handle_request(req, ctx) File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 338, in _handle_request res = method(**params) File: "/usr/lib/python3.6/site-packages/vdsm/rpc/Bridge.py", line 182, in _dynamicMethod result = fn(*methodArgs) File: "/usr/lib/python3.6/site-packages/vdsm/API.py", line 1117, in connectStorageServer connectionParams) File: "/usr/lib/python3.6/site-packages/vdsm/storage/dispatcher.py", line 57, in wrapper result = ctask.prepare(func, *args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 93, in wrapper return m(self, *a, **kw) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 1163, in prepare result = self._run(func, *args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 867, in _run return fn(*args, **kargs) File: "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-117>", line 2, in connectStorageServer File: "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 33, in method ret = func(*args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2164, in connectStorageServer results = storageServer.connect(domType, conList) File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 914, in connect return con_class.connect_all(connections) File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 617, in connect_all for res in login_results: File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 207, in tmap yield results.get() File: "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File: "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() (executor:346) 2023-10-16 18:22:12,654+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=837a670b-015f-44ad-9f05-d3ca615f3981 (api:31) 2023-10-16 18:22:12,654+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.8', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=837a670b-015f-44ad-9f05-d3ca615f3981 (api:37) 2023-10-16 18:22:13,908+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=399b71ba-f296-4c9c-9ead-870d40d751fe (api:31) 2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.1', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=399b71ba-f296-4c9c-9ead-870d40d751fe (api:37) 2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=0b89ae1d-b45c-4274-a63d-f5dfa0f9392e (api:31) 2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=0b89ae1d-b45c-4274-a63d-f5dfa0f9392e (api:37) 2023-10-16 18:22:13,927+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:14,488+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=8f652e74-faae-4ac9-a6cb-1c114c23cf48 (api:31) 2023-10-16 18:22:14,488+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=8f652e74-faae-4ac9-a6cb-1c114c23cf48 (api:37) 2023-10-16 18:22:15,726+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=f85241ec-5b32-4efa-b1f9-8592a9fd57d5 (api:31) 2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.9', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=f85241ec-5b32-4efa-b1f9-8592a9fd57d5 (api:37) 2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::1,50158, task_id=01d80faa-e134-42d1-bce2-28f91748a9e3 (api:31) 2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=01d80faa-e134-42d1-bce2-28f91748a9e3 (api:37) 2023-10-16 18:22:15,743+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:16,763+1100 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:22:16,763+1100 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:18,750+1100 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=468af5ce-aa49-48f1-ac7c-762cc713710a (api:31) 2023-10-16 18:22:18,750+1100 INFO (periodic/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=468af5ce-aa49-48f1-ac7c-762cc713710a (api:37) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Closing client (__init__:641) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Closing client (__init__:641) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Closing client (__init__:641) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (Global) Closing client (__init__:641) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Closing client (__init__:641) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/87027) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/87022) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/87016) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/86760) [IOProcessClient] (Global) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/77141) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,815+1100 INFO (ioprocess/87022) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:20,815+1100 INFO (ioprocess/86760) [IOProcessClient] (Global) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:20,815+1100 INFO (ioprocess/87016) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:20,816+1100 INFO (ioprocess/87027) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:20,816+1100 INFO (ioprocess/77141) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:22,741+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=62257d75-2f2b-432d-a752-e0ee3e8fd29b (api:31) 2023-10-16 18:22:22,741+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.9', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=62257d75-2f2b-432d-a752-e0ee3e8fd29b (api:37) 2023-10-16 18:22:23,954+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=78b74721-7d6c-4ed2-88f0-4fff08c4b407 (api:31) 2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.1', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=78b74721-7d6c-4ed2-88f0-4fff08c4b407 (api:37) 2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=78ff047f-a712-4dbb-99e3-25909b977799 (api:31) 2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=78ff047f-a712-4dbb-99e3-25909b977799 (api:37) 2023-10-16 18:22:23,974+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:24,503+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=0a2888a2-c8ae-49e5-89fc-a6b5bebfd77f (api:31) 2023-10-16 18:22:24,503+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=0a2888a2-c8ae-49e5-89fc-a6b5bebfd77f (api:37) 2023-10-16 18:22:25,769+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=59f0f76f-e6d4-47cf-8ce0-6b3f9a7e850d (api:31) 2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.0', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=59f0f76f-e6d4-47cf-8ce0-6b3f9a7e850d (api:37) 2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=c5576936-9a21-4b28-886e-b6cd73c90cd4 (api:31) 2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=c5576936-9a21-4b28-886e-b6cd73c90cd4 (api:37) 2023-10-16 18:22:25,786+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:26,678+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:22:26,678+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:31,782+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:22:31,783+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:22:32,825+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=66cc2698-9c5a-4c9d-803b-d89bd549f4f0 (api:31) 2023-10-16 18:22:32,826+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.0', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=66cc2698-9c5a-4c9d-803b-d89bd549f4f0 (api:37) 2023-10-16 18:22:33,771+1100 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=c267fd6f-633d-4c6d-94ad-b5cb5bb8d02d (api:31) 2023-10-16 18:22:33,772+1100 INFO (periodic/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=c267fd6f-633d-4c6d-94ad-b5cb5bb8d02d (api:37) 2023-10-16 18:22:33,998+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=2654a62d-b4f9-425c-8442-b4c418c4f542 (api:31) 2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.2', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=2654a62d-b4f9-425c-8442-b4c418c4f542 (api:37) 2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=c02547e7-20f6-437a-84b4-4fdf98cfd003 (api:31) 2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=c02547e7-20f6-437a-84b4-4fdf98cfd003 (api:37) 2023-10-16 18:22:34,014+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:34,515+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=667f9b48-b2ab-4113-ae86-1b586bbdf5d6 (api:31) 2023-10-16 18:22:34,516+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=667f9b48-b2ab-4113-ae86-1b586bbdf5d6 (api:37) 2023-10-16 18:22:35,812+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=c04a2aa5-1f9d-4c87-b51f-34eaeeedb8e2 (api:31) 2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.0', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=c04a2aa5-1f9d-4c87-b51f-34eaeeedb8e2 (api:37) 2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=f6ac211c-50fa-481a-aeaa-01779cc54412 (api:31) 2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=f6ac211c-50fa-481a-aeaa-01779cc54412 (api:37) 2023-10-16 18:22:35,830+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:37,110+1100 INFO (jsonrpc/2) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:37,111+1100 INFO (jsonrpc/2) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:37,111+1100 INFO (jsonrpc/2) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:37,111+1100 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:41,681+1100 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:22:41,681+1100 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:22:42,911+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=c1a8bf5a-4aa0-4c85-a9fa-8c240a07daa9 (api:31) 2023-10-16 18:22:42,911+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=c1a8bf5a-4aa0-4c85-a9fa-8c240a07daa9 (api:37) 2023-10-16 18:22:44,040+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=d7eb5985-0470-4aea-bd23-937c1a03da72 (api:31) 2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.2', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=d7eb5985-0470-4aea-bd23-937c1a03da72 (api:37) 2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::1,50158, task_id=a32bf2e0-f35c-46f4-b582-a28de56e1960 (api:31) 2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=a32bf2e0-f35c-46f4-b582-a28de56e1960 (api:37) 2023-10-16 18:22:44,059+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:44,528+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=68c2e209-d655-437d-add5-769c0a4932ba (api:31) 2023-10-16 18:22:44,528+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=68c2e209-d655-437d-add5-769c0a4932ba (api:37) 2023-10-16 18:22:45,855+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=b93990c2-b263-4c89-ba4b-ab0e962f6ce8 (api:31) 2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.0', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=b93990c2-b263-4c89-ba4b-ab0e962f6ce8 (api:37) 2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=444bef31-e827-478f-a2fc-16e18dc0b9a6 (api:31) 2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=444bef31-e827-478f-a2fc-16e18dc0b9a6 (api:37) 2023-10-16 18:22:45,873+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:46,786+1100 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:22:46,787+1100 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:48,792+1100 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=50b7c32e-202c-40ec-b484-5eb3f45f7f28 (api:31) 2023-10-16 18:22:48,792+1100 INFO (periodic/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '8.0', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=50b7c32e-202c-40ec-b484-5eb3f45f7f28 (api:37) 2023-10-16 18:22:52,996+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=28687ab3-2149-42d2-92aa-d28cc5ad8d96 (api:31) 2023-10-16 18:22:52,996+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=28687ab3-2149-42d2-92aa-d28cc5ad8d96 (api:37) 2023-10-16 18:22:54,086+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=96f37ad8-6f60-4c9b-8986-cad8c4a5bdf3 (api:31) 2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.2', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=96f37ad8-6f60-4c9b-8986-cad8c4a5bdf3 (api:37) 2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=aa6dc50d-c1c9-4a8c-bf9d-b7ca456f73a6 (api:31) 2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=aa6dc50d-c1c9-4a8c-bf9d-b7ca456f73a6 (api:37) 2023-10-16 18:22:54,106+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:54,533+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ada1e11f-23e9-4704-a8c4-27bfecfd9974 (api:31) 2023-10-16 18:22:54,533+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=ada1e11f-23e9-4704-a8c4-27bfecfd9974 (api:37) 2023-10-16 18:22:55,899+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=a0dd0ff4-5a0a-43cf-8612-ac5ff8e953a8 (api:31) 2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=a0dd0ff4-5a0a-43cf-8612-ac5ff8e953a8 (api:37) 2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=9211755e-e9f9-4c7b-b0dc-8a7689a7884a (api:31) 2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=9211755e-e9f9-4c7b-b0dc-8a7689a7884a (api:37) 2023-10-16 18:22:55,916+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:56,684+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:22:56,684+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:22:57,141+1100 INFO (jsonrpc/5) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:57,141+1100 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:57,141+1100 INFO (jsonrpc/5) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:57,142+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:01,805+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:23:01,805+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:23:03,079+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=95612d72-c18c-4dfe-87f2-f8501f713aca (api:31) 2023-10-16 18:23:03,079+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.2', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=95612d72-c18c-4dfe-87f2-f8501f713aca (api:37) 2023-10-16 18:23:03,813+1100 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=c193e1ec-73f7-4858-aa76-fb9f7d977517 (api:31) 2023-10-16 18:23:03,814+1100 INFO (periodic/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=c193e1ec-73f7-4858-aa76-fb9f7d977517 (api:37) 2023-10-16 18:23:04,130+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:04,141+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=1df7dacf-ebad-4493-ad11-a7ce10fbf851 (api:31) 2023-10-16 18:23:04,141+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.3', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=1df7dacf-ebad-4493-ad11-a7ce10fbf851 (api:37) 2023-10-16 18:23:04,142+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=cbc0b187-25a2-43aa-bfa5-9cf54f675e08 (api:31) 2023-10-16 18:23:04,142+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=cbc0b187-25a2-43aa-bfa5-9cf54f675e08 (api:37) 2023-10-16 18:23:04,147+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:04,541+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=77176abe-b0e4-4f22-bce7-2f26f571743d (api:31) 2023-10-16 18:23:04,541+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=77176abe-b0e4-4f22-bce7-2f26f571743d (api:37) 2023-10-16 18:23:05,942+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:05,954+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=99736bd8-b5b6-4e29-a345-cec8db30f976 (api:31) 2023-10-16 18:23:05,954+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=99736bd8-b5b6-4e29-a345-cec8db30f976 (api:37) 2023-10-16 18:23:05,955+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=286f88d9-373c-42c6-b990-b70d2eb73137 (api:31) 2023-10-16 18:23:05,955+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=286f88d9-373c-42c6-b990-b70d2eb73137 (api:37) 2023-10-16 18:23:05,960+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:11,687+1100 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:11,687+1100 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:11,925+1100 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'jsonrpc': '2.0', 'method': 'StoragePool.connectStorageServer', 'params': {'storagepoolID': '7bfaa890-f892-4b3b-bf18-33dabc13e646', 'domainType': 3, 'connectionParams': [{'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.24', 'ipv6_enabled': 'false', 'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'user': 'cephiscsichapinitiator', 'tpgt': '3'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.25', 'ipv6_enabled': 'false', 'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'user': 'cephiscsichapinitiator', 'tpgt': '2'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.23', 'ipv6_enabled': 'false', 'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'user': 'cephiscsich apinitiator', 'tpgt': '1'}]}, 'id': 'd19073fd-9420-4dee-ac37-509a5b321b58'} at 0x7f877a5fa128> timeout=60, duration=120.00 at 0x7f877a69d550> task#=1179 at 0x7f8778933240>, traceback: File: "/usr/lib64/python3.6/threading.py", line 905, in _bootstrap self._bootstrap_inner() File: "/usr/lib64/python3.6/threading.py", line 937, in _bootstrap_inner self.run() File: "/usr/lib64/python3.6/threading.py", line 885, in run self._target(*self._args, **self._kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 243, in run ret = func(*args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 284, in _run self._execute_task() File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 298, in _execute_task task() File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 374, in __call__ self._callable() File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 253, in __call__ self._handler(self._ctx, self._req) File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 296, in _serveRequest response = self._handle_request(req, ctx) File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 338, in _handle_request res = method(**params) File: "/usr/lib/python3.6/site-packages/vdsm/rpc/Bridge.py", line 182, in _dynamicMethod result = fn(*methodArgs) File: "/usr/lib/python3.6/site-packages/vdsm/API.py", line 1117, in connectStorageServer connectionParams) File: "/usr/lib/python3.6/site-packages/vdsm/storage/dispatcher.py", line 57, in wrapper result = ctask.prepare(func, *args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 93, in wrapper return m(self, *a, **kw) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 1163, in prepare result = self._run(func, *args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 867, in _run return fn(*args, **kargs) File: "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-117>", line 2, in connectStorageServer File: "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 33, in method ret = func(*args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2164, in connectStorageServer results = storageServer.connect(domType, conList) File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 914, in connect return con_class.connect_all(connections) File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 617, in connect_all for res in login_results: File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 207, in tmap yield results.get() File: "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File: "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() (executor:346) 2023-10-16 18:23:13,165+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=0cb1a91c-f096-4daa-bb08-ec407a86433a (api:31) 2023-10-16 18:23:13,165+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.3', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=0cb1a91c-f096-4daa-bb08-ec407a86433a (api:37) 2023-10-16 18:23:14,173+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:14,185+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=fbf38d2b-bc3e-49ad-afd9-de55861129db (api:31) 2023-10-16 18:23:14,185+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.3', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=fbf38d2b-bc3e-49ad-afd9-de55861129db (api:37) 2023-10-16 18:23:14,185+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=3e936d0b-7879-45c8-89d4-ae6bbae466a2 (api:31) 2023-10-16 18:23:14,186+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=3e936d0b-7879-45c8-89d4-ae6bbae466a2 (api:37) 2023-10-16 18:23:14,192+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:14,554+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=316fa299-920a-4c26-a829-9e93647ea804 (api:31) 2023-10-16 18:23:14,554+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=316fa299-920a-4c26-a829-9e93647ea804 (api:37) 2023-10-16 18:23:15,986+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=c7094132-9870-44f0-8274-54527984da18 (api:31) 2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=c7094132-9870-44f0-8274-54527984da18 (api:37) 2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=61b7e2c2-106d-42ff-b6cd-e91e557af60d (api:31) 2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=61b7e2c2-106d-42ff-b6cd-e91e557af60d (api:37) 2023-10-16 18:23:16,003+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:16,823+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:23:16,823+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:18,834+1100 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=5086291d-9172-41f1-9db9-7011a6a34abb (api:31) 2023-10-16 18:23:18,834+1100 INFO (periodic/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=5086291d-9172-41f1-9db9-7011a6a34abb (api:37) 2023-10-16 18:23:23,249+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=bf9f76c3-bead-4087-947b-ebd063ebab6d (api:31) 2023-10-16 18:23:23,249+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.3', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=bf9f76c3-bead-4087-947b-ebd063ebab6d (api:37) 2023-10-16 18:23:24,217+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=3296546b-846b-48f8-b2f3-9fa13e06864e (api:31) 2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.3', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=3296546b-846b-48f8-b2f3-9fa13e06864e (api:37) 2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=77d3abd5-6cf4-4a3f-874e-0a0a1adcf3d5 (api:31) 2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=77d3abd5-6cf4-4a3f-874e-0a0a1adcf3d5 (api:37) 2023-10-16 18:23:24,236+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:24,566+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=12ad1f12-b845-4107-9bc0-326453b99721 (api:31) 2023-10-16 18:23:24,566+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=12ad1f12-b845-4107-9bc0-326453b99721 (api:37) 2023-10-16 18:23:26,030+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=cf9b3b2a-8182-440a-82b8-0c2dd0e33cee (api:31) 2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=cf9b3b2a-8182-440a-82b8-0c2dd0e33cee (api:37) 2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=20a2b03c-eee3-4d1d-a47e-cc6e581df61f (api:31) 2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=20a2b03c-eee3-4d1d-a47e-cc6e581df61f (api:37) 2023-10-16 18:23:26,047+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:26,373+1100 INFO (iscsi-login/0) [storage.iscsi] Removing iscsi node for target 192.168.4.24:3260,3 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:227) 2023-10-16 18:23:26,388+1100 INFO (iscsi-login/1) [storage.iscsi] Removing iscsi node for target 192.168.4.25:3260,2 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:227) 2023-10-16 18:23:26,436+1100 ERROR (iscsi-login/0) [storage.storageServer] Could not login to target (storageServer:607) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 301, in node_login "-p", portal, "-l"]) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 104, in run_cmd out = commands.run(cmd, sudo=True) File "/usr/lib/python3.6/site-packages/vdsm/common/commands.py", line 85, in run raise cmdutils.Error(args, p.returncode, out, err) vdsm.common.cmdutils.Error: Command ['/sbin/iscsiadm', '-m', 'node', '-T', 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', '-I', 'default', '-p', '192.168.4.24:3260,3', '-l'] failed with rc=11 out=b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260]\n' err=b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 604, in iscsi_login iscsi.loginToIscsiNode(con.iface, con.target) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsi.py", line 216, in loginToIscsiNode iscsiadm.node_login(iface.name, target.address, target.iqn) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 321, in node_login raise IscsiNodeError(e.rc, e.out, e.err) vdsm.storage.iscsiadm.IscsiNodeError: (11, b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260]\n', b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n') 2023-10-16 18:23:26,497+1100 ERROR (iscsi-login/1) [storage.storageServer] Could not login to target (storageServer:607) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 301, in node_login "-p", portal, "-l"]) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 104, in run_cmd out = commands.run(cmd, sudo=True) File "/usr/lib/python3.6/site-packages/vdsm/common/commands.py", line 85, in run raise cmdutils.Error(args, p.returncode, out, err) vdsm.common.cmdutils.Error: Command ['/sbin/iscsiadm', '-m', 'node', '-T', 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', '-I', 'default', '-p', '192.168.4.25:3260,2', '-l'] failed with rc=11 out=b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260]\n' err=b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 604, in iscsi_login iscsi.loginToIscsiNode(con.iface, con.target) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsi.py", line 216, in loginToIscsiNode iscsiadm.node_login(iface.name, target.address, target.iqn) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 321, in node_login raise IscsiNodeError(e.rc, e.out, e.err) vdsm.storage.iscsiadm.IscsiNodeError: (11, b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260]\n', b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n') 2023-10-16 18:23:26,514+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Refreshing storage domain cache (resize=True) (sdc:63) 2023-10-16 18:23:26,514+1100 INFO (jsonrpc/3) [storage.iscsi] Scanning iSCSI devices (iscsi:445) 2023-10-16 18:23:26,609+1100 INFO (jsonrpc/3) [storage.iscsi] Scanning iSCSI devices: 0.09 seconds (utils:373) 2023-10-16 18:23:26,609+1100 INFO (jsonrpc/3) [storage.hba] Scanning FC devices (hba:42) 2023-10-16 18:23:26,663+1100 INFO (jsonrpc/3) [storage.hba] Scanning FC devices: 0.05 seconds (utils:373) 2023-10-16 18:23:26,663+1100 INFO (jsonrpc/3) [storage.multipath] Waiting until multipathd is ready (multipath:95) 2023-10-16 18:23:26,690+1100 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:26,690+1100 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:28,705+1100 INFO (jsonrpc/3) [storage.multipath] Waited 2.04 seconds for multipathd (tries=2, ready=2) (multipath:122) 2023-10-16 18:23:28,705+1100 INFO (jsonrpc/3) [storage.multipath] Resizing multipath devices (multipath:223) 2023-10-16 18:23:28,706+1100 INFO (jsonrpc/3) [storage.multipath] Resizing multipath devices: 0.00 seconds (utils:373) 2023-10-16 18:23:28,706+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Refreshing storage domain cache: 2.19 seconds (utils:373) 2023-10-16 18:23:28,796+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Removing domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 from storage domain cache (sdc:194) 2023-10-16 18:23:28,796+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57) 2023-10-16 18:23:28,796+1100 INFO (jsonrpc/3) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'status': 0}, {'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'status': 465}, {'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'status': 465}]} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=32063392-d746-458c-b143-919fbf9df35b (api:37) 2023-10-16 18:23:28,797+1100 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer took more than 1.00 seconds to succeed: 136.87 (__init__:305) 2023-10-16 18:23:28,847+1100 INFO (jsonrpc/5) [vdsm.api] START connectStoragePool(spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', hostID=1, msdUUID='ee5b0268-8141-43e0-995e-8b4586217123', masterVersion=1, domainsMap={'e311ddf1-7f2c-49ef-a618-050d9a2b947f': 'active', 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': 'active', '30775056-38ef-483a-889c-70453e9aa5f9': 'active', 'ee5b0268-8141-43e0-995e-8b4586217123': 'active', '3e33b7ce-7279-4765-a203-cca37eff8cb0': 'active', '4412e492-f752-4a6c-98ff-f289043618da': 'active', '0dde39a4-42e4-4fba-8068-e63e42dd3334': 'active', 'd44ec7f2-3629-4110-93a5-7d8d57230efc': 'active'}) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=22550292-d106-45d1-b5b7-17ff11f6f8a7 (api:31) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagepoolmemorybackend] new storage pool master version 1 and domains map {'e311ddf1-7f2c-49ef-a618-050d9a2b947f': 'Active', 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': 'Active', '30775056-38ef-483a-889c-70453e9aa5f9': 'Active', 'ee5b0268-8141-43e0-995e-8b4586217123': 'Active', '3e33b7ce-7279-4765-a203-cca37eff8cb0': 'Active', '4412e492-f752-4a6c-98ff-f289043618da': 'Active', '0dde39a4-42e4-4fba-8068-e63e42dd3334': 'Active', 'd44ec7f2-3629-4110-93a5-7d8d57230efc': 'Active'} (spbackends:417) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagepool] updating pool 7bfaa890-f892-4b3b-bf18-33dabc13e646 backend from type NoneType instance 0x7f87cc236b70 to type StoragePoolMemoryBackend instance 0x7f877a7cdcc8 (sp:149) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagepool] Connect host #1 to the storage pool 7bfaa890-f892-4b3b-bf18-33dabc13e646 with master domain: ee5b0268-8141-43e0-995e-8b4586217123 (ver = 1) (sp:699) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Clearing storage domain cache (sdc:182) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Refreshing storage domain cache (resize=True) (sdc:63) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.iscsi] Scanning iSCSI devices (iscsi:445) 2023-10-16 18:23:28,946+1100 INFO (jsonrpc/5) [storage.iscsi] Scanning iSCSI devices: 0.10 seconds (utils:373) 2023-10-16 18:23:28,946+1100 INFO (jsonrpc/5) [storage.hba] Scanning FC devices (hba:42) 2023-10-16 18:23:28,996+1100 INFO (jsonrpc/5) [storage.hba] Scanning FC devices: 0.05 seconds (utils:373) 2023-10-16 18:23:28,996+1100 INFO (jsonrpc/5) [storage.multipath] Waiting until multipathd is ready (multipath:95) 2023-10-16 18:23:31,033+1100 INFO (jsonrpc/5) [storage.multipath] Waited 2.04 seconds for multipathd (tries=2, ready=2) (multipath:122) 2023-10-16 18:23:31,033+1100 INFO (jsonrpc/5) [storage.multipath] Resizing multipath devices (multipath:223) 2023-10-16 18:23:31,034+1100 INFO (jsonrpc/5) [storage.multipath] Resizing multipath devices: 0.00 seconds (utils:373) 2023-10-16 18:23:31,034+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Refreshing storage domain cache: 2.19 seconds (utils:373) 2023-10-16 18:23:31,035+1100 INFO (tmap-757/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Starting client (__init__:340) 2023-10-16 18:23:31,043+1100 INFO (tmap-757/1) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting client (__init__:340) 2023-10-16 18:23:31,045+1100 INFO (ioprocess/87928) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Starting ioprocess (__init__:465) 2023-10-16 18:23:31,052+1100 INFO (tmap-757/2) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting client (__init__:340) 2023-10-16 18:23:31,054+1100 INFO (ioprocess/87936) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting ioprocess (__init__:465) 2023-10-16 18:23:31,059+1100 INFO (tmap-757/3) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting client (__init__:340) 2023-10-16 18:23:31,062+1100 INFO (ioprocess/87940) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting ioprocess (__init__:465) 2023-10-16 18:23:31,068+1100 INFO (ioprocess/87948) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting ioprocess (__init__:465) 2023-10-16 18:23:31,842+1100 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:23:31,843+1100 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:23:32,194+1100 INFO (jsonrpc/5) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:23:32,195+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 0dde39a4-42e4-4fba-8068-e63e42dd3334 (monitor:176) 2023-10-16 18:23:32,195+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 3e33b7ce-7279-4765-a203-cca37eff8cb0 (monitor:176) 2023-10-16 18:23:32,201+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring e311ddf1-7f2c-49ef-a618-050d9a2b947f (monitor:176) 2023-10-16 18:23:32,206+1100 INFO (jsonrpc/5) [storage.monitor] Attaching monitor for ee5b0268-8141-43e0-995e-8b4586217123 to the pool (monitor:172) 2023-10-16 18:23:32,207+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 4412e492-f752-4a6c-98ff-f289043618da (monitor:176) 2023-10-16 18:23:32,208+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring fa8d922a-df3c-4f5b-adf6-a77c3689c397 (monitor:176) 2023-10-16 18:23:32,210+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 30775056-38ef-483a-889c-70453e9aa5f9 (monitor:176) 2023-10-16 18:23:32,211+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154) 2023-10-16 18:23:32,218+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring d44ec7f2-3629-4110-93a5-7d8d57230efc (monitor:176) 2023-10-16 18:23:32,219+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154) 2023-10-16 18:23:32,230+1100 INFO (monitor/3e33b7c) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Starting client (__init__:340) 2023-10-16 18:23:32,234+1100 INFO (jsonrpc/5) [storage.storagepool] Creating pool directory '/rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646' (sp:1323) 2023-10-16 18:23:32,234+1100 INFO (jsonrpc/5) [storage.fileutils] Creating directory: /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646 mode: None (fileUtils:214) 2023-10-16 18:23:32,239+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154) 2023-10-16 18:23:32,260+1100 INFO (monitor/e311ddf) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Starting client (__init__:340) 2023-10-16 18:23:32,263+1100 INFO (ioprocess/87989) [IOProcess] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Starting ioprocess (__init__:465) 2023-10-16 18:23:32,268+1100 INFO (monitor/4412e49) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) Starting client (__init__:340) 2023-10-16 18:23:32,271+1100 INFO (ioprocess/88003) [IOProcess] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Starting ioprocess (__init__:465) 2023-10-16 18:23:32,278+1100 INFO (ioprocess/88012) [IOProcess] (4412e492-f752-4a6c-98ff-f289043618da) Starting ioprocess (__init__:465) 2023-10-16 18:23:32,323+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.11 seconds (utils:373) 2023-10-16 18:23:32,323+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',) 2023-10-16 18:23:32,323+1100 INFO (monitor/fa8d922) [storage.monitor] Domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 became INVALID (monitor:465) 2023-10-16 18:23:32,334+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.11 seconds (utils:373) 2023-10-16 18:23:32,334+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',) 2023-10-16 18:23:32,334+1100 INFO (monitor/3077505) [storage.monitor] Domain 30775056-38ef-483a-889c-70453e9aa5f9 became INVALID (monitor:465) 2023-10-16 18:23:32,339+1100 WARN (jsonrpc/5) [storage.lvm] Marked vgs=[0dde39a4-42e4-4fba-8068-e63e42dd3334] as Unreadable due to reload failure (lvm:679) 2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.storagepool] Creating domain metadata directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md' (sp:1354) 2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md mode: None (fileUtils:214) 2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.storagepool] Creating domain images directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images' (sp:1357) 2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images mode: None (fileUtils:214) 2023-10-16 18:23:32,340+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.10 seconds (utils:373) 2023-10-16 18:23:32,340+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',) 2023-10-16 18:23:32,340+1100 INFO (monitor/d44ec7f) [storage.monitor] Domain d44ec7f2-3629-4110-93a5-7d8d57230efc became INVALID (monitor:465) 2023-10-16 18:23:32,345+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/0dde39a4-42e4-4fba-8068-e63e42dd3334 (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/0dde39a4-42e4-4fba-8068-e63e42dd3334 (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/e311ddf1-7f2c-49ef-a618-050d9a2b947f (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/e311ddf1-7f2c-49ef-a618-050d9a2b947f (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/3e33b7ce-7279-4765-a203-cca37eff8cb0 (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/3e33b7ce-7279-4765-a203-cca37eff8cb0 (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/4412e492-f752-4a6c-98ff-f289043618da (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/4412e492-f752-4a6c-98ff-f289043618da (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/ee5b0268-8141-43e0-995e-8b4586217123 (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/ee5b0268-8141-43e0-995e-8b4586217123 (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/mastersd (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/mastersd (sp:1295) 2023-10-16 18:23:32,383+1100 INFO (jsonrpc/5) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=22550292-d106-45d1-b5b7-17ff11f6f8a7 (api:37) 2023-10-16 18:23:32,384+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect took more than 1.00 seconds to succeed: 3.54 (__init__:305) 2023-10-16 18:23:32,385+1100 INFO (monitor/0dde39a) [storage.lvm] Activating lvs: vg=0dde39a4-42e4-4fba-8068-e63e42dd3334 lvs=['ids'] (lvm:1839) 2023-10-16 18:23:32,390+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.2.28,43698, flow_id=706708f9 (api:31) 2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=ac4e0a4f-b2d6-4d28-b765-6ec18d5ecb94 (api:31) 2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.2', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '0.1', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actua l': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '0.1', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '0.1', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=ac4e0a4f-b2d6-4d28-b765-6ec18d5ecb94 (api:37) 2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=c006784d-0604-489e-a8a8-8be4c44e3bec (api:31) 2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=c006784d-0604-489e-a8a8-8be4c44e3bec (api:37) 2023-10-16 18:23:32,413+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698, flow_id=706708f9 (api:37) 2023-10-16 18:23:32,572+1100 INFO (monitor/0dde39a) [storage.lvm] Activating lvs: vg=0dde39a4-42e4-4fba-8068-e63e42dd3334 lvs=['leases'] (lvm:1839) 2023-10-16 18:23:32,771+1100 INFO (monitor/0dde39a) [storage.lvm] Activating lvs: vg=0dde39a4-42e4-4fba-8068-e63e42dd3334 lvs=['metadata', 'inbox', 'outbox', 'xleases', 'master'] (lvm:1839) 2023-10-16 18:23:32,975+1100 INFO (monitor/0dde39a) [storage.blocksd] Setting up domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 (blockSD:973) 2023-10-16 18:23:33,056+1100 INFO (monitor/0dde39a) [storage.check] Start checking '/dev/0dde39a4-42e4-4fba-8068-e63e42dd3334/metadata' (check:104) 2023-10-16 18:23:33,143+1100 INFO (monitor/0dde39a) [storage.monitor] Domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 became VALID (monitor:465) 2023-10-16 18:23:33,144+1100 INFO (monitor/0dde39a) [storage.sanlock] Acquiring host id for domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 (id=1, wait=False) (clusterlock:359) 2023-10-16 18:23:33,145+1100 INFO (event/81) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:23:33,145+1100 INFO (event/82) [storage.storagedomainmanifest] Creating domain images directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images' (blockSD:766) 2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images mode: None (fileUtils:214) 2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.storagedomainmanifest] Creating domain special volumes directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md' (blockSD:771) 2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md mode: None (fileUtils:214) 2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.storagepool] Linking /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/0dde39a4-42e4-4fba-8068-e63e42dd3334 (sp:1278) 2023-10-16 18:23:33,338+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=50cf6835-08a1-4b31-8a61-6e18b85fb86e (api:31) 2023-10-16 18:23:33,338+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.1', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=50cf6835-08a1-4b31-8a61-6e18b85fb86e (api:37) 2023-10-16 18:23:33,470+1100 INFO (monitor/3e33b7c) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:23:33,471+1100 INFO (monitor/3e33b7c) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0/dom_md/metadata' (check:104) 2023-10-16 18:23:33,474+1100 INFO (monitor/4412e49) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:23:33,475+1100 INFO (monitor/4412e49) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da/dom_md/metadata' (check:104) 2023-10-16 18:23:33,519+1100 INFO (check/loop) [storage.monitor] Domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 became VALID (monitor:465) 2023-10-16 18:23:33,521+1100 INFO (event/84) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:23:33,522+1100 INFO (check/loop) [storage.monitor] Domain 4412e492-f752-4a6c-98ff-f289043618da became VALID (monitor:465) 2023-10-16 18:23:33,523+1100 INFO (event/85) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/3e33b7ce-7279-4765-a203-cca37eff8cb0 (sp:1278) 2023-10-16 18:23:33,524+1100 INFO (monitor/e311ddf) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:23:33,524+1100 INFO (monitor/e311ddf) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f/dom_md/metadata' (check:104) 2023-10-16 18:23:33,528+1100 INFO (event/87) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:23:33,534+1100 INFO (event/88) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/4412e492-f752-4a6c-98ff-f289043618da (sp:1278) 2023-10-16 18:23:33,542+1100 INFO (check/loop) [storage.monitor] Domain e311ddf1-7f2c-49ef-a618-050d9a2b947f became VALID (monitor:465) 2023-10-16 18:23:33,543+1100 INFO (event/90) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:23:33,543+1100 INFO (event/91) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/e311ddf1-7f2c-49ef-a618-050d9a2b947f (sp:1278) 2023-10-16 18:23:33,856+1100 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=67318041-086e-4992-a2c3-dc0b7e5ebd4e (api:31) 2023-10-16 18:23:33,856+1100 INFO (periodic/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '0.7', 'delay': '0.00174884', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00116006', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00604746', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00091358', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '1.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '1.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '1.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=internal, task_id=67318041-086e-4992-a2c3-dc0b7e5ebd4e (api:37) 2023-10-16 18:23:34,263+1100 INFO (jsonrpc/3) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=8cf2bf3c-b4ac-4f24-9179-29054ce22a46 (api:31) 2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '1.1', 'delay': '0.00174884', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.8', 'delay': '0.00116006', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.7', 'delay': '0.00604746', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.8', 'delay': '0.00091358', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '2.0', 'delay': '0', 'valid': False, 'version': - 1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=8cf2bf3c-b4ac-4f24-9179-29054ce22a46 (api:37) 2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::1,50158, task_id=dfd9aafe-1cbf-445f-afa9-6b19287c9955 (api:31) 2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=dfd9aafe-1cbf-445f-afa9-6b19287c9955 (api:37) 2023-10-16 18:23:34,280+1100 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:34,572+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=efc38225-ad10-4a67-97d8-1f7f2854456d (api:31) 2023-10-16 18:23:34,572+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.4', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=efc38225-ad10-4a67-97d8-1f7f2854456d (api:37) 2023-10-16 18:23:36,073+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:36,084+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=e1664ede-6677-4756-97d9-d1b033ebb9b6 (api:31) 2023-10-16 18:23:36,085+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00174884', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.6', 'delay': '0.00116006', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.5', 'delay': '0.00604746', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.6', 'delay': '0.00091358', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.8', 'delay': '0', 'valid': False, 'version': - 1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.8', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=e1664ede-6677-4756-97d9-d1b033ebb9b6 (api:37) 2023-10-16 18:23:36,085+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=592b6419-5402-4f74-83ae-b088d9b4b978 (api:31) 2023-10-16 18:23:36,085+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=592b6419-5402-4f74-83ae-b088d9b4b978 (api:37) 2023-10-16 18:23:36,090+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:41,693+1100 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:41,694+1100 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:42,324+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154) 2023-10-16 18:23:42,335+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154) 2023-10-16 18:23:42,341+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154) 2023-10-16 18:23:42,423+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.09 seconds (utils:373) 2023-10-16 18:23:42,423+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',) 2023-10-16 18:23:42,426+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.10 seconds (utils:373) 2023-10-16 18:23:42,426+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',) 2023-10-16 18:23:42,428+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=139f761e-b9f7-4451-8864-f96a498176ed (api:31) 2023-10-16 18:23:42,428+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.2', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=139f761e-b9f7-4451-8864-f96a498176ed (api:37) 2023-10-16 18:23:42,432+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.09 seconds (utils:373) 2023-10-16 18:23:42,432+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',) 2023-10-16 18:23:43,488+1100 INFO (monitor/4412e49) [storage.sanlock] Acquiring host id for domain 4412e492-f752-4a6c-98ff-f289043618da (id=1, wait=False) (clusterlock:359) 2023-10-16 18:23:43,488+1100 INFO (monitor/3e33b7c) [storage.sanlock] Acquiring host id for domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 (id=1, wait=False) (clusterlock:359) 2023-10-16 18:23:43,541+1100 INFO (monitor/e311ddf) [storage.sanlock] Acquiring host id for domain e311ddf1-7f2c-49ef-a618-050d9a2b947f (id=1, wait=False) (clusterlock:359) 2023-10-16 18:23:44,308+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=0d42d65f-0534-47c0-aaf9-7dbac3067a91 (api:31) 2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '1.2', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.8', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.7', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.8', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=0d42d65f-0534-47c0-aaf9-7dbac3067a91 (api:37) 2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=dad656d9-6a05-443b-82ca-c4b7632334dd (api:31) 2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=dad656d9-6a05-443b-82ca-c4b7632334dd (api:37) 2023-10-16 18:23:44,328+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:44,587+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ae63dc53-56a4-4d9a-8bc1-63c6f3c2f160 (api:31) 2023-10-16 18:23:44,587+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.4', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=ae63dc53-56a4-4d9a-8bc1-63c6f3c2f160 (api:37) 2023-10-16 18:23:46,096+1100 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:46,107+1100 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, task_id=9d13808b-d90d-449b-9f9c-85e91b946eeb (api:31) 2023-10-16 18:23:46,107+1100 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.5', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::ffff:192.168.2.28,43698, task_id=9d13808b-d90d-449b-9f9c-85e91b946eeb (api:37) 2023-10-16 18:23:46,108+1100 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, task_id=53f94ff6-5532-41b5-ae7a-3271108c20d2 (api:31) 2023-10-16 18:23:46,108+1100 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, task_id=53f94ff6-5532-41b5-ae7a-3271108c20d2 (api:37) 2023-10-16 18:23:46,113+1100 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:46,121+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=19f3fa69-f932-4edd-96f6-76ba56449cf2 (api:31) 2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=19f3fa69-f932-4edd-96f6-76ba56449cf2 (api:37) 2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=78db57fe-4ab9-4256-b239-2fb8fa2c6a61 (api:31) 2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=78db57fe-4ab9-4256-b239-2fb8fa2c6a61 (api:37) 2023-10-16 18:23:46,137+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:46,860+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:23:46,860+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:23:47,219+1100 INFO (jsonrpc/4) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:47,220+1100 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:47,220+1100 INFO (jsonrpc/4) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:47,220+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:48,878+1100 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=56a338bf-2d23-4dfe-8065-e8e5acc5736a (api:31) 2023-10-16 18:23:48,878+1100 INFO (periodic/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.7', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '5.7', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '5.4', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '5.3', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '5.3', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '6.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '6.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '6.4', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=internal, task_id=56a338bf-2d23-4dfe-8065-e8e5acc5736a (api:37) 2023-10-16 18:23:52,424+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154) 2023-10-16 18:23:52,428+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154) 2023-10-16 18:23:52,433+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154) 2023-10-16 18:23:52,516+1100 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=a77c47c6-03ba-40fd-9bfa-29375ea32ae4 (api:31) 2023-10-16 18:23:52,516+1100 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=a77c47c6-03ba-40fd-9bfa-29375ea32ae4 (api:37) 2023-10-16 18:23:52,542+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.12 seconds (utils:373) 2023-10-16 18:23:52,543+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',) 2023-10-16 18:23:52,546+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.11 seconds (utils:373) 2023-10-16 18:23:52,546+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',) 2023-10-16 18:23:52,551+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.12 seconds (utils:373) 2023-10-16 18:23:52,551+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',) 2023-10-16 18:23:53,353+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=72f4cb81-c54d-4153-9425-0a5766308ed4 (api:31) 2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.1', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '0.2', 'delay': '0.00305946', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '9.9', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '9.8', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '9.8', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '0.8', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '0.8', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '0.8', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=72f4cb81-c54d-4153-9425-0a5766308ed4 (api:37) 2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=65572c4a-c042-49c8-834d-dee8d879896a (api:31) 2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=65572c4a-c042-49c8-834d-dee8d879896a (api:37) 2023-10-16 18:23:53,373+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:54,605+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=1ba074d9-5e35-4527-93ca-0423ed3187f8 (api:31) 2023-10-16 18:23:54,605+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.4', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=1ba074d9-5e35-4527-93ca-0423ed3187f8 (api:37) 2023-10-16 18:23:56,164+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=cdb6b35e-ab67-48e1-a210-2f4ab737a00f (api:31) 2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00305946', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.7', 'delay': '0.00109704', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.6', 'delay': '0.00924556', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.7', 'delay': '0.000824828', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=cdb6b35e-ab67-48e1-a210-2f4ab737a00f (api:37) 2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=9099f061-8efd-4c2e-9669-ff848b5aa791 (api:31) 2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=9099f061-8efd-4c2e-9669-ff848b5aa791 (api:37) 2023-10-16 18:23:56,182+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:56,696+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:56,697+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:24:01,111+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:24:01,123+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, task_id=49598180-2b1c-493c-a4ff-f49a3e988b19 (api:31) 2023-10-16 18:24:01,123+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.4', 'delay': '0.00137321', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '8.0', 'delay': '0.00305946', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '7.6', 'delay': '0.00109704', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '7.6', 'delay': '0.00924556', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '7.6', 'delay': '0.000824828', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '8.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '8.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '8.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::ffff:192.168.2.28,43698, task_id=49598180-2b1c-493c-a4ff-f49a3e988b19 (api:37) 2023-10-16 18:24:01,124+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, task_id=ed16e0f7-5985-4608-94dd-28a587cc92d2 (api:31) 2023-10-16 18:24:01,124+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, task_id=ed16e0f7-5985-4608-94dd-28a587cc92d2 (api:37) 2023-10-16 18:24:01,130+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:24:01,879+1100 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:24:01,879+1100 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:24:02,543+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154) 2023-10-16 18:24:02,547+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154) 2023-10-16 18:24:02,552+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154) 2023-10-16 18:24:02,610+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=6b15ba3d-d2fe-4568-b235-25ded159d77c (api:31) 2023-10-16 18:24:02,610+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00137321', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=6b15ba3d-d2fe-4568-b235-25ded159d77c (api:37) 2023-10-16 18:24:02,744+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.20 seconds (utils:373) 2023-10-16 18:24:02,744+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',) 2023-10-16 18:24:02,755+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.21 seconds (utils:373) 2023-10-16 18:24:02,755+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',) 2023-10-16 18:24:02,755+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.20 seconds (utils:373) 2023-10-16 18:24:02,755+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',) 2023-10-16 18:24:02,828+1100 INFO (jsonrpc/3) [vds] Setting Hosted Engine HA local maintenance to True (API:1697) ~~~ Further info can be provided if required (I don't know what else to include) Thanks in advance for any and all help.

1 0

Ansible - export_ova
by duparchy＠esrf.fr 16 Oct '23

16 Oct '23

We're using a script to export some VMs. This involves export_ova task - Is there a way to parallelize OVA exports (running on different hosts) ? When exporting to OVA from the GUI, we have to select a host, but ansible export_ova has no "hosts" parameter.

3 2

What is the status of the whole Ovirt Project?
by Arman Khalatyan 15 Oct '23

15 Oct '23

Hello everybody, What is the status of the ovirt project, would be continued it in the Rocky Linunx9? the last message is so sad: https://lists.ovirt.org/archives/list/users@ovirt.org/thread/HEKKBM6MZEKBEA… Any good news/progress there? As a happy ovirt user since 2016 should we move to an another system? Thank you beforehand, Arman.

9 14

Ansible - export_ova
by duparchy＠esrf.fr 13 Oct '23

13 Oct '23

We're using a script to export some VMs. This involves export_ova task - Is there a way to parallelize OVA exports (running on different hosts) ? When exporting to OVA from the GUI, we have to select a host, but ansible export_ova has no "hosts" parameter.

1 0

How do i to drive Intel 82599ES 10-Gigabit SFP+ Adapter
by ejzhang＠hotmail.com 13 Oct '23

13 Oct '23

I can find the Adapter. [root@localhost ~]# lspci | grep 82599 06:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) But cannot to install the driver, I try to upgrade kernel to 6.1.57, but the problem is remains. How can i to solve the problem? Thanks very much! [root@localhost tmp]# dnf localinstall kmod-ixgbe-5.19.6-1.rhel9u1.x86_64.rpm Last metadata expiration check: 0:24:26 ago on Thu 12 Oct 2023 05:25:27 PM CST. Error: Problem: conflicting requests - nothing provides kernel(__SCK__tp_func_xdp_exception) = 0xc15ae5f5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(___pskb_trim) = 0x80ed79e7 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__alloc_pages) = 0x26458776 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__alloc_skb) = 0xbc60587f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__dev_kfree_skb_any) = 0x8d62db77 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__free_pages) = 0x7ddd045c needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__hw_addr_sync_dev) = 0x84c17451 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__hw_addr_unsync_dev) = 0xb05b216e needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__napi_alloc_skb) = 0x5f802418 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__napi_schedule_irqoff) = 0xd022b30c needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__netif_napi_del) = 0x310d1afd needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__page_frag_cache_drain) = 0xb6ccef70 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__pci_register_driver) = 0x1cb89f6b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__skb_pad) = 0x16cd8a38 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__tracepoint_xdp_exception) = 0xad42e583 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(_dev_err) = 0x3a3ec6c6 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(_dev_info) = 0xff4f021a needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(_dev_warn) = 0x1e5fac62 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(alloc_etherdev_mqs) = 0x665d1b75 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(bpf_prog_put) = 0xa1b79421 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(bpf_warn_invalid_xdp_action) = 0xb05e54b3 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(build_skb) = 0x5c034c8f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(consume_skb) = 0x014a9f1b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dca3_get_tag) = 0x258e3aac needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dca_add_requester) = 0x14542454 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dca_remove_requester) = 0x7e094b21 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dcb_getapp) = 0x535a6190 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dcb_ieee_delapp) = 0xc6496dd5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dcb_ieee_setapp) = 0x68aabfb5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dcb_setapp) = 0x0bf4f0bb needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(debugfs_create_dir) = 0x11b4f7a8 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(debugfs_create_file) = 0xa6202222 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(debugfs_remove) = 0x20033468 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dev_addr_add) = 0x96486dae needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dev_addr_del) = 0xe222c9e8 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dev_driver_string) = 0xcdafdb2a needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dev_get_stats) = 0x9a9222bd needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(device_set_wakeup_enable) = 0xac4a0980 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(devm_hwmon_device_register_with_groups) = 0x70a8831d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(devm_kmalloc) = 0xe9ad0580 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_alloc_attrs) = 0xed08591d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_free_attrs) = 0x5c4b9170 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_map_page_attrs) = 0xe2dc579b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_set_coherent_mask) = 0x97c42987 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_set_mask) = 0xf4110cd1 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_sync_single_for_cpu) = 0x51b1550f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_sync_single_for_device) = 0x9980b0a6 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_unmap_page_attrs) = 0xa41d1497 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(driver_for_each_device) = 0x9465cb85 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(eth_get_headlen) = 0x98601bb3 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(eth_type_trans) = 0x25c1aba0 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(eth_validate_addr) = 0xfc0d5980 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ethtool_op_get_link) = 0xc3aa9108 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ethtool_op_get_ts_info) = 0x7eb31781 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(flow_block_cb_setup_simple) = 0x4dd5fafe needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(free_netdev) = 0x52c68cbc needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ipv6_find_hdr) = 0x9c90a5b6 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(kfree_skb_reason) = 0xc87ace65 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(kmalloc_caches) = 0x5f73fb3c needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(kmem_cache_alloc_trace) = 0xa3cddcae needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(module_layout) = 0xa7f1f8eb needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(napi_complete_done) = 0x40402bca needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(napi_consume_skb) = 0x10e9a3dc needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(napi_disable) = 0x284fd3bf needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(napi_enable) = 0xaf328093 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(napi_gro_receive) = 0x4bd5d8dd needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(napi_schedule_prep) = 0x76bae8ab needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ndo_dflt_bridge_getlink) = 0x089345b5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ndo_dflt_fdb_add) = 0x77dcc759 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_crit) = 0x6e6d1264 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_err) = 0xc66472cf needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_info) = 0xa71c2c40 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_notice) = 0x07efd02f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_printk) = 0x2c04b9df needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_reset_tc) = 0x8849ff8a needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_set_num_tc) = 0xd82a15a6 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_set_tc_queue) = 0x9d6a1782 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_warn) = 0xfe96cfed needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_carrier_off) = 0x14111c62 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_carrier_on) = 0x96781ba1 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_device_attach) = 0x6d7af90c needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_device_detach) = 0xf3103b00 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_napi_add) = 0xec77a489 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_schedule_queue) = 0xe191e05f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_set_real_num_rx_queues) = 0x8379688b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_set_real_num_tx_queues) = 0xa1e2fa4b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_set_xps_queue) = 0x2813f1cd needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_tx_stop_all_queues) = 0x13392855 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_tx_wake_queue) = 0x24e11830 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(param_array_ops) = 0x7a0b1ed0 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(param_ops_int) = 0xaad18ecd needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_aer_clear_nonfatal_status) = 0x254f8d5e needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_dev_driver) = 0x45689a09 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_dev_get) = 0xdd00d2c6 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_dev_put) = 0x516c634d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_disable_device) = 0x0921c2a1 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_disable_link_state) = 0xc5408d7d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_disable_msi) = 0xe3ee7e3c needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_disable_msix) = 0x5d88de56 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_disable_pcie_error_reporting) = 0xf4167337 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_disable_sriov) = 0xae1adbfa needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_enable_device_mem) = 0x74ebcbd1 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_enable_msi) = 0x01bdea04 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_enable_msix_range) = 0x98ff98d5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_enable_pcie_error_reporting) = 0x717d4b61 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_enable_sriov) = 0x68c478f7 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_find_capability) = 0x3b06b1de needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_find_ext_capability) = 0xce3f4b59 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_get_device) = 0x7e3c03c5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_num_vf) = 0xbf36cc3e needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_prepare_to_sleep) = 0x4d75b46f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_read_config_byte) = 0x595813a6 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_read_config_dword) = 0x5d999172 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_read_config_word) = 0x3911c65c needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_release_selected_regions) = 0x7a3abda2 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_request_selected_regions) = 0xfab14d9f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_restore_state) = 0x24bbcb8b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_save_state) = 0xfd2da512 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_select_bars) = 0x1191f7a3 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_set_master) = 0x2f6d2bb5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_set_power_state) = 0x54621869 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_sriov_set_totalvfs) = 0xc0efcf8f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_unregister_driver) = 0xa972d633 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_vfs_assigned) = 0x6bdb5a77 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_wake_from_d3) = 0x2464b789 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_write_config_word) = 0x767cdb47 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pcie_capability_read_word) = 0xd51ae9d7 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pcie_print_link_status) = 0x26df835b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(printk) = 0xc5850110 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pskb_expand_head) = 0x0d526b2b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ptp_clock_event) = 0xe7d77369 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ptp_clock_index) = 0x3876beca needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ptp_clock_register) = 0x0ed9b290 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ptp_clock_unregister) = 0xb77a6063 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(register_netdev) = 0xba3ff097 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(simple_open) = 0xf9fc90f3 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(skb_add_rx_frag) = 0x21ea980a needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(skb_checksum_help) = 0x08d53501 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(skb_clone_tx_timestamp) = 0x5b57e627 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(skb_copy_bits) = 0x632061f3 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(skb_put) = 0xe6c6ba90 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(skb_tstamp_tx) = 0xc6a9499d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(softnet_data) = 0x3bf76a4e needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(udp_tunnel_nic_ops) = 0x9b1d2e6b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(unregister_netdev) = 0x01d13501 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_convert_zc_to_xdp_frame) = 0xcdb7b8cd needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_do_redirect) = 0x635f6fef needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_master_redirect) = 0xd895cc1b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_return_frame) = 0x5b1c3ac9 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_return_frame_rx_napi) = 0x4742580d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_rxq_info_reg) = 0xc5f644e9 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_rxq_info_reg_mem_model) = 0xcabfedb2 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_rxq_info_unreg) = 0x0d8f96fc needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_rxq_info_unreg_mem_model) = 0x610cb9e1 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_alloc) = 0xa0815c99 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_dma_map) = 0x0198092e needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_dma_sync_for_cpu_slow) = 0x0cac6528 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_dma_sync_for_device_slow) = 0xe91374d9 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_dma_unmap) = 0xb4f7ca78 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_free) = 0xa353e08d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_raw_get_dma) = 0x0eb88c7f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_set_rxq_info) = 0x86b2db4e needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xsk_tx_completed) = 0x5205b281 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xsk_tx_peek_desc) = 0xe67282ce needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xsk_tx_release) = 0xb016016f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

1 0

Missing data for some VM in Grafana Monitoring Portal
by Guillaume Pavese 12 Oct '23

12 Oct '23

Hi all. We are using oVirt 4.5.4 with the provided grafana monitoring tool. Everything seemed to be working as expected, however we have discovered that for some vm (not all), there is no available Disk or Ethernet data (while CPU and Memory data is available) : [image: image.png] While for some other VMs running on the same host, all data is available: [image: image.png] Does anyone have an idea what might be going wrong? -- Ce message et toutes les pièces jointes (ci-après le “message”) sont établis à l’intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse. L’internet ne permettant pas d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse ou il aurait été modifié. IT, ES, UK. <https://interactiv-group.com/disclaimer.html>

1 0

PKIX path validation failed:
by Marcelo Vieira 11 Oct '23

11 Oct '23

Prezados, boa tarde! Como faço para atualizar o certificado Java? Estou recebendo o seguinte erro na console do engine : PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed e recebendo erro na console do root" This web browser is too old to run the Web Console (missing selector(:is():where())) [root@node1 ~]# cat /etc/centos-release CentOS Stream release 8 [root@node1 ~]# uname -a Linux node1 4.18.0-383.el8.x86_64 #1 SMP Wed Apr 20 15:38:08 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux Att., Marcelo

1 0

After upgrade to 4.5.4: host reboot fails and api-user unable to log in
by Thyen, Niko 11 Oct '23

11 Oct '23

Hi all, i just upgraded a cluster from 4.3.10 to 4.5.4. 3 hosts, hosted-engine, engine deployment via shell and with restore from backup. Engine update worked fine, but host update from 4.4.10 to 4.5.4 failed because the engine could not initiate a host reboot: 2023-10-10 13:37:27,802+02 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (EE-ManagedThreadFactory-engine-Thread-2) [7d4d6 a58] The message key 'SshHostReboot' is missing from 'bundles/ExecutionMessages' 2023-10-10 13:37:27,823+02 INFO [org.ovirt.engine.core.bll.SshHostRebootCommand] (EE-ManagedThreadFactory-engine-Thread-2) [7d4d6a58] Running command: SshHostRebootCommand internal: true. Entities affected : ID: b578519d-7f34-4509-8b8b-dbab1b2cf6d2 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2023-10-10 13:37:27,827+02 INFO [org.ovirt.engine.core.bll.SshHostRebootCommand] (EE-ManagedThreadFactory-engine-Thread-2) [7d4d6a58] Opening SSH reboot session on host <host fqdn> 2023-10-10 13:37:28,030+02 ERROR [org.ovirt.engine.core.bll.SshHostRebootCommand] (EE-ManagedThreadFactory-engine-Thread-2) [7d4d6a58] SSH reboot command failed on host '<host fqdn>': SSH session closed during connection 'root@<host fqdn>' Stdout: Stderr: 2023-10-10 13:37:28,038+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-2) [7d4d6a58] EVENT_ID: SYSTEM_FAILED_SSH_HOST_RESTART(198), A restart using SSH initiated by the engine to Host <host fqdn> has failed. This also happens when a host is reinstalled via engine webinterface. How can i fix this? The only Workaround i found is rebooting the host via SSH and reactivating it via engine webinterface. Also, i constantly find error messages like these in the engine.log: 2023-10-11 13:54:11,338+02 INFO [org.ovirt.engine.extension.aaa.jdbc.core.Authentication] (default task-815) [] locking user: api-user due to interval failures 2023-10-11 13:54:14,932+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-812) [] EVENT_ID: USER_VDC_LOGIN_FAILED(114), User api-user@internal connecting from '<engine-ip>' failed to log in : 'Unable to log in because the password has expired. Please change the password to proceed.'. 2023-10-11 13:54:14,933+02 ERROR [org.ovirt.engine.core.sso.service.SsoService] (default task-815) [] OAuthException access_denied: Cannot authenticate user 'api-user@internal': Unable to log in because the password has expired. Please change the password to proceed.. 2023-10-11 13:54:14,934+02 ERROR [org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default task-814) [] Cannot authenticate using authentication Headers: access_denied: Cannot authenticate user 'api-user@internal': Unable to log in because the password has expired. Please change the password to proceed.. This only affects the user "api-user". I am still new to oVirt and have no idea what the api-user is trying to do and where. I found https://www.ovirt.org/develop/release-management/features/infra/aaa-jdbc.ht… but havent tried resetting yet because i dont know what will be affected by that. Can anyone help to point me in the right direction? Many thanks in advance :) Regards, Niko

1 0

Impossible to snaphot certain virtual machines
by nolhan.bertille＠unige.ch 10 Oct '23

10 Oct '23

Hello all, When performing snapshots I recieve the following error 2023-10-09 16:09:49,951: !!! No snapshot found !!! 2023-10-09 16:09:49,951: All backups done 2023-10-09 16:09:49,951: Backup failure for: 2023-10-09 16:09:49,951: ******* 2023-10-09 16:09:49,951: Some errors occurred during the backup, please check the log file Where can I look for more information? Where can I rise the verbosity ? What process should I look at ? Thanks all

1 0

Getting error when performing snapshots
by nolhan.bertille＠unige.ch 09 Oct '23

09 Oct '23

Hello Guys, I'm repeatly getting the following error when running snapshots with the script backup.py 2023-10-06 14:54:09,773: !!! No snapshot found !!! 2023-10-06 14:54:09,773: All backups done 2023-10-06 14:54:09,773: Backup failure for: 2023-10-06 14:54:09,773: ******* 2023-10-06 14:54:09,773: Some errors occurred during the backup, please check the log file I have no idea what/where to lok at ? Could somebody help me ?

1 0

Hosted engine VM not coming up after Storage rebuild.
by Sumit Basu 06 Oct '23

06 Oct '23

Hi, We are running Ovirt 4.3 on IBM x3650, x3550 servers with our SAN on IBM Midrange storage DS5300. All the storage domains are separate LUN's in the storage with a dedicated LUN for the HostedEngine VM. The storage had a failure due to a major power issue. One of the storage array's that have the storage domains had to be re-constructed using IBM's StorageManager tool. After booting the hosts, i can see all the storage domains as LUN's with the multipath -ll from all the hosts, but on starting the hosted engine with #hosted-engine --vm-start, and checking with --vm-status i get "Engine status : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}'. The global maintenance mode is enabled and i tried "hosted-engine --console" - after logging in i checked the boot.log, i find [FAILED] Failed to mount /var/log. See 'systemctl status var-log.mount' for details. [DEPEND] Dependency failed for Update UTMP about System Boot/Shutdown. [DEPEND] Dependency failed for Update UTMP about System Runlevel Changes. [DEPEND] Dependency failed for Flush Journal to Persistent Storage. [DEPEND] Dependency failed for /var/log/audit. [DEPEND] Dependency failed for Local File Systems. [DEPEND] Dependency failed for Mark the need to relabel after reboot. [DEPEND] Dependency failed for Migrate local... structure to the new structure. [DEPEND] Dependency failed for Relabel all filesystems, if necessary. the "systemctl status var-log.mount" shows ● var-log.mount - /var/log Loaded: loaded (/etc/fstab; bad; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2023-10-06 14:38:58 IST; 10min ago Where: /var/log What: /dev/mapper/ovirt-log Docs: man:fstab(5) man:systemd-fstab-generator(8) Process: 691 ExecMount=/bin/mount /dev/mapper/ovirt-log /var/log -t xfs -o nodev (code=exited, status=32) Oct 06 14:38:57 ovman systemd[1]: Mounting /var/log... Oct 06 14:38:58 ovman mount[691]: mount: mount /dev/mapper/ovirt-...g Oct 06 14:38:58 ovman systemd[1]: var-log.mount mount process exi...2 Oct 06 14:38:58 ovman systemd[1]: Failed to mount /var/log. Oct 06 14:38:58 ovman systemd[1]: Unit var-log.mount entered fail.... Hint: Some lines were ellipsized, use -l to show in full. I need to recover from this quick. Sumit Basu

1 0

keycloak Active Directory BufferOverFlowException
by karl.morgan＠gmail.com 05 Oct '23

05 Oct '23

been trying to get a configuration working with ovirt 4.5.4 and keycloak and windows Active Directory I have had partial success in that i can with a little fiddling use ldap authentication for users. The fiddling involves going in and adding specific permissions to each ID after the first attempt has been made. I am now trying to get groups working such that I can have an AD user group automatically determine a users capabilities dependent on inclusion in specific groups. I have keycloak configured where, from within keycloak I can list users, And i can see which groups a user is a member of. It looks like is should be working. However, in Ovirt im getting invalid data error in the web and logs print the following ==> error_log <== [Thu Oct 05 09:22:58.037130 2023] [proxy_ajp:error] [pid 75583:tid 140622648301312] AH03229: ajp_msg_append_cvt_string(): BufferOverflowException 4 5694 ==> ssl_error_log <== [Thu Oct 05 09:22:58.037162 2023] [proxy_ajp:error] [pid 75583:tid 140622648301312] [client 172.28.96.180:43980] AH00971: ajp_marshal_into_msgb: Error appending the header value, referer: https://ov2ctl01-mn.internal.shutterfly.com/ovirt-engine/ [Thu Oct 05 09:22:58.037171 2023] [proxy_ajp:error] [pid 75583:tid 140622648301312] [client 172.28.96.180:43980] AH00988: ajp_send_header: ajp_marshal_into_msgb failed, referer: https://ov2ctl01-mn.internal.shutterfly.com/ovirt-engine/ [Thu Oct 05 09:22:58.037176 2023] [proxy_ajp:error] [pid 75583:tid 140622648301312] (120001)APR does not understand this error code: [client 172.28.96.180:43980] AH00868: request failed to 127.0.0.1:8702 (127.0.0.1), referer: https://ov2ctl01-mn.internal.shutterfly.com/ovirt-engine/ Pretty sure i need to be able to adjust a buffer size between keycloak and ajp module but am clueless how to proceed. Any help would be appreciated

1 0

How to obtain vm snapshots status
by anton.alymov＠cyberprotect.ru 04 Oct '23

04 Oct '23

Hi! I use ovirt rest api to start vm, backup vm and then remove vm. I start vm, wait for vmstatus up, then start backup, wait for starting, finalize, wait for succeeded, wait for disk unlock. Looks like backup is finished here from my side.Because ovirt repost succeed status and unlocks disk. But if i try shutdown and remove vm ovirt will throw error Cannot remove VM. The VM is performing an operation on a Snapshot. Please wait for the operation to finish, and try again. Ok, ovirt is right here, I see from web interface that operation hasn't finished yet. How can I obtain correct status where vm can be removed? I also tried to get info about vm snapshots but all of them had Status: ok

4 3

Cannot remove template because a disk is based on it
by nicolas＠devels.es 04 Oct '23

04 Oct '23

Hi, We're running oVirt 4.5. We have a template which we'd like to get rid of, there are no VMs based on it. However, trying to remove it oVirt states that: Cannot remove Template. The following Disk(s) are based on it: (b54ee1cb-ed64-4db4-bd3d-eac8b22ea095) . When opening the 'Disks' subtab in the template, it appears a disk (screenshot attached). However, when opening the oVirt Storage->Disks option, I cannot find the disk by the ID, not even ordering the ID column and trying to find it visually. Can anyone point to the problem and a possible solution/workaround? Thanks.

2 4

Image upload paused by system
by muchiri.maina＠gmail.com 03 Oct '23

03 Oct '23

I recently renewed my certs for the manager and hosts in ovirt 4.4 now when i try to upload an iso it gets paused by system then the error ,Unable to upload image to disk d840c8a3-21f5-4ad9-bd64-f569a93b4e74 due to a network error. Ensure ovirt-engine's CA certificate is registered as a trusted CA in the browser. i have registered the CA certificate but still wont upload

1 0

What certs need monitoring for expiration for Ovirt hosts ?
by morgan cox 03 Oct '23

03 Oct '23

Hi. I am aware of the following certs - "/etc/pki/vdsm/certs/vdsmcert.pem" - "/etc/pki/vdsm/libvirt-spice/server-cert.pem" - "/etc/pki/vdsm/libvirt-vnc/server-cert.pem" - "/etc/pki/libvirt/clientcert.pem" - "/etc/pki/vdsm/libvirt-migrate/server-cert.pem" And are monitoring them to avoid certs being expired. We have a 3rd party cert/ca - do I also need to monitor the following - "/etc/pki/ovirt-vmconsole/ca.pub" - "/etc/pki/vdsm/certs/cacert.pem" - "/etc/pki/vdsm/libvirt-migrate/ca-cert.pem" - "/etc/pki/vdsm/libvirt-spice/ca-cert.pem" - "/etc/pki/vdsm/libvirt-vnc/ca-cert.pem" - "/etc/pki/CA/cacert.pem" If the CA is updated on the engine do the above ca certs get updated with an update or re-enroll ? Thanks

1 0

Re: host deploy problem
by David Slaine 27 Sep '23

27 Sep '23

installed python3.11-netaddr the following error appeared: FAILED! => {\"msg\": \"The conditional check 'elasticsearch_host == None or elasticsearch_host is undefined' failed. The error was: error while evaluating conditional (elasticsearch_host == None or elasticsearch_host is undefined): 'elasticsearch_host' is undefined. 'elasticsearch_host' is undefined\\n\\nThe error appears to be in '/usr/share/ansible/roles/oVirt.metrics/roles/ovirt_initial_validations/tasks/check_logging_collectors.yml': line 4, column 3, but may\\nbe elsewhere in the file depending on the exact syntax problem.\\n\\nThe offending line appears to be:\\n\\n\\n- name: If output plugin is elasticsearch, validate host address is set\\n ^ here\\n\"}", ср, 27 сент. 2023 г. в 10:54, David Slaine <david.slaine(a)gmail.com>: > Thanks for your answer > I installed python38-netaddr-0.8.0-2.el8.noarch on both servers(host and > engine), but it didn't help and the same error appears > > ср, 27 сент. 2023 г. в 08:56, David Slaine <david.slaine(a)gmail.com>: > >> Thanks for your answer >> I installed python38-netaddr-0.8.0-2.el8.noarch on both servers(host and >> engine), but it didn't help and the same error appears >> >> вт, 26 сент. 2023 г. в 22:12, Strahil Nikolov <hunter86_bg(a)yahoo.com>: >> >>> Install python3-netaddr or python38-netaddr package. >>> >>> Best Regards, >>> Strahil Nikolov >>> >>> >>> On Tuesday, September 26, 2023, 9:04 PM, David Slaine < >>> david.slaine(a)gmail.com> wrote: >>> >>> Ovirt-engine 4.5.4-1 >>> Rocky 8.8 >>> >>> Hello! >>> >>> problem with adding a host to the cluster, it fails with the following >>> message in /var/log/ovirt-engine/host-deploy/: >>> >>> FAILED! => {\"msg\": \"The conditional check 'cluster_switch == >>> \\\"ovs\\\" or (ovn_central is defined and ovn_central | ipaddr)' failed. >>> The error was: The ipaddr filter requires python's netaddr be installed on >>> the ansible controller\\n\\nThe error appears to be in >>> '/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-provider-ovn-driver/tasks/configure.yml': >>> line 3, column 5, but may\\nbe elsewhere in the file depending on the exact >>> syntax problem.\\n\\nThe offending line appears to be:\\n\\n- block:\\n - >>> name: Install ovs\\n ^ here\\n\"} >>> >>> how to fix this issue ? >>> thanks >>> _______________________________________________ >>> Users mailing list -- users(a)ovirt.org >>> To unsubscribe send an email to users-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/FDEAQXI7XHUL6… >>> >>>

1 0

host deploy problem
by David Slaine 26 Sep '23

26 Sep '23

Ovirt-engine 4.5.4-1 Rocky 8.8 Hello! problem with adding a host to the cluster, it fails with the following message in /var/log/ovirt-engine/host-deploy/: FAILED! => {\"msg\": \"The conditional check 'cluster_switch == \\\"ovs\\\" or (ovn_central is defined and ovn_central | ipaddr)' failed. The error was: The ipaddr filter requires python's netaddr be installed on the ansible controller\\n\\nThe error appears to be in '/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-provider-ovn-driver/tasks/configure.yml': line 3, column 5, but may\\nbe elsewhere in the file depending on the exact syntax problem.\\n\\nThe offending line appears to be:\\n\\n- block:\\n - name: Install ovs\\n ^ here\\n\"} how to fix this issue ? thanks

2 1

Unable to add KVM (ubuntu flavour)hypervisors on Ovirt engine.
by spp16121991＠gmail.com 26 Sep '23

26 Sep '23

Hi all, I am a noob in KVM. I am a VMware engineer. I have created a lab environment to where i have hosted KVM as nested hypervisors within esxi. I have a vsan environment. I want to create a similar KVM environment. hence, i have deployed 3 kvm hypervisors (nested) i have deployed Ovirt-manager ( equivalent to vcenter) to host/manage my KVM hypervisors. to deploy Ovirt-manager, i installed ovirt packages in centos VM. Because ovirt pacjages are not available in ubuntu, hence choose centos. next is i want to add kvm hypervisors on ovirt manager similar to how we add esxi hosts to vcenter. That is where i am failing. From my limited understanding in KVM, ovirt is looking for yum package manager , KVM are ubuntu flavoured, hence it has apt package manager. What my ask is, can this be fixed? is there a file i need to edit in ovirt-engine.? Can Ovirt-manager host KVM of ubuntu flavour? Please advise. Checked the ovirt logs: Here is the log excerpts: ---------------------------------------------- } ], "python3-oauthlib" : [ { "name" : "python3-oauthlib", "version" : "3.1.0-1ubuntu2", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-openssl" : [ { "name" : "python3-openssl", "version" : "19.0.0-1build1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-pexpect" : [ { "name" : "python3-pexpect", "version" : "4.6.0-1build1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-pkg-resources" : [ { "name" : "python3-pkg-resources", "version" : "45.2.0-1ubuntu0.1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-problem-report" : [ { "name" : "python3-problem-report", "version" : "2.20.11-0ubuntu27.27", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-ptyprocess" : [ { "name" : "python3-ptyprocess", "version" : "0.6.0-1ubuntu1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-pyasn1" : [ { "name" : "python3-pyasn1", "version" : "0.4.2-3build1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-pyasn1-modules" : [ { "name" : "python3-pyasn1-modules", "version" : "0.2.1-0.2build1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-pymacaroons" : [ { "name" : "python3-pymacaroons", "version" : "0.13.0-3", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-pyrsistent" : [ { "name" : "python3-pyrsistent", "version" : "0.15.5-1build1", "arch" : "amd64", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-requests" : [ { "name" : "python3-requests", "version" : "2.22.0-2ubuntu1.1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-requests-unixsocket" : [ { "name" : "python3-requests-unixsocket", "version" : "0.2.0-2", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-secretstorage" : [ { "name" : "python3-secretstorage", "version" : "2.3.1-2ubuntu1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-serial" : [ { "name" : "python3-serial", "version" : "3.4-5.1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-service-identity" : [ { "name" : "python3-service-identity", "version" : "18.1.0-5build1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-setuptools" : [ { "name" : "python3-setuptools", "version" : "45.2.0-1ubuntu0.1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-simplejson" : [ { "name" : "python3-simplejson", "version" : "3.16.0-2ubuntu2", "arch" : "amd64", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-six" : [ { "name" : "python3-six", "version" : "1.14.0-2", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-software-properties" : [ { "name" : "python3-software-properties", "version" : "0.99.9.12", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-systemd" : [ { "name" : "python3-systemd", "version" : "234-3build2", "arch" : "amd64", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-twisted" : [ { "name" : "python3-twisted", "version" : "18.9.0-11ubuntu0.20.04.2", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-twisted-bin" : [ { "name" : "python3-twisted-bin", "version" : "18.9.0-11ubuntu0.20.04.2", "arch" : "amd64", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-update-manager" : [ { "name" : "python3-update-manager", "version" : "1:20.04.10.11", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], ============================== "duration" : 2.05353, "event_loop" : null, "uuid" : "f72c96ea-2491-4612-807e-37da9d563d4d" } } } 2023-09-15 03:18:41 EDT - TASK [ovirt-host-deploy-facts : Check if vdsm is preinstalled] ***************** 2023-09-15 03:18:41 EDT - ok: [10.125.0.110] 2023-09-15 03:18:41 EDT - { "status" : "OK", "msg" : "", "data" : { "uuid" : "89475143-4bb1-41ec-806d-53def5431773", "counter" : 23, "stdout" : "ok: [10.125.0.110]", "start_line" : 18, "end_line" : 19, "runner_ident" : "137dc514-5398-11ee-8c1e-00505686a372", "event" : "runner_on_ok", "pid" : 498377, "created" : "2023-09-15T07:18:39.085171", "parent_uuid" : "00505686-a372-da37-7bda-00000000016b", ============================================================ "task_action" : "lineinfile", "task_args" : "", "task_path" : "/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-host-deploy-facts/tasks/host-os.yml:19", "role" : "ovirt-host-deploy-facts", "host" : "10.125.0.110", "remote_addr" : "10.125.0.110", "res" : { "changed" : true, "msg" : "line added", "backup" : "", "diff" : [ { "before" : "", "after" : "", "before_header" : "/etc/os-release (content)", "after_header" : "/etc/os-release (content)" }, { "before_header" : "/etc/os-release (file attributes)", "after_header" : "/etc/os-release (file attributes)" } ], "invocation" : { "module_args" : { ============================================================= "duration" : 0.094784, "event_loop" : "items", "uuid" : "c29c53ba-c732-4237-bb6b-decac24d15b1" } } } 2023-09-15 03:18:41 EDT - TASK [ovirt-host-deploy-vdsm : Install ovirt-hosted-engine-setup package] ****** 2023-09-15 03:18:41 EDT - fatal: [10.125.0.110]: FAILED! => {"changed": false, "msg": ["Could not detect which major revision of yum is in use, which is required to determine module backend.", "You can manually specify use_backend to tell the module whether to use the yum (yum3) or dnf (yum4) backend})"]} 2023-09-15 03:18:41 EDT - { "status" : "OK", "msg" : "", "data" : { "uuid" : "f730ac20-fe5b-4240-8074-f5c04dddeeec", "counter" : 78, "stdout" : "fatal: [10.125.0.110]: FAILED! => {\"changed\": false, \"msg\": [\"Could not detect which major revision of yum is in use, which is required to determine module backend.\", \"You can manually specify use_backend to tell the module whether to use the yum (yum3) or dnf (yum4) backend})\"]}", "start_line" : 67, "end_line" : 68, "runner_ident" : "137dc514-5398-11ee-8c1e-00505686a372", "event" : "runner_on_failed", "pid" : 498377, "created" : "2023-09-15T07:18:41.351584", "parent_uuid" : "00505686-a372-da37-7bda-0000000001ba", "event_data" : { "playbook" : "ovirt-host-deploy.yml", "playbook_uuid" : "ba8a6834-12cf-4837-ae46-f70182bbad00", "play" : "all", "play_uuid" : "00505686-a372-da37-7bda-000000000008", "play_pattern" : "all", "task" : "Install ovirt-hosted-engine-setup package", "task_uuid" : "00505686-a372-da37-7bda-0000000001ba", "task_action" : "yum", "task_args" : "", "task_path" : "/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-host-deploy-vdsm/tasks/packages.yml:6", "role" : "ovirt-host-deploy-vdsm", "host" : "10.125.0.110", "remote_addr" : "10.125.0.110", "res" : { "msg" : [ "Could not detect which major revision of yum is in use, which is required to determine module backend.", "You can manually specify use_backend to tell the module whether to use the yum (yum3) or dnf (yum4) backend})" ], "_ansible_no_log" : false, "changed" : false

2 1

Impossibile to install a new system
by Andrea Chierici 22 Sep '23

22 Sep '23

Dear all, I've been struggling with a new ovirt installation for several weeks but I seem to have no luck at all. I've tried both starting from a centos8 stream system and a ovirt-node, but the engine installation always breaks here: TASK [ovirt.ovirt.hosted_engine_setup : Wait for the host to be up] I keep getting the same problem, it looks like the engine is not "migrated" to the public ip and is stuck on the natted address that the installation process is using during the preliminary phase. Here is a message that seems to confirm my theory: 2023-09-19 15:43:29,259+0200 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'msg': "The task includes an option with an undefined variable. The error was: 'local_vm_ip' is undefined\n\nThe error appears to be in '/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_engine_setup/tasks/sync_on_engine_machine.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Set the name for add_host\n ^ here\n", '_ansible_no_log': False} 2 I know the problem the development of ovirt is facing lately, but I am simply wondering: is there a way to install ovirt 4.5.4 today or is it all already lost??! Any kind of help would be really appreciated. Thanks, Andrea -- Andrea Chierici - INFN-CNAF Viale Berti Pichat 6/2, 40127 BOLOGNA Office Tel: +39 051 2095463 SkypeID ataruz --

3 4

ovirt self-hosted engine 4.5 with cluster version 4.6
by Николаев Алексей 20 Sep '23

20 Sep '23

1 0

In-place upgrade of Windows OS
by Hean-Seng Tan 19 Sep '23

19 Sep '23

We are unable to do an in-place upgrade of the Windows OS for a vm from Windows Server 2012r2 to version 2016. Has anyone ever performed successfully an in-place upgrade? Can you please share the steps?

1 0

oVirt 4.5.4-1: engine-setup does not renew certificates
by Lars Stolpe 15 Sep '23

15 Sep '23

Hi, I wanted to add short names to the webgui apache certificate. I followed these instructions: „Engine can be accessed using alternate host names (or IP addresses) that can configured by adding a new configuration file (for example /etc/ovirt-engine/engine.conf.d/99-custom-sso-setup.conf) with following content: SSO_ALTERNATE_ENGINE_FQDNS="alias1.example.com alias2.example.com" The list of alternate names has to be listed separated by spaces. It's possible to add also IP addresses of engine host, but using IP addresses instead of DNS names is not considered to be a good practise. Run engine-setup to generate new certificates.“ Certificates where not renewed. How do i get engine-setup to renew the apache certificate? oVirt: 4.5.4-1 Regards, Lars

2 2