- Users - oVirt List Archives

No default gateway defined
by gbv.kris＠mail.ru 22 Oct '23

22 Oct '23

When deploying isolated nested virtualization, we received the error "default gateway is not defined". Two virtual machines were created in the zVirt virtualization environment: the first OS AstraLinux, the second OS zvirtNode. VM are connected to each other by a virtual network. After starting the installation in Hosted Engine mode using the command line, this error was received. The setup was performed without DNS and the default gateway

2 1

moving engine VM outside one ovirt environment
by vishal.merrit＠gmail.com 21 Oct '23

21 Oct '23

What is the best way and steps to move engine VM outside the Ovirt environment. As of now the engine VM is controlling the host but running also on the same Ovirt host. VM must run on a different Ovirt environment. Want to move/migrate engine from hosted engine to stand alone VM

2 1

VDSM host_devops command HSMGetAllTasksStatusesVDS failed: Not SPM: ()
by tapz.malz＠gmail.com 21 Oct '23

21 Oct '23

Hi All, We are facing one issue in our environment. After importing a .ova image from Virtualbox to KVM, when we tried to remove the imported vm, suddenly getting below errors in event VDSM host_devops command HSMGetAllTasksStatusesVDS failed: Not SPM: () Failed to activate Storage Domain domain_devops (Data Center datacenter_devops) by admin@internal-authz It is also observed that our datacenter is non-responsive. When we try to activate it, getting below error Failed Activating Storage Domain domain_devops on Data Center datacenter_devops Can somebody help?

3 2

Strange events during upgrade process to 4.5.4
by james.kump＠missouri.edu 21 Oct '23

21 Oct '23

I recently began managing an Ovirt cluster. It has 5 cpu nodes and 3 gpu nodes in two clusters. When I started it was at 4.2.8 with a standalone engine. I have henceforth been steadily upgrading it to 4.5.4. I followed the upgrade guide and was successful in upgrading all 9 nodes from 4.2 to 4.3.10. I then upgraded the engine to centos 8 with 4.4 engine. I then rebuilt each of the nodes in the two data clusters to centos 8 ovirt 4.4 using the ovirt node installer iso. They all joined the engine correctly. I then upgraded the compatibility to 4.3 in both clusters and rebooted all the running VM's to also be at 4.3 compatibility. I then upgraded the engine to 4.5.4 and proceeded to upgrade two nodes to 4.5.4. They were successful and reattached to the engine correctly. Here is where my issue arises. Any VM that I migrate to the 4.5.4 node from a 4.4.10 node migrates correctly and then within 5 minutes will automatically migrate back. If I force pin a VM it stays running. I have searched the logs but not found anything to explain such behavior. Any assistance or idea ?

2 1

Request to get get an account
by ReFea_IM[e] Original 17 Oct '23

17 Oct '23

Hello, I would like to participate in the oVirt translation project. How can this be done? Thank you.

2 1

[org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for <JsonRpcResponse id: "..."
by nicolas＠devels.es 17 Oct '23

17 Oct '23

Hi, We're running oVirt 4.5.4, recently we had a problem due to /var partition being full, and we realized the /var/log/ovirt-engine/engine.log was pretty huge, approximately 16GB in size, despite it's configured to rotate daily. Checking the log, we have a lot (about 53000 at the time of this mail) of WARN events in this form: 2023-10-17 03:34:51,347+01 WARN [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for <JsonRpcResponse id: "d6cc2209-3126-48e1-97c8-97ca84d00f3a" result: [{"statusTime":"3329996823","status":"Up","vmId":"0fad0aa4-ec91-4beb-82a3-5641f0da674f","vmName":"SYTW-Backend-aluXXXXXXXXX","vmType":"kvm","kvmEnable":"true","acpiEnable":"true","elapsedTime":"285568","monitorResponse":"0","clientIp":"","timeOffset":"2","cpuUser":"-0.04","cpuSys": "0.40","cpuUsage":"12369410000000","cpuActual":true,"network":{"vnet3":{"macAddr":"00:1a:4a:4d:cc:d5","name":"vnet3","speed":"1000","state" ... This is just a chunk of the full line, which is actually pretty big as it contains a lot of parameters. So about 53.000 times this long line makes the log huge. Why are we getting such a big amount of these WARNings and what can we do to solve them? Thanks.

1 0

Host reports about one of the Active Storage Domains as Problematic
by Matthew J Black 16 Oct '23

16 Oct '23

Hi All, I'm hoping someone(s) can point me in the correct direction to fix this. The Architecture +++++++++++++ 3 identical VM-Hosts (vmhost{1-3}.mydomain.local): - Rocky Linux 8.8 - 4 local gluster volumes - 4 iscsi volumes provided by 3-node Ceph Cluster (also running Rocky v8.8) - 3 NICS: - One Gluster Transfer - One Ceph Data Transfer - One General Data access / OVirt management Each node of the Ceph Cluster is running an iscsi gateway, and each vmhost connects to each iscsi gateway Everything is working AOK, except vmhost1 complains that it can't reach 3 of the 4 iscsi volumes/domains. I've looked in the engine.log and vdsm.log but can't figure out how to fix things, and I don't know where else to look. engine.log (extract) +++++++++++++++ ~~~ 2023-10-16 18:21:08,892+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-27) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Lock Acquired to object 'EngineLock:{exclusiveLocks='[8e0e99ab-c212-41c4-8318-acd34acb6289=VDS]', sharedLocks=''}' 2023-10-16 18:21:09,179+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2023-10-16 18:21:09,179+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Before acquiring lock in order to prevent monitoring for host 'vmhost1.mydomain.local' from data-center 'my_datacentre' 2023-10-16 18:21:09,179+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Lock acquired, from now a monitoring of host will be skipped for host 'vmhost1.mydomain.local' from data-center 'my_datacentre' 2023-10-16 18:21:09,181+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] START, SetVdsStatusVDSCommand(HostName = vmhost1.mydomain.local, SetVdsStatusVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 732b1116 2023-10-16 18:21:09,245+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] FINISH, SetVdsStatusVDSCommand, return: , log id: 732b1116 2023-10-16 18:21:09,248+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Activate host finished. Lock released. Monitoring can run now for host 'vmhost1.mydomain.local' from data-center 'my_datacentre' 2023-10-16 18:21:09,314+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] EVENT_ID: VDS_ACTIVATE(16), Activation of host vmhost1.mydomain.local initiated by admin@internal-authz. 2023-10-16 18:21:09,373+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Lock freed to object 'EngineLock:{exclusiveLocks='[8e0e99ab-c212-41c4-8318-acd34acb6289=VDS]', sharedLocks=''}' 2023-10-16 18:21:10,620+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [] START, GetHardwareInfoAsyncVDSCommand(HostName = vmhost1.mydomain.local, VdsIdAndVdsVDSCommandParametersBase:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', vds='Host[vmhost1.mydomain.local,8e0e99ab-c212-41c4-8318-acd34acb6289]'}), log id: 3dac178f 2023-10-16 18:21:10,620+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [] FINISH, GetHardwareInfoAsyncVDSCommand, return: , log id: 3dac178f 2023-10-16 18:21:10,688+11 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [44be9d7f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS 2023-10-16 18:21:10,690+11 INFO [org.ovirt.engine.core.bll.UpdateClusterCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] Lock Acquired to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[]'}' 2023-10-16 18:21:10,703+11 INFO [org.ovirt.engine.core.bll.UpdateClusterCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] Running command: UpdateClusterCommand internal: true. Entities affected : ID: c892710d-8611-4146-aea2-05d910f1829b Type: ClusterAction group EDIT_CLUSTER_CONFIGURATION with role type ADMIN 2023-10-16 18:21:10,761+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] EVENT_ID: SYSTEM_UPDATE_CLUSTER(835), Host cluster my_cluster was updated by system 2023-10-16 18:21:10,761+11 INFO [org.ovirt.engine.core.bll.UpdateClusterCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] Lock freed to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[]'}' 2023-10-16 18:21:10,965+11 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [5f0caae0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 7bfaa890-f892-4b3b-bf18-33dabc13e646 Type: StoragePool 2023-10-16 18:21:11,032+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [5f0caae0] START, SetHaMaintenanceModeVDSCommand(HostName = vmhost1.mydomain.local, SetHaMaintenanceModeVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289'}), log id: 630966ec 2023-10-16 18:21:11,034+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [5f0caae0] FINISH, SetHaMaintenanceModeVDSCommand, return: , log id: 630966ec 2023-10-16 18:21:11,170+11 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 7bfaa890-f892-4b3b-bf18-33dabc13e646 Type: StoragePool 2023-10-16 18:21:11,196+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] START, ConnectStorageServerVDSCommand(HostName = vmhost1.mydomain.local, StorageServerConnectionManagementVDSParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='8d7b930e-a758-4010-aaf5-a230447f2761', connection='vmhosttran1.mydomain.local:/gv0', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local:vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 30a928ed 2023-10-16 18:21:11,211+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] FINISH, ConnectStorageServerVDSCommand, return: {8d7b930e-a758-4010-aaf5-a230447f2761=0}, log id: 30a928ed 2023-10-16 18:21:11,211+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] START, ConnectStorageServerVDSCommand(HostName = vmhost1.mydomain.local, StorageServerConnectionManagementVDSParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='044a086b-e94b-4e5d-a118-283ffceb8de8', connection='vmhosttran1.mydomain.local:gv1', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local:vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='2d328788-ade0-4e25-b947-1c392d4f133e', connection='vmhosttran1.mydomain.local:/gv2', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local :vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='420b9d49-eebb-4dea-99bb-bcb89fbe2bf2', connection='vmhosttran1.mydomain.local:/gv3', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local:vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 6aaeee49 2023-10-16 18:21:11,664+11 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [] Fetched 0 VMs from VDS '8e0e99ab-c212-41c4-8318-acd34acb6289' 2023-10-16 18:21:11,917+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] FINISH, ConnectStorageServerVDSCommand, return: {044a086b-e94b-4e5d-a118-283ffceb8de8=0, 2d328788-ade0-4e25-b947-1c392d4f133e=0, 420b9d49-eebb-4dea-99bb-bcb89fbe2bf2=0}, log id: 6aaeee49 2023-10-16 18:21:11,920+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] START, ConnectStorageServerVDSCommand(HostName = vmhost1.mydomain.local, StorageServerConnectionManagementVDSParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', storageType='ISCSI', connectionList='[StorageServerConnections:{id='78f16782-9a06-45f2-92e5-d1ca84a8c67e', connection='192.168.4.24', iqn='iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='868d7c6a-c933-4475-86ce-ab1359d6f4c3', connection='192.168.4.25', iqn='iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='20a52c08-a4c5-4b07-8ecc-bec0e82e582a', connection='192.168.4.23', iqn='iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 4e953edb 2023-10-16 18:21:19,245+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='TPM'}), log id: 694529cf 2023-10-16 18:21:19,247+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@a9992e5, log id: 694529cf 2023-10-16 18:21:19,247+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='NVRAM'}), log id: 786d585b 2023-10-16 18:21:19,248+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@1cf19aa, log id: 786d585b 2023-10-16 18:21:19,248+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='TPM'}), log id: 4104430d 2023-10-16 18:21:19,250+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@7ccef72, log id: 4104430d 2023-10-16 18:21:19,250+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='NVRAM'}), log id: 15f1c050 2023-10-16 18:21:19,251+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@6c8a990d, log id: 15f1c050 2023-10-16 18:21:34,401+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='TPM'}), log id: 3f508f9e 2023-10-16 18:21:34,403+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@5690567a, log id: 3f508f9e 2023-10-16 18:21:34,403+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='NVRAM'}), log id: 270bce0b 2023-10-16 18:21:34,404+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@2a7c9b44, log id: 270bce0b 2023-10-16 18:21:34,404+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='TPM'}), log id: 11ebddea 2023-10-16 18:21:34,406+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@459c7a2b, log id: 11ebddea 2023-10-16 18:21:34,406+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='NVRAM'}), log id: 3c1e9f05 2023-10-16 18:21:34,408+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@72f69579, log id: 3c1e9f05 2023-10-16 18:21:49,594+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='TPM'}), log id: 6a366a0f 2023-10-16 18:21:49,597+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@95edf66, log id: 6a366a0f 2023-10-16 18:21:49,655+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='TPM'}), log id: 40d156 2023-10-16 18:21:49,656+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@3f11df90, log id: 40d156 2023-10-16 18:23:28,802+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] FINISH, ConnectStorageServerVDSCommand, return: {78f16782-9a06-45f2-92e5-d1ca84a8c67e=465, 868d7c6a-c933-4475-86ce-ab1359d6f4c3=465, 20a52c08-a4c5-4b07-8ecc-bec0e82e582a=0}, log id: 4e953edb 2023-10-16 18:23:28,811+11 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] Host 'vmhost1.mydomain.local' storage connection was succeeded 2023-10-16 18:23:28,846+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] START, ConnectStoragePoolVDSCommand(HostName = vmhost1.mydomain.local, ConnectStoragePoolVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', vdsId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', masterVersion='1'}), log id: 3b47a386 2023-10-16 18:23:28,846+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Executing with domain map: {e311ddf1-7f2c-49ef-a618-050d9a2b947f=active, fa8d922a-df3c-4f5b-adf6-a77c3689c397=active, 30775056-38ef-483a-889c-70453e9aa5f9=active, ee5b0268-8141-43e0-995e-8b4586217123=active, 3e33b7ce-7279-4765-a203-cca37eff8cb0=active, 4412e492-f752-4a6c-98ff-f289043618da=active, 0dde39a4-42e4-4fba-8068-e63e42dd3334=active, d44ec7f2-3629-4110-93a5-7d8d57230efc=active} 2023-10-16 18:23:32,385+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 3b47a386 2023-10-16 18:23:32,418+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '0dde39a4-42e4-4fba-8068-e63e42dd3334:iscsi-vmimages' report isn't an actual report 2023-10-16 18:23:32,419+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '3e33b7ce-7279-4765-a203-cca37eff8cb0:gluster-generaldata' report isn't an actual report 2023-10-16 18:23:32,419+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain 'e311ddf1-7f2c-49ef-a618-050d9a2b947f:gluster-vmimages' report isn't an actual report 2023-10-16 18:23:32,419+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '4412e492-f752-4a6c-98ff-f289043618da:gluster-isos' report isn't an actual report 2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain 'fa8d922a-df3c-4f5b-adf6-a77c3689c397:iscsi-databases' was reported with error code '358' 2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '30775056-38ef-483a-889c-70453e9aa5f9:iscsi-generaldata' was reported with error code '358' 2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain 'd44ec7f2-3629-4110-93a5-7d8d57230efc:iscsi-backups' was reported with error code '358' 2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Storage Domain 'iscsi-databases' of pool 'my_datacentre' is in problem in host 'vmhost1.mydomain.local' 2023-10-16 18:23:32,420+11 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Storage Domain 'iscsi-generaldata' of pool 'my_datacentre' is in problem in host 'vmhost1.mydomain.local' 2023-10-16 18:23:32,420+11 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Storage Domain 'iscsi-backups' of pool 'my_datacentre' is in problem in host 'vmhost1.mydomain.local' 2023-10-16 18:23:32,454+11 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] EVENT_ID: VDS_STORAGE_VDS_STATS_FAILED(189), Host vmhost1.mydomain.local reports about one of the Active Storage Domains as Problematic. 2023-10-16 18:23:32,564+11 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS 2023-10-16 18:23:32,566+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] START, SetVdsStatusVDSCommand(HostName = vmhost1.mydomain.local, SetVdsStatusVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 992c9e 2023-10-16 18:23:32,566+11 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] Clearing domains data for host vmhost1.mydomain.local 2023-10-16 18:23:32,619+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] FINISH, SetVdsStatusVDSCommand, return: , log id: 992c9e 2023-10-16 18:23:32,689+11 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Host vmhost1.mydomain.local cannot access the Storage Domain(s) iscsi-databases, iscsi-generaldata, iscsi-backups attached to the Data Center my_datacentre. Setting Host state to Non-Operational. 2023-10-16 18:23:32,789+11 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] EVENT_ID: VDS_ALERT_FENCE_TEST_FAILED(9,001), Power Management test failed for Host vmhost1.mydomain.local.There is no other host in the data center that can be used to test the power management settings. 2023-10-16 18:23:32,856+11 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Failed to connect Host vmhost1.mydomain.local to Storage Pool my_datacentre 2023-10-16 18:23:33,016+11 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [f7474e7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS 2023-10-16 18:23:33,018+11 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [f7474e7] Host 'vmhost1.mydomain.local'(8e0e99ab-c212-41c4-8318-acd34acb6289) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. 2023-10-16 18:23:33,019+11 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [f7474e7] Received first domain report for host vmhost1.mydomain.local 2023-10-16 18:24:02,559+11 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2023-10-16 18:24:02,561+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] START, SetVdsStatusVDSCommand(HostName = vmhost1.mydomain.local, SetVdsStatusVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', status='PreparingForMaintenance', nonOperationalReason='NONE', stopSpmFailureLogged='true', maintenanceReason='null'}), log id: 51d95bef 2023-10-16 18:24:02,592+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] FINISH, SetVdsStatusVDSCommand, return: , log id: 51d95bef 2023-10-16 18:24:02,799+11 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS 2023-10-16 18:24:02,827+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] START, SetHaMaintenanceModeVDSCommand(HostName = vmhost1.mydomain.local, SetHaMaintenanceModeVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289'}), log id: 63f61978 2023-10-16 18:24:02,832+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] FINISH, SetHaMaintenanceModeVDSCommand, return: , log id: 63f61978 2023-10-16 18:24:02,914+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] EVENT_ID: USER_VDS_MAINTENANCE_WITHOUT_REASON(620), Host vmhost1.mydomain.local was switched to Maintenance mode by admin@internal-authz. ~~~ vdsm.log (extract - from vmhost1) ++++++++++++++++++++++++++ ~~~ 2023-10-16 18:21:10,069+1100 INFO (jsonrpc/7) [api.host] START getCapabilities() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:10,410+1100 ERROR (jsonrpc/7) [root] Error getting managedvolume connector info: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc=1 out=b\'\' err=b\'Traceback (most recent call last):\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 91, in <module>\\nfrom importlib import resources as _importlib_resources\\nImportError: cannot import name \\\'resources\\\'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File "/usr/libexec/vdsm/managedvolume-helper", line 26, in <module>\\nfrom os_brick.initiator import connector\\n File "/usr/lib/python3.6/site-packages/os_brick/initiator/connector.py", line 27, in <module>\\nfrom oslo_concurrency import lockutils\\n File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 29, in <module>\\nfrom oslo_config imp ort cfg\\n File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 45, in <module>\\nfrom oslo_config import types\\n File "/usr/lib/python3.6/site-packages/oslo_config/types.py", line 29, in <module>\\nimport netaddr\\n File "/usr/lib/python3.6/site-packages/netaddr/__init__.py", line 18, in <module>\\nfrom netaddr.core import (AddrConversionError, AddrFormatError,\\n File "/usr/lib/python3.6/site-packages/netaddr/core.py", line 11, in <module>\\nfrom netaddr.compat import _callable, _iter_dict_keys\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 93, in <module>\\nimport importlib_resources as _importlib_resources\\nModuleNotFoundError: No module named \\\'importlib_resources\\\'\\n\'',) (caps:165) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/host/caps.py", line 161, in get caps["connector_info"] = managedvolume.connector_info() File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 63, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 78, in connector_info return run_helper("connector_info") File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 184, in run_helper sub_cmd, cmd_input=cmd_input) File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 38, in __call__ return callMethod() File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 36, in <lambda> **kwargs) File "<string>", line 2, in managedvolume_run_helper File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod raise convert_to_error(kind, result) vdsm.storage.exception.ManagedVolumeHelperFailed: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc=1 out=b\'\' err=b\'Traceback (most recent call last):\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 91, in <module>\\nfrom importlib import resources as _importlib_resources\\nImportError: cannot import name \\\'resources\\\'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File "/usr/libexec/vdsm/managedvolume-helper", line 26, in <module>\\nfrom os_brick.initiator import connector\\n File "/usr/lib/python3.6/site-packages/os_brick/initiator/connector.py", line 27, in <module>\\nfrom oslo_concurrency import lockutils\\n File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 29, in <module>\\nfrom oslo_config import cfg\\n File "/usr/lib/python3.6/site-packag es/oslo_config/cfg.py", line 45, in <module>\\nfrom oslo_config import types\\n File "/usr/lib/python3.6/site-packages/oslo_config/types.py", line 29, in <module>\\nimport netaddr\\n File "/usr/lib/python3.6/site-packages/netaddr/__init__.py", line 18, in <module>\\nfrom netaddr.core import (AddrConversionError, AddrFormatError,\\n File "/usr/lib/python3.6/site-packages/netaddr/core.py", line 11, in <module>\\nfrom netaddr.compat import _callable, _iter_dict_keys\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 93, in <module>\\nimport importlib_resources as _importlib_resources\\nModuleNotFoundError: No module named \\\'importlib_resources\\\'\\n\'',) 2023-10-16 18:21:10,600+1100 INFO (jsonrpc/7) [root] /usr/libexec/vdsm/hooks/after_get_caps/ovirt_provider_ovn_hook: rc=0 err=b'' (hooks:106) 2023-10-16 18:21:10,601+1100 INFO (jsonrpc/7) [api.host] FINISH getCapabilities return={'status': {'code': 0, 'message': 'Done'}, 'info': {'kvmEnabled': 'true', 'cpuCores': '16', 'cpuThreads': '32', 'cpuSockets': '2', 'onlineCpus': '0,1,2,3,4,5,6,7,16,17,18,19,20,21,22,23,8,9,10,11,12,13,14,15,24,25,26,27,28,29,30,31', 'cpuTopology': [{'cpu_id': 0, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 0}, {'cpu_id': 1, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 1}, {'cpu_id': 2, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 2}, {'cpu_id': 3, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 3}, {'cpu_id': 4, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 4}, {'cpu_id': 5, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 5}, {'cpu_id': 6, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 6}, {'cpu_id': 7, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 7}, {'cpu_id': 16, 'numa_cell_id': 0, 'socket_id ': 0, 'die_id': 0, 'core_id': 0}, {'cpu_id': 17, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 1}, {'cpu_id': 18, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 2}, {'cpu_id': 19, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 3}, {'cpu_id': 20, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 4}, {'cpu_id': 21, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 5}, {'cpu_id': 22, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 6}, {'cpu_id': 23, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 7}, {'cpu_id': 8, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 0}, {'cpu_id': 9, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 1}, {'cpu_id': 10, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 2}, {'cpu_id': 11, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 3}, {'cpu_id': 12, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 4}, {'cpu_id': 13, 'numa_cell_id' : 1, 'socket_id': 1, 'die_id': 0, 'core_id': 5}, {'cpu_id': 14, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 6}, {'cpu_id': 15, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 7}, {'cpu_id': 24, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 0}, {'cpu_id': 25, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 1}, {'cpu_id': 26, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 2}, {'cpu_id': 27, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 3}, {'cpu_id': 28, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 4}, {'cpu_id': 29, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 5}, {'cpu_id': 30, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 6}, {'cpu_id': 31, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 7}], 'cpuSpeed': '4000.000', 'cpuModel': 'Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz', 'cpuFlags': 'pku,clflushopt,avx2,nopl,pni,pln,dca,smep,pat,art,cx8,msr,pts,arch_capa bilities,xsave,ht,lm,smap,de,mce,pse36,cx16,skip-l1dfl-vmentry,cpuid,bts,lahf_lm,avx512vl,tsc_deadline_timer,bmi2,md-clear,tm2,cmov,sep,sse4_1,invtsc,md_clear,intel_pt,amd-ssbd,mpx,ssse3,rdt_a,cpuid_fault,x2apic,mmx,avx512dq,dtes64,avx512cd,vme,pclmulqdq,cqm_occup_llc,xsavec,ibpb,umip,pschange-mc-no,mtrr,avx512f,stibp,clwb,xsaves,f16c,invpcid_single,avx512bw,arat,pdpe1gb,ibrs,ept,tm,invpcid,cqm_mbm_total,ibrs-all,3dnowprefetch,popcnt,cqm,rdctl-no,xsaveopt,pdcm,aperfmperf,cat_l3,xgetbv1,xtpr,syscall,smx,pbe,sse2,amd-stibp,pebs,xtopology,tsc_adjust,mca,pae,ida,nonstop_tsc,tpr_shadow,ssbd,nx,tsc,dtherm,aes,fxsr,fma,cqm_mbm_local,movbe,erms,intel_ppin,mds-no,ds_cpl,rdseed,dts,ibrs_enhanced,constant_tsc,ept_ad,flexpriority,avx512_vnni,adx,clflush,acpi,cdp_l3,sse,rdtscp,fsgsbase,ss,bmi1,arch_perfmon,monitor,pse,ospke,pcid,rdrand,hypervisor,flush_l1d,tsx-ctrl,arch-capabilities,vmx,avx,pge,rep_good,vpid,sdbg,sse4_2,mba,cqm_llc,est,epb,fpu,vnmi,apic,abm,spec_ctrl,model_pentium3,model_Penryn, model_Cascadelake-Server-noTSX,model_Haswell-noTSX-IBRS,model_Haswell-noTSX,model_kvm32,model_Nehalem-IBRS,model_pentium,model_Opteron_G1,model_IvyBridge-IBRS,model_Westmere,model_n270,model_qemu32,model_coreduo,model_486,model_IvyBridge,model_SandyBridge,model_Nehalem,model_Skylake-Client-noTSX-IBRS,model_pentium2,model_Westmere-IBRS,model_SandyBridge-IBRS,model_Conroe,model_qemu64,model_Opteron_G2,model_kvm64,model_Broadwell-noTSX,model_Skylake-Server-noTSX-IBRS,model_Broadwell-noTSX-IBRS,model_core2duo', 'vdsmToCpusAffinity': [1], 'version_name': 'Snow Man', 'software_version': '4.50.3.4', 'software_revision': '1', 'supportedENGINEs': ['4.2', '4.3', '4.4', '4.5'], 'clusterLevels': ['4.2', '4.3', '4.4', '4.5', '4.6', '4.7'], 'networks': {'ovirtmgmt': {'ports': ['bond0.20'], 'stp': 'off', 'iface': 'ovirtmgmt', 'bridged': True, 'addr': '192.168.2.20', 'netmask': '255.255.255.0', 'ipv4addrs': ['192.168.2.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.2.1', 'ipv6 gateway': '::', 'ipv4defaultroute': True, 'mtu': '1500', 'switch': 'legacy', 'southbound': 'bond0.20', 'vlanid': 20, 'dhcpv4': False, 'dhcpv6': False}, 'DMZ': {'ports': ['bond0.10'], 'stp': 'off', 'iface': 'DMZ', 'bridged': True, 'addr': '192.168.1.20', 'netmask': '255.255.255.0', 'ipv4addrs': ['192.168.1.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.1.1', 'ipv6gateway': '::', 'ipv4defaultroute': False, 'mtu': '1500', 'switch': 'legacy', 'southbound': 'bond0.10', 'vlanid': 10, 'dhcpv4': False, 'dhcpv6': False}, 'Migration': {'interface': 'ens1', 'iface': 'ens1', 'bridged': False, 'addr': '192.168.5.20', 'netmask': '255.255.255.0', 'ipv4addrs': ['192.168.5.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'ipv4defaultroute': False, 'mtu': '9000', 'switch': 'legacy', 'southbound': 'ens1', 'dhcpv4': False, 'dhcpv6': False}, 'SAN': {'interface': 'bond0.40', 'iface': 'bond0.40', 'bridged': False, 'addr': '192.168.4.20', 'netmask': '255.255.255.0', 'ipv4addrs': ['192.168.4.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'ipv4defaultroute': False, 'mtu': '9000', 'switch': 'legacy', 'southbound': 'bond0.40', 'vlanid': 40, 'dhcpv4': False, 'dhcpv6': False}}, 'bondings': {'bond0': {'hwaddr': 'ac:1f:6b:3c:cc:e8', 'slaves': ['eno1', 'eno2'], 'active_slave': '', 'opts': {'updelay': '1000', 'mode': '4', 'miimon': '1000', 'downdelay': '1000'}, 'ad_aggregator_id': '1', 'ad_partner_mac': '10:b3:c6:ae:a8:2d', 'switch': 'legacy', 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'speed': 2000}}, 'bridges': {'ovirtmgmt': {'ports': ['bond0.20'], 'stp': 'off', 'addr': '192.168.2.20', 'ipv4addrs': ['192.168.2.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.2.1', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'ipv4defaultroute': True, 'dhcpv4': False, 'dhcpv6': False, 'opts': {'ageing_time': '30000', 'multicast_query_use_ifaddr': '0', 'gc_timer': '17569', 'hello_time': '200', 'multicast_router': '1', 'nf_call_iptables': '0', 'group_addr': '01:80:c2:00:00:00', 'group_fwd_mask': '0x0', 'multicast_querier': '0', 'hash_max': '4096', 'tcn_timer': '0', 'vlan_protocol': '0x8100', 'vlan_stats_per_port': '0', 'root_port': '0', 'vlan_filtering': '0', 'multicast_query_interval': '12500', 'bridge_id': '8000.ac1f6b3ccce8', 'max_age': '2000', 'nf_call_arptables': '0', 'multicast_startup_query_interval': '3125', 'multicast_stats_enabled': '0', 'multicast_query_response_interval': '1000', 'topology_change': '0', 'priority': '32768', 'multicast_mld_version': '1', 'hash_elasticity': '16', 'hello_timer': '0', 'default_pvid': '1', 'root_path_cost': '0', 'multicast_igmp_version': '2', 'stp_state': '0', 'multicast_startup_query_count': '2', 'topology_change_detected': '0', 'multicast_last_member_interval': '1 00', 'topology_change_timer': '0', 'root_id': '8000.ac1f6b3ccce8', 'no_linklocal_learn': '0', 'forward_delay': '1500', 'multicast_membership_interval': '26000', 'multicast_querier_interval': '25500', 'vlan_stats_enabled': '0', 'multicast_snooping': '1', 'nf_call_ip6tables': '0', 'multicast_last_member_count': '2'}}, 'DMZ': {'ports': ['bond0.10'], 'stp': 'off', 'addr': '192.168.1.20', 'ipv4addrs': ['192.168.1.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.1.1', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'opts': {'ageing_time': '30000', 'multicast_query_use_ifaddr': '0', 'gc_timer': '8995', 'hello_time': '200', 'multicast_router': '1', 'nf_call_iptables': '0', 'group_addr': '01:80:c2:00:00:00', 'group_fwd_mask': '0x0', 'multicast_querier': '0', 'hash_max': '4096', 'tcn_timer': '0', 'vlan_protocol': '0x8100', 'vlan_stats_per_port': '0', 'root_port': '0', 'vlan_filtering': '0', 'multi cast_query_interval': '12500', 'bridge_id': '8000.ac1f6b3ccce8', 'max_age': '2000', 'nf_call_arptables': '0', 'multicast_startup_query_interval': '3125', 'multicast_stats_enabled': '0', 'multicast_query_response_interval': '1000', 'topology_change': '0', 'priority': '32768', 'multicast_mld_version': '1', 'hash_elasticity': '16', 'hello_timer': '0', 'default_pvid': '1', 'root_path_cost': '0', 'multicast_igmp_version': '2', 'stp_state': '0', 'multicast_startup_query_count': '2', 'topology_change_detected': '0', 'multicast_last_member_interval': '100', 'topology_change_timer': '0', 'root_id': '8000.ac1f6b3ccce8', 'no_linklocal_learn': '0', 'forward_delay': '1500', 'multicast_membership_interval': '26000', 'multicast_querier_interval': '25500', 'vlan_stats_enabled': '0', 'multicast_snooping': '1', 'nf_call_ip6tables': '0', 'multicast_last_member_count': '2'}}, 'virbr0': {'ports': [], 'stp': 'on', 'addr': '192.168.122.1', 'ipv4addrs': ['192.168.122.1/24'], 'ipv6addrs': [], 'ipv6autoconf' : False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'opts': {'ageing_time': '30000', 'multicast_query_use_ifaddr': '0', 'gc_timer': '22103', 'hello_time': '200', 'multicast_router': '1', 'nf_call_iptables': '0', 'group_addr': '01:80:c2:00:00:00', 'group_fwd_mask': '0x0', 'multicast_querier': '0', 'hash_max': '4096', 'tcn_timer': '0', 'vlan_protocol': '0x8100', 'vlan_stats_per_port': '0', 'root_port': '0', 'vlan_filtering': '0', 'multicast_query_interval': '12500', 'bridge_id': '8000.525400a8ef1c', 'max_age': '2000', 'nf_call_arptables': '0', 'multicast_startup_query_interval': '3125', 'multicast_stats_enabled': '0', 'multicast_query_response_interval': '1000', 'topology_change': '0', 'priority': '32768', 'multicast_mld_version': '1', 'hash_elasticity': '16', 'hello_timer': '103', 'default_pvid': '1', 'root_path_cost': '0', 'multicast_igmp_version': '2', 'stp_state': '1', 'multicast_start up_query_count': '2', 'topology_change_detected': '0', 'multicast_last_member_interval': '100', 'topology_change_timer': '0', 'root_id': '8000.525400a8ef1c', 'no_linklocal_learn': '0', 'forward_delay': '200', 'multicast_membership_interval': '26000', 'multicast_querier_interval': '25500', 'vlan_stats_enabled': '0', 'multicast_snooping': '1', 'nf_call_ip6tables': '0', 'multicast_last_member_count': '2'}}}, 'nics': {'ens1': {'hwaddr': '00:1b:21:e4:a6:f4', 'addr': '192.168.5.20', 'ipv4addrs': ['192.168.5.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'speed': 1000}, 'eno1': {'hwaddr': 'ac:1f:6b:3c:cc:e8', 'ad_aggregator_id': '1', 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'ipv4defaultroute': False, 'permhwaddr': 'ac:1f:6b:3c:cc:e8', 'dhcpv4': False, 'dhcpv 6': False, 'speed': 1000}, 'eno2': {'hwaddr': 'ac:1f:6b:3c:cc:e8', 'ad_aggregator_id': '1', 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'ipv4defaultroute': False, 'permhwaddr': 'ac:1f:6b:3c:cc:e9', 'dhcpv4': False, 'dhcpv6': False, 'speed': 1000}}, 'vlans': {'bond0.20': {'iface': 'bond0', 'vlanid': 20, 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False}, 'bond0.10': {'iface': 'bond0', 'vlanid': 10, 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False}, 'bond0.40': {'iface': 'bond0', 'vlanid': 40, 'addr': '192.168.4.20', 'ipv4addrs': ['192.168.4.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6 gateway': '::', 'mtu': '9000', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False}}, 'nameservers': ['192.168.2.16', '192.168.2.17'], 'supportsIPv6': True, 'ovnConfigured': False, 'hooks': {'after_get_caps': {'ovirt_provider_ovn_hook': {'checksum': 'a2bdefca38b96c8ddab39822cc8282bf3f67d875c4879003ffc9661826c92421'}}, 'before_device_create': {'10_ovirt_provider_ovn_hook': {'checksum': 'b9d67afb41bd86a521ada2121e3505297b3b6dfd478275ce0bd9501fcda6dafc'}}, 'before_nic_hotplug': {'10_ovirt_provider_ovn_hook': {'checksum': 'b9d67afb41bd86a521ada2121e3505297b3b6dfd478275ce0bd9501fcda6dafc'}}, 'before_vm_start': {'50_hostedengine': {'checksum': 'e5f5262cf22e06cd34e227afb27647e479351266876019a64210dbcbd2a43830'}}}, 'operatingSystem': {'release': '1.8.el8', 'version': '8.8', 'name': 'RHEL', 'pretty_name': 'Rocky Linux 8.8 (Green Obsidian)'}, 'uuid': '18e93400-808a-11ea-8000-ac1f6b3ccce8', 'packages2': {'kernel': {'version': '4.18.0', 'release': '477.27.1.e l8_8.x86_64'}, 'glusterfs-cli': {'version': '10.4', 'release': '1.el8s'}, 'librbd1': {'version': '16.2.13', 'release': '1.el8s'}, 'libvirt': {'version': '8.0.0', 'release': '19.2.module+el8.8.0+1321+70d049b2'}, 'mom': {'version': '0.6.4', 'release': '1.el8'}, 'ovirt-hosted-engine-ha': {'version': '2.5.0', 'release': '1.el8'}, 'openvswitch': {'version': '2.15', 'release': '4.el8'}, 'nmstate': {'version': '1.4.4', 'release': '2.el8_8'}, 'qemu-img': {'version': '6.2.0', 'release': '33.module+el8.8.0+1454+0b2cbfb8'}, 'qemu-kvm': {'version': '6.2.0', 'release': '33.module+el8.8.0+1454+0b2cbfb8'}, 'spice-server': {'version': '0.14.3', 'release': '4.el8'}, 'vdsm': {'version': '4.50.3.4', 'release': '1.el8'}, 'glusterfs': {'version': '10.4', 'release': '1.el8s'}, 'glusterfs-fuse': {'version': '10.4', 'release': '1.el8s'}, 'glusterfs-geo-replication': {'version': '10.4', 'release': '1.el8s'}, 'glusterfs-server': {'version': '10.4', 'release': '1.el8s'}}, 'realtimeKernel': False, 'kernelArgs' : 'BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-477.27.1.el8_8.x86_64 root=UUID=9acf17b0-558a-46d3-b21c-74c208680fcb ro crashkernel=auto resume=UUID=9c5f897c-8e34-48a7-9f06-786af3c079e9', 'nestedVirtualization': True, 'emulatedMachines': ['pc-q35-rhel8.0.0', 'pc-q35-rhel7.4.0', 'pc-i440fx-rhel7.3.0', 'pc-q35-rhel8.2.0', 'pc-q35-rhel7.3.0', 'pc-i440fx-rhel7.4.0', 'pc-q35-rhel8.4.0', 'pc-q35-rhel7.6.0', 'pc-i440fx-rhel7.5.0', 'pc-q35-rhel8.1.0', 'pc-q35-rhel8.6.0', 'pc-q35-rhel8.3.0', 'pc-q35-rhel7.5.0', 'pc-q35-rhel8.5.0', 'pc-i440fx-rhel7.1.0', 'q35', 'pc-i440fx-rhel7.2.0', 'pc', 'pc-i440fx-rhel7.6.0', 'pc-i440fx-rhel7.0.0'], 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:dc85f9619f1d', 'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:dc85f9619f1d'}], 'FC': []}, 'vmTypes': ['kvm'], 'memSize': '63805', 'reservedMem': '321', 'guestOverhead': '65', 'rngSources': ['random', 'hwrng'], 'numaNodes': {'0': {'totalMemory': '31553', 'hugepages': {'4': {'totalPages': '8077814'}, ' 2048': {'totalPages': '0'}, '1048576': {'totalPages': '0'}}, 'cpus': [0, 1, 2, 3, 4, 5, 6, 7, 16, 17, 18, 19, 20, 21, 22, 23]}, '1': {'totalMemory': '32251', 'hugepages': {'4': {'totalPages': '8256270'}, '2048': {'totalPages': '0'}, '1048576': {'totalPages': '0'}}, 'cpus': [8, 9, 10, 11, 12, 13, 14, 15, 24, 25, 26, 27, 28, 29, 30, 31]}}, 'numaNodeDistance': {'0': [10, 21], '1': [21, 10]}, 'autoNumaBalancing': 2, 'selinux': {'mode': '1'}, 'liveSnapshot': 'true', 'liveMerge': 'true', 'kdumpStatus': 1, 'deferred_preallocation': True, 'hostdevPassthrough': 'false', 'additionalFeatures': ['libgfapi_supported', 'GLUSTER_SNAPSHOT', 'GLUSTER_GEO_REPLICATION', 'GLUSTER_BRICK_MANAGEMENT'], 'hostedEngineDeployed': True, 'hugepages': [2048, 1048576], 'kernelFeatures': {'SPECTRE_V2': '(Mitigation: Enhanced IBRS, IBPB: conditional, RSB filling, PBRSB-eIBRS: SW sequence)', 'ITLB_MULTIHIT': '(KVM: Mitigation: VMX disabled)', 'MMIO_STALE_DATA': '(Mitigation: Clear CPU buffers; SMT vulnerable)', 'MDS ': '(Not affected)', 'L1TF': '(Not affected)', 'SPEC_STORE_BYPASS': '(Mitigation: Speculative Store Bypass disabled via prctl)', 'TSX_ASYNC_ABORT': '(Mitigation: TSX disabled)', 'SPECTRE_V1': '(Mitigation: usercopy/swapgs barriers and __user pointer sanitization)', 'RETBLEED': '(Mitigation: Enhanced IBRS)', 'SRBDS': '(Not affected)', 'MELTDOWN': '(Not affected)'}, 'vncEncrypted': True, 'backupEnabled': True, 'coldBackupEnabled': True, 'clearBitmapsEnabled': True, 'fipsEnabled': False, 'boot_uuid': '68e5a5ca-3dd5-4103-9613-6065a5fc74da', 'tscFrequency': '3199999000', 'tscScaling': True, 'domain_versions': [0, 2, 3, 4, 5], 'supported_block_size': {'FCP': [512], 'GLUSTERFS': [0, 512, 4096], 'ISCSI': [512], 'LOCALFS': [0, 512, 4096], 'NFS': [512], 'POSIXFS': [512]}, 'cd_change_pdiv': True, 'refresh_disk_supported': True, 'replicate_extend': True, 'measure_subchain': True, 'measure_active': True, 'mailbox_events': True, 'zerocopy_migrations': True, 'netConfigDirty': 'False', 'openstack_b inding_host_ids': {'OVIRT_PROVIDER_OVN': 'vmhost1.mydoamin.local'}, 'lastClientIface': 'ovirtmgmt'}} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:10,620+1100 INFO (jsonrpc/1) [api.host] START getHardwareInfo() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:10,621+1100 INFO (jsonrpc/1) [api.host] FINISH getHardwareInfo return={'status': {'code': 0, 'message': 'Done'}, 'info': {'systemManufacturer': 'Supermicro', 'systemProductName': 'X11DPi-N(T)', 'systemVersion': '123456789', 'systemSerialNumber': '', 'systemUUID': '0034e918-8a80-ea11-8000-ac1f6b3ccce8', 'systemFamily': 'SMC X11'}} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:10,767+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.2.28,43698, flow_id=c4e6178 (api:31) 2023-10-16 18:21:10,778+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=bab905a7-0651-41a3-a9f6-49707358a0eb (api:31) 2023-10-16 18:21:10,778+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={} from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=bab905a7-0651-41a3-a9f6-49707358a0eb (api:37) 2023-10-16 18:21:10,779+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=0f382ce1-8a94-4764-94b7-394b65b3db99 (api:31) 2023-10-16 18:21:10,779+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=0f382ce1-8a94-4764-94b7-394b65b3db99 (api:37) 2023-10-16 18:21:10,786+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698, flow_id=c4e6178 (api:37) 2023-10-16 18:21:11,032+1100 INFO (jsonrpc/6) [vds] Setting Hosted Engine HA local maintenance to False (API:1697) 2023-10-16 18:21:11,198+1100 INFO (jsonrpc/4) [vdsm.api] START connectStorageServer(domType=7, spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', conList=[{'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:/gv0', 'ipv6_enabled': 'false', 'id': '8d7b930e-a758-4010-aaf5-a230447f2761', 'user': '', 'tpgt': '1'}]) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=be7fffbe-f4ca-48f3-b1af-064268ad9eac (api:31) 2023-10-16 18:21:11,199+1100 INFO (tmap-752/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Closing client (__init__:641) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77159) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Shutdown requested (__init__:126) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77159) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,200+1100 INFO (tmap-752/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Closing client (__init__:641) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77153) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Shutdown requested (__init__:126) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77153) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,200+1100 INFO (tmap-752/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Closing client (__init__:641) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77159) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,201+1100 INFO (ioprocess/77153) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,201+1100 INFO (ioprocess/77148) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Shutdown requested (__init__:126) 2023-10-16 18:21:11,201+1100 INFO (ioprocess/77148) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,202+1100 INFO (ioprocess/77148) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194) 2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) Closing client (__init__:641) 2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57) 2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'id': '8d7b930e-a758-4010-aaf5-a230447f2761', 'status': 0}]} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=be7fffbe-f4ca-48f3-b1af-064268ad9eac (api:37) 2023-10-16 18:21:11,210+1100 INFO (ioprocess/76327) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,210+1100 INFO (ioprocess/76327) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,216+1100 INFO (jsonrpc/2) [vdsm.api] START connectStorageServer(domType=7, spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', conList=[{'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:gv1', 'ipv6_enabled': 'false', 'id': '044a086b-e94b-4e5d-a118-283ffceb8de8', 'user': '', 'tpgt': '1'}, {'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:/gv2', 'ipv6_enabled': 'false', 'id': '2d328788-ade0-4e25-b947-1c392d4f133e', 'user': '', 'tpgt': '1'}, {'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:/gv3', 'i pv6_enabled': 'false', 'id': '420b9d49-eebb-4dea-99bb-bcb89fbe2bf2', 'user': '', 'tpgt': '1'}]) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=92bf27ed-8388-4485-abc3-cd79f0d4163f (api:31) 2023-10-16 18:21:11,293+1100 INFO (jsonrpc/2) [storage.storageServer] Creating directory '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1' (storageServer:217) 2023-10-16 18:21:11,293+1100 INFO (jsonrpc/2) [storage.fileutils] Creating directory: /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1 mode: None (fileUtils:214) 2023-10-16 18:21:11,293+1100 WARN (jsonrpc/2) [storage.storageServer] Using user specified backup-volfile-servers option (storageServer:326) 2023-10-16 18:21:11,293+1100 INFO (jsonrpc/2) [storage.mount] mounting vmhosttran1.mydoamin.local:gv1 at /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1 (mount:190) 2023-10-16 18:21:11,415+1100 INFO (jsonrpc/2) [IOProcessClient] (Global) Starting client (__init__:340) 2023-10-16 18:21:11,424+1100 INFO (ioprocess/86760) [IOProcess] (Global) Starting ioprocess (__init__:465) 2023-10-16 18:21:11,518+1100 INFO (jsonrpc/2) [storage.storageServer] Creating directory '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2' (storageServer:217) 2023-10-16 18:21:11,518+1100 INFO (jsonrpc/2) [storage.fileutils] Creating directory: /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2 mode: None (fileUtils:214) 2023-10-16 18:21:11,518+1100 WARN (jsonrpc/2) [storage.storageServer] Using user specified backup-volfile-servers option (storageServer:326) 2023-10-16 18:21:11,518+1100 INFO (jsonrpc/2) [storage.mount] mounting vmhosttran1.mydoamin.local:/gv2 at /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2 (mount:190) 2023-10-16 18:21:11,662+1100 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:11,662+1100 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:11,730+1100 INFO (jsonrpc/2) [storage.storageServer] Creating directory '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3' (storageServer:217) 2023-10-16 18:21:11,730+1100 INFO (jsonrpc/2) [storage.fileutils] Creating directory: /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3 mode: None (fileUtils:214) 2023-10-16 18:21:11,730+1100 WARN (jsonrpc/2) [storage.storageServer] Using user specified backup-volfile-servers option (storageServer:326) 2023-10-16 18:21:11,730+1100 INFO (jsonrpc/2) [storage.mount] mounting vmhosttran1.mydoamin.local:/gv3 at /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3 (mount:190) 2023-10-16 18:21:11,875+1100 INFO (tmap-753/1) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting client (__init__:340) 2023-10-16 18:21:11,883+1100 INFO (tmap-753/2) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting client (__init__:340) 2023-10-16 18:21:11,885+1100 INFO (ioprocess/87016) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting ioprocess (__init__:465) 2023-10-16 18:21:11,889+1100 INFO (tmap-753/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting client (__init__:340) 2023-10-16 18:21:11,892+1100 INFO (ioprocess/87022) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting ioprocess (__init__:465) 2023-10-16 18:21:11,898+1100 INFO (ioprocess/87027) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting ioprocess (__init__:465) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain e311ddf1-7f2c-49ef-a618-050d9a2b947f from storage domain cache (sdc:194) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Closing client (__init__:641) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 from storage domain cache (sdc:194) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Closing client (__init__:641) 2023-10-16 18:21:11,904+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 4412e492-f752-4a6c-98ff-f289043618da from storage domain cache (sdc:194) 2023-10-16 18:21:11,904+1100 INFO (jsonrpc/2) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) Closing client (__init__:641) 2023-10-16 18:21:11,904+1100 INFO (ioprocess/77213) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,904+1100 INFO (ioprocess/77197) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,904+1100 INFO (ioprocess/77222) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,905+1100 INFO (ioprocess/77197) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,905+1100 INFO (ioprocess/77213) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,906+1100 INFO (ioprocess/77222) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194) 2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain e311ddf1-7f2c-49ef-a618-050d9a2b947f from storage domain cache (sdc:194) 2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 from storage domain cache (sdc:194) 2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 4412e492-f752-4a6c-98ff-f289043618da from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain e311ddf1-7f2c-49ef-a618-050d9a2b947f from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 4412e492-f752-4a6c-98ff-f289043618da from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57) 2023-10-16 18:21:11,915+1100 INFO (jsonrpc/2) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'id': '044a086b-e94b-4e5d-a118-283ffceb8de8', 'status': 0}, {'id': '2d328788-ade0-4e25-b947-1c392d4f133e', 'status': 0}, {'id': '420b9d49-eebb-4dea-99bb-bcb89fbe2bf2', 'status': 0}]} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=92bf27ed-8388-4485-abc3-cd79f0d4163f (api:37) 2023-10-16 18:21:11,924+1100 INFO (jsonrpc/3) [vdsm.api] START connectStorageServer(domType=3, spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', conList=[{'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.24', 'ipv6_enabled': 'false', 'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'user': 'cephiscsichapinitiator', 'tpgt': '3'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.25', 'ipv6_enabled': 'false', 'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'user': 'cephiscsichapinitiator', 'tpgt': '2'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.23', 'ipv6_enabled': 'false', 'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'user': 'cephiscsichapinitiator', 'tpgt': '1'}]) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=32063392-d746-458c-b143-919fbf9df35b (api:31) 2023-10-16 18:21:11,925+1100 INFO (jsonrpc/3) [storage.iscsi] Adding iscsi node for target 192.168.4.24:3260,3 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:192) 2023-10-16 18:21:12,053+1100 INFO (jsonrpc/3) [storage.iscsi] Adding iscsi node for target 192.168.4.25:3260,2 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:192) 2023-10-16 18:21:12,174+1100 INFO (jsonrpc/3) [storage.iscsi] Adding iscsi node for target 192.168.4.23:3260,1 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:192) 2023-10-16 18:21:12,293+1100 INFO (jsonrpc/3) [storage.storageServer] Log in to 3 targets using 3 workers (storageServer:600) 2023-10-16 18:21:12,294+1100 INFO (iscsi-login/0) [storage.iscsi] Logging in to iscsi target 192.168.4.24:3260,3 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw via iface default (iscsi:214) 2023-10-16 18:21:12,297+1100 INFO (iscsi-login/1) [storage.iscsi] Logging in to iscsi target 192.168.4.25:3260,2 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw via iface default (iscsi:214) 2023-10-16 18:21:12,303+1100 INFO (iscsi-login/2) [storage.iscsi] Logging in to iscsi target 192.168.4.23:3260,1 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw via iface default (iscsi:214) 2023-10-16 18:21:13,667+1100 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=a9c9c716-740c-424c-8a49-62f84192416a (api:31) 2023-10-16 18:21:13,667+1100 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=a9c9c716-740c-424c-8a49-62f84192416a (api:37) 2023-10-16 18:21:13,676+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=093dfcd6-6f98-49bf-9723-0aca86957167 (api:31) 2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=093dfcd6-6f98-49bf-9723-0aca86957167 (api:37) 2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=69a16b74-67a6-4e74-9fb5-1c738e8bf029 (api:31) 2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=69a16b74-67a6-4e74-9fb5-1c738e8bf029 (api:37) 2023-10-16 18:21:13,693+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:14,410+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=1c78fec1-ad8d-4c4a-9396-bd8bb30248e4 (api:31) 2023-10-16 18:21:14,410+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=1c78fec1-ad8d-4c4a-9396-bd8bb30248e4 (api:37) 2023-10-16 18:21:15,485+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=0f779632-8240-4d02-b43e-ed4737312c36 (api:31) 2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=0f779632-8240-4d02-b43e-ed4737312c36 (api:37) 2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=2375aa06-ec3c-4658-b775-59683a0d25de (api:31) 2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=2375aa06-ec3c-4658-b775-59683a0d25de (api:37) 2023-10-16 18:21:15,503+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:16,692+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:21:16,693+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:17,362+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=bd2bdfd6-4d5e-4e60-8cab-3d25a2f31745 (api:31) 2023-10-16 18:21:17,362+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=bd2bdfd6-4d5e-4e60-8cab-3d25a2f31745 (api:37) 2023-10-16 18:21:17,366+1100 INFO (jsonrpc/1) [vdsm.api] START startMonitoringDomain(sdUUID='ee5b0268-8141-43e0-995e-8b4586217123', hostID=1) from=::1,50158, task_id=fca171ce-9c55-4300-affa-5caa1227495e (api:31) 2023-10-16 18:21:17,367+1100 INFO (jsonrpc/1) [storage.monitor] Start monitoring ee5b0268-8141-43e0-995e-8b4586217123 (monitor:176) 2023-10-16 18:21:17,368+1100 INFO (jsonrpc/1) [vdsm.api] FINISH startMonitoringDomain return=None from=::1,50158, task_id=fca171ce-9c55-4300-affa-5caa1227495e (api:37) 2023-10-16 18:21:17,368+1100 INFO (monitor/ee5b026) [storage.storagedomaincache] Refreshing storage domain cache (resize=True) (sdc:63) 2023-10-16 18:21:17,368+1100 INFO (monitor/ee5b026) [storage.iscsi] Scanning iSCSI devices (iscsi:445) 2023-10-16 18:21:17,377+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=750f996b-757e-4ccf-b08f-e82b91a88678 (api:31) 2023-10-16 18:21:17,378+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.0', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}} from=::1,50158, task_id=750f996b-757e-4ccf-b08f-e82b91a88678 (api:37) 2023-10-16 18:21:17,489+1100 INFO (monitor/ee5b026) [storage.iscsi] Scanning iSCSI devices: 0.12 seconds (utils:373) 2023-10-16 18:21:17,490+1100 INFO (monitor/ee5b026) [storage.hba] Scanning FC devices (hba:42) 2023-10-16 18:21:17,544+1100 INFO (monitor/ee5b026) [storage.hba] Scanning FC devices: 0.05 seconds (utils:373) 2023-10-16 18:21:17,544+1100 INFO (monitor/ee5b026) [storage.multipath] Waiting until multipathd is ready (multipath:95) 2023-10-16 18:21:19,586+1100 INFO (monitor/ee5b026) [storage.multipath] Waited 2.04 seconds for multipathd (tries=2, ready=2) (multipath:122) 2023-10-16 18:21:19,586+1100 INFO (monitor/ee5b026) [storage.multipath] Resizing multipath devices (multipath:223) 2023-10-16 18:21:19,588+1100 INFO (monitor/ee5b026) [storage.multipath] Resizing multipath devices: 0.00 seconds (utils:373) 2023-10-16 18:21:19,588+1100 INFO (monitor/ee5b026) [storage.storagedomaincache] Refreshing storage domain cache: 2.22 seconds (utils:373) 2023-10-16 18:21:19,594+1100 INFO (monitor/ee5b026) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) Starting client (__init__:340) 2023-10-16 18:21:19,603+1100 INFO (ioprocess/87289) [IOProcess] (ee5b0268-8141-43e0-995e-8b4586217123) Starting ioprocess (__init__:465) 2023-10-16 18:21:20,734+1100 INFO (monitor/ee5b026) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:21:20,734+1100 INFO (monitor/ee5b026) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123/dom_md/metadata' (check:104) 2023-10-16 18:21:20,777+1100 INFO (check/loop) [storage.monitor] Domain ee5b0268-8141-43e0-995e-8b4586217123 became VALID (monitor:465) 2023-10-16 18:21:20,779+1100 INFO (event/70) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:21:22,387+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=cb5a305c-dc34-4a94-bf20-fbec9643d784 (api:31) 2023-10-16 18:21:22,387+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=cb5a305c-dc34-4a94-bf20-fbec9643d784 (api:37) 2023-10-16 18:21:23,720+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=c4472b6f-95ab-4803-a5fd-b91b006f0e59 (api:31) 2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=c4472b6f-95ab-4803-a5fd-b91b006f0e59 (api:37) 2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=74b73927-5e56-4483-9f00-d69e5798b4c8 (api:31) 2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=74b73927-5e56-4483-9f00-d69e5798b4c8 (api:37) 2023-10-16 18:21:24,421+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ce22cf89-56ab-4757-97fb-c8e10692223b (api:31) 2023-10-16 18:21:24,422+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.6', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=ce22cf89-56ab-4757-97fb-c8e10692223b (api:37) 2023-10-16 18:21:25,529+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:25,540+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=9465754e-6bf1-4328-9a45-e3b521725225 (api:31) 2023-10-16 18:21:25,540+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.8', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=9465754e-6bf1-4328-9a45-e3b521725225 (api:37) 2023-10-16 18:21:25,541+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=dab8366e-1533-4155-9635-911729326a1a (api:31) 2023-10-16 18:21:25,541+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=dab8366e-1533-4155-9635-911729326a1a (api:37) 2023-10-16 18:21:26,665+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:26,665+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:27,399+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=e69c93c1-639c-4640-b7de-23724deb52f2 (api:31) 2023-10-16 18:21:27,399+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.6', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=e69c93c1-639c-4640-b7de-23724deb52f2 (api:37) 2023-10-16 18:21:28,689+1100 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=fc8a5b61-e928-425a-bd60-36a14d9334e2 (api:31) 2023-10-16 18:21:28,689+1100 INFO (periodic/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=internal, task_id=fc8a5b61-e928-425a-bd60-36a14d9334e2 (api:37) 2023-10-16 18:21:28,741+1100 WARN (jsonrpc/1) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:28,742+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:28,744+1100 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:30,548+1100 WARN (jsonrpc/5) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:30,549+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:30,550+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:30,763+1100 INFO (monitor/ee5b026) [storage.sanlock] Acquiring host id for domain ee5b0268-8141-43e0-995e-8b4586217123 (id=1, wait=False) (clusterlock:359) 2023-10-16 18:21:31,712+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:21:31,712+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:21:32,411+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=b0a218d5-91b0-4dc8-9f4c-45825e601a3c (api:31) 2023-10-16 18:21:32,412+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=b0a218d5-91b0-4dc8-9f4c-45825e601a3c (api:37) 2023-10-16 18:21:33,697+1100 WARN (periodic/0) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:33,753+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=39f0b3ae-475f-4951-90ff-581637710dc9 (api:31) 2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=39f0b3ae-475f-4951-90ff-581637710dc9 (api:37) 2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=2fd4952e-b521-4c0a-93e0-d1186a376d20 (api:31) 2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=2fd4952e-b521-4c0a-93e0-d1186a376d20 (api:37) 2023-10-16 18:21:34,435+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=fad247c4-b608-4df8-88e6-51bc025f21cd (api:31) 2023-10-16 18:21:34,435+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.6', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=fad247c4-b608-4df8-88e6-51bc025f21cd (api:37) 2023-10-16 18:21:35,562+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:35,573+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=37f91533-90ae-4c76-a3e2-c1b8362a9494 (api:31) 2023-10-16 18:21:35,574+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.8', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=37f91533-90ae-4c76-a3e2-c1b8362a9494 (api:37) 2023-10-16 18:21:35,574+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=a25a55a8-10df-48fc-82c2-f603096e58cd (api:31) 2023-10-16 18:21:35,574+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=a25a55a8-10df-48fc-82c2-f603096e58cd (api:37) 2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:37,420+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=8270e681-279e-44a0-a9b0-b7a0110803d7 (api:31) 2023-10-16 18:21:37,420+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.6', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=8270e681-279e-44a0-a9b0-b7a0110803d7 (api:37) 2023-10-16 18:21:38,773+1100 WARN (jsonrpc/4) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:38,774+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:38,775+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:40,581+1100 WARN (jsonrpc/7) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:40,582+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:40,583+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:41,668+1100 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:41,668+1100 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:42,430+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=3be41433-ac1d-4d72-963d-2a69eef1fb26 (api:31) 2023-10-16 18:21:42,431+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=3be41433-ac1d-4d72-963d-2a69eef1fb26 (api:37) 2023-10-16 18:21:43,788+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=5ab069ab-377b-47a4-a19c-e8b2606c7453 (api:31) 2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=5ab069ab-377b-47a4-a19c-e8b2606c7453 (api:37) 2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=14c99af0-527b-444a-92df-d84724c0d215 (api:31) 2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=14c99af0-527b-444a-92df-d84724c0d215 (api:37) 2023-10-16 18:21:44,450+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=88a44308-aac3-441f-a05b-bb0809854292 (api:31) 2023-10-16 18:21:44,450+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=88a44308-aac3-441f-a05b-bb0809854292 (api:37) 2023-10-16 18:21:45,595+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=a871b766-4ae8-4c0e-bd44-2d88034cd1b7 (api:31) 2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.8', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=a871b766-4ae8-4c0e-bd44-2d88034cd1b7 (api:37) 2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=5db4e011-a522-4c96-8a33-6eba3170787f (api:31) 2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=5db4e011-a522-4c96-8a33-6eba3170787f (api:37) 2023-10-16 18:21:45,610+1100 WARN (jsonrpc/6) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155) 2023-10-16 18:21:45,610+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:46,726+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:21:46,726+1100 INFO (jsonrpc/1) [throttled] Current getAllVmStats: {} (throttledlog:87) 2023-10-16 18:21:46,726+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:47,440+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=7e1eac36-3a04-4e4f-8a20-7cb84b703d1a (api:31) 2023-10-16 18:21:47,441+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.7', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=7e1eac36-3a04-4e4f-8a20-7cb84b703d1a (api:37) 2023-10-16 18:21:48,712+1100 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=e6378dc2-3d21-4cd5-851d-5db5bc5c18d4 (api:31) 2023-10-16 18:21:48,712+1100 INFO (periodic/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=internal, task_id=e6378dc2-3d21-4cd5-851d-5db5bc5c18d4 (api:37) 2023-10-16 18:21:48,715+1100 WARN (periodic/3) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155) 2023-10-16 18:21:48,808+1100 WARN (jsonrpc/4) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:48,809+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:48,810+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:52,453+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=10828206-04bc-4cd4-b5a4-38bd0cf9dc8d (api:31) 2023-10-16 18:21:52,453+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.7', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=10828206-04bc-4cd4-b5a4-38bd0cf9dc8d (api:37) 2023-10-16 18:21:53,821+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:53,833+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=6c48b607-2157-4404-9927-fbfca271c525 (api:31) 2023-10-16 18:21:53,833+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=6c48b607-2157-4404-9927-fbfca271c525 (api:37) 2023-10-16 18:21:53,834+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=4e007b1a-3480-4a60-91c9-ccc4a2b8a552 (api:31) 2023-10-16 18:21:53,834+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=4e007b1a-3480-4a60-91c9-ccc4a2b8a552 (api:37) 2023-10-16 18:21:53,837+1100 WARN (jsonrpc/7) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155) 2023-10-16 18:21:53,837+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:54,463+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=91a5858c-775d-4172-8bd7-085a03865e4b (api:31) 2023-10-16 18:21:54,463+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=91a5858c-775d-4172-8bd7-085a03865e4b (api:37) 2023-10-16 18:21:55,639+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=20121b0e-9d04-4a74-a7b8-3bbf48d1fd09 (api:31) 2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.9', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=20121b0e-9d04-4a74-a7b8-3bbf48d1fd09 (api:37) 2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=efa8dcb2-b1cf-4ab5-ae22-7953bdf379e2 (api:31) 2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=efa8dcb2-b1cf-4ab5-ae22-7953bdf379e2 (api:37) 2023-10-16 18:21:55,653+1100 WARN (jsonrpc/6) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155) 2023-10-16 18:21:55,653+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:56,671+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:56,671+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:57,462+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=0a931307-57e8-43ed-bffa-d832f421327a (api:31) 2023-10-16 18:21:57,462+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.7', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=0a931307-57e8-43ed-bffa-d832f421327a (api:37) 2023-10-16 18:22:00,800+1100 INFO (monitor/ee5b026) [storage.sanlock] Host id 1 for domain ee5b0268-8141-43e0-995e-8b4586217123 acquired in 30 seconds (clusterlock:427) 2023-10-16 18:22:01,744+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:22:01,745+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:22:02,471+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=4813e4e9-f151-4727-9cf7-fcf87d1ffdcd (api:31) 2023-10-16 18:22:02,471+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.7', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=4813e4e9-f151-4727-9cf7-fcf87d1ffdcd (api:37) 2023-10-16 18:22:02,569+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=859cacca-c5a4-4eb9-b5a8-c28ef96b464d (api:31) 2023-10-16 18:22:02,569+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.8', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=859cacca-c5a4-4eb9-b5a8-c28ef96b464d (api:37) 2023-10-16 18:22:03,663+1100 INFO (health) [health] LVM cache hit ratio: 39.35% (hits: 170 misses: 262) (health:114) 2023-10-16 18:22:03,729+1100 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=bb148f84-5eab-4cac-b563-b89d690db0bb (api:31) 2023-10-16 18:22:03,729+1100 INFO (periodic/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=bb148f84-5eab-4cac-b563-b89d690db0bb (api:37) 2023-10-16 18:22:03,862+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:03,873+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=305776e1-ad6e-4530-ae48-0f58899b0f6e (api:31) 2023-10-16 18:22:03,873+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.1', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=305776e1-ad6e-4530-ae48-0f58899b0f6e (api:37) 2023-10-16 18:22:03,874+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=f1300145-b974-4f32-90a7-553a6cefe028 (api:31) 2023-10-16 18:22:03,874+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=f1300145-b974-4f32-90a7-553a6cefe028 (api:37) 2023-10-16 18:22:03,879+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:04,475+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ee277021-41dc-414c-9886-c01c807e5658 (api:31) 2023-10-16 18:22:04,475+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=ee277021-41dc-414c-9886-c01c807e5658 (api:37) 2023-10-16 18:22:05,682+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=d1d6d329-10cb-4439-9218-e00b797220ab (api:31) 2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.9', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=d1d6d329-10cb-4439-9218-e00b797220ab (api:37) 2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::1,50158, task_id=1e8c24fa-9d0c-49ea-8f85-f06527068e73 (api:31) 2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=1e8c24fa-9d0c-49ea-8f85-f06527068e73 (api:37) 2023-10-16 18:22:05,699+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:11,675+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:22:11,675+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:22:11,925+1100 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'jsonrpc': '2.0', 'method': 'StoragePool.connectStorageServer', 'params': {'storagepoolID': '7bfaa890-f892-4b3b-bf18-33dabc13e646', 'domainType': 3, 'connectionParams': [{'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.24', 'ipv6_enabled': 'false', 'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'user': 'cephiscsichapinitiator', 'tpgt': '3'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.25', 'ipv6_enabled': 'false', 'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'user': 'cephiscsichapinitiator', 'tpgt': '2'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.23', 'ipv6_enabled': 'false', 'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'user': 'cephiscsich apinitiator', 'tpgt': '1'}]}, 'id': 'd19073fd-9420-4dee-ac37-509a5b321b58'} at 0x7f877a5fa128> timeout=60, duration=60.00 at 0x7f877a69d550> task#=1179 at 0x7f8778933240>, traceback: File: "/usr/lib64/python3.6/threading.py", line 905, in _bootstrap self._bootstrap_inner() File: "/usr/lib64/python3.6/threading.py", line 937, in _bootstrap_inner self.run() File: "/usr/lib64/python3.6/threading.py", line 885, in run self._target(*self._args, **self._kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 243, in run ret = func(*args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 284, in _run self._execute_task() File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 298, in _execute_task task() File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 374, in __call__ self._callable() File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 253, in __call__ self._handler(self._ctx, self._req) File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 296, in _serveRequest response = self._handle_request(req, ctx) File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 338, in _handle_request res = method(**params) File: "/usr/lib/python3.6/site-packages/vdsm/rpc/Bridge.py", line 182, in _dynamicMethod result = fn(*methodArgs) File: "/usr/lib/python3.6/site-packages/vdsm/API.py", line 1117, in connectStorageServer connectionParams) File: "/usr/lib/python3.6/site-packages/vdsm/storage/dispatcher.py", line 57, in wrapper result = ctask.prepare(func, *args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 93, in wrapper return m(self, *a, **kw) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 1163, in prepare result = self._run(func, *args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 867, in _run return fn(*args, **kargs) File: "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-117>", line 2, in connectStorageServer File: "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 33, in method ret = func(*args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2164, in connectStorageServer results = storageServer.connect(domType, conList) File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 914, in connect return con_class.connect_all(connections) File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 617, in connect_all for res in login_results: File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 207, in tmap yield results.get() File: "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File: "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() (executor:346) 2023-10-16 18:22:12,654+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=837a670b-015f-44ad-9f05-d3ca615f3981 (api:31) 2023-10-16 18:22:12,654+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.8', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=837a670b-015f-44ad-9f05-d3ca615f3981 (api:37) 2023-10-16 18:22:13,908+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=399b71ba-f296-4c9c-9ead-870d40d751fe (api:31) 2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.1', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=399b71ba-f296-4c9c-9ead-870d40d751fe (api:37) 2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=0b89ae1d-b45c-4274-a63d-f5dfa0f9392e (api:31) 2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=0b89ae1d-b45c-4274-a63d-f5dfa0f9392e (api:37) 2023-10-16 18:22:13,927+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:14,488+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=8f652e74-faae-4ac9-a6cb-1c114c23cf48 (api:31) 2023-10-16 18:22:14,488+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=8f652e74-faae-4ac9-a6cb-1c114c23cf48 (api:37) 2023-10-16 18:22:15,726+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=f85241ec-5b32-4efa-b1f9-8592a9fd57d5 (api:31) 2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.9', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=f85241ec-5b32-4efa-b1f9-8592a9fd57d5 (api:37) 2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::1,50158, task_id=01d80faa-e134-42d1-bce2-28f91748a9e3 (api:31) 2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=01d80faa-e134-42d1-bce2-28f91748a9e3 (api:37) 2023-10-16 18:22:15,743+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:16,763+1100 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:22:16,763+1100 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:18,750+1100 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=468af5ce-aa49-48f1-ac7c-762cc713710a (api:31) 2023-10-16 18:22:18,750+1100 INFO (periodic/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=468af5ce-aa49-48f1-ac7c-762cc713710a (api:37) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Closing client (__init__:641) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Closing client (__init__:641) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Closing client (__init__:641) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (Global) Closing client (__init__:641) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Closing client (__init__:641) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/87027) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/87022) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/87016) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/86760) [IOProcessClient] (Global) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/77141) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,815+1100 INFO (ioprocess/87022) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:20,815+1100 INFO (ioprocess/86760) [IOProcessClient] (Global) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:20,815+1100 INFO (ioprocess/87016) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:20,816+1100 INFO (ioprocess/87027) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:20,816+1100 INFO (ioprocess/77141) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:22,741+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=62257d75-2f2b-432d-a752-e0ee3e8fd29b (api:31) 2023-10-16 18:22:22,741+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.9', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=62257d75-2f2b-432d-a752-e0ee3e8fd29b (api:37) 2023-10-16 18:22:23,954+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=78b74721-7d6c-4ed2-88f0-4fff08c4b407 (api:31) 2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.1', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=78b74721-7d6c-4ed2-88f0-4fff08c4b407 (api:37) 2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=78ff047f-a712-4dbb-99e3-25909b977799 (api:31) 2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=78ff047f-a712-4dbb-99e3-25909b977799 (api:37) 2023-10-16 18:22:23,974+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:24,503+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=0a2888a2-c8ae-49e5-89fc-a6b5bebfd77f (api:31) 2023-10-16 18:22:24,503+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=0a2888a2-c8ae-49e5-89fc-a6b5bebfd77f (api:37) 2023-10-16 18:22:25,769+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=59f0f76f-e6d4-47cf-8ce0-6b3f9a7e850d (api:31) 2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.0', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=59f0f76f-e6d4-47cf-8ce0-6b3f9a7e850d (api:37) 2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=c5576936-9a21-4b28-886e-b6cd73c90cd4 (api:31) 2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=c5576936-9a21-4b28-886e-b6cd73c90cd4 (api:37) 2023-10-16 18:22:25,786+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:26,678+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:22:26,678+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:31,782+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:22:31,783+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:22:32,825+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=66cc2698-9c5a-4c9d-803b-d89bd549f4f0 (api:31) 2023-10-16 18:22:32,826+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.0', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=66cc2698-9c5a-4c9d-803b-d89bd549f4f0 (api:37) 2023-10-16 18:22:33,771+1100 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=c267fd6f-633d-4c6d-94ad-b5cb5bb8d02d (api:31) 2023-10-16 18:22:33,772+1100 INFO (periodic/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=c267fd6f-633d-4c6d-94ad-b5cb5bb8d02d (api:37) 2023-10-16 18:22:33,998+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=2654a62d-b4f9-425c-8442-b4c418c4f542 (api:31) 2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.2', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=2654a62d-b4f9-425c-8442-b4c418c4f542 (api:37) 2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=c02547e7-20f6-437a-84b4-4fdf98cfd003 (api:31) 2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=c02547e7-20f6-437a-84b4-4fdf98cfd003 (api:37) 2023-10-16 18:22:34,014+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:34,515+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=667f9b48-b2ab-4113-ae86-1b586bbdf5d6 (api:31) 2023-10-16 18:22:34,516+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=667f9b48-b2ab-4113-ae86-1b586bbdf5d6 (api:37) 2023-10-16 18:22:35,812+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=c04a2aa5-1f9d-4c87-b51f-34eaeeedb8e2 (api:31) 2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.0', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=c04a2aa5-1f9d-4c87-b51f-34eaeeedb8e2 (api:37) 2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=f6ac211c-50fa-481a-aeaa-01779cc54412 (api:31) 2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=f6ac211c-50fa-481a-aeaa-01779cc54412 (api:37) 2023-10-16 18:22:35,830+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:37,110+1100 INFO (jsonrpc/2) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:37,111+1100 INFO (jsonrpc/2) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:37,111+1100 INFO (jsonrpc/2) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:37,111+1100 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:41,681+1100 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:22:41,681+1100 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:22:42,911+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=c1a8bf5a-4aa0-4c85-a9fa-8c240a07daa9 (api:31) 2023-10-16 18:22:42,911+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=c1a8bf5a-4aa0-4c85-a9fa-8c240a07daa9 (api:37) 2023-10-16 18:22:44,040+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=d7eb5985-0470-4aea-bd23-937c1a03da72 (api:31) 2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.2', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=d7eb5985-0470-4aea-bd23-937c1a03da72 (api:37) 2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::1,50158, task_id=a32bf2e0-f35c-46f4-b582-a28de56e1960 (api:31) 2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=a32bf2e0-f35c-46f4-b582-a28de56e1960 (api:37) 2023-10-16 18:22:44,059+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:44,528+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=68c2e209-d655-437d-add5-769c0a4932ba (api:31) 2023-10-16 18:22:44,528+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=68c2e209-d655-437d-add5-769c0a4932ba (api:37) 2023-10-16 18:22:45,855+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=b93990c2-b263-4c89-ba4b-ab0e962f6ce8 (api:31) 2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.0', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=b93990c2-b263-4c89-ba4b-ab0e962f6ce8 (api:37) 2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=444bef31-e827-478f-a2fc-16e18dc0b9a6 (api:31) 2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=444bef31-e827-478f-a2fc-16e18dc0b9a6 (api:37) 2023-10-16 18:22:45,873+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:46,786+1100 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:22:46,787+1100 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:48,792+1100 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=50b7c32e-202c-40ec-b484-5eb3f45f7f28 (api:31) 2023-10-16 18:22:48,792+1100 INFO (periodic/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '8.0', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=50b7c32e-202c-40ec-b484-5eb3f45f7f28 (api:37) 2023-10-16 18:22:52,996+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=28687ab3-2149-42d2-92aa-d28cc5ad8d96 (api:31) 2023-10-16 18:22:52,996+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=28687ab3-2149-42d2-92aa-d28cc5ad8d96 (api:37) 2023-10-16 18:22:54,086+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=96f37ad8-6f60-4c9b-8986-cad8c4a5bdf3 (api:31) 2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.2', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=96f37ad8-6f60-4c9b-8986-cad8c4a5bdf3 (api:37) 2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=aa6dc50d-c1c9-4a8c-bf9d-b7ca456f73a6 (api:31) 2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=aa6dc50d-c1c9-4a8c-bf9d-b7ca456f73a6 (api:37) 2023-10-16 18:22:54,106+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:54,533+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ada1e11f-23e9-4704-a8c4-27bfecfd9974 (api:31) 2023-10-16 18:22:54,533+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=ada1e11f-23e9-4704-a8c4-27bfecfd9974 (api:37) 2023-10-16 18:22:55,899+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=a0dd0ff4-5a0a-43cf-8612-ac5ff8e953a8 (api:31) 2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=a0dd0ff4-5a0a-43cf-8612-ac5ff8e953a8 (api:37) 2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=9211755e-e9f9-4c7b-b0dc-8a7689a7884a (api:31) 2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=9211755e-e9f9-4c7b-b0dc-8a7689a7884a (api:37) 2023-10-16 18:22:55,916+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:56,684+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:22:56,684+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:22:57,141+1100 INFO (jsonrpc/5) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:57,141+1100 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:57,141+1100 INFO (jsonrpc/5) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:57,142+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:01,805+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:23:01,805+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:23:03,079+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=95612d72-c18c-4dfe-87f2-f8501f713aca (api:31) 2023-10-16 18:23:03,079+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.2', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=95612d72-c18c-4dfe-87f2-f8501f713aca (api:37) 2023-10-16 18:23:03,813+1100 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=c193e1ec-73f7-4858-aa76-fb9f7d977517 (api:31) 2023-10-16 18:23:03,814+1100 INFO (periodic/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=c193e1ec-73f7-4858-aa76-fb9f7d977517 (api:37) 2023-10-16 18:23:04,130+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:04,141+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=1df7dacf-ebad-4493-ad11-a7ce10fbf851 (api:31) 2023-10-16 18:23:04,141+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.3', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=1df7dacf-ebad-4493-ad11-a7ce10fbf851 (api:37) 2023-10-16 18:23:04,142+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=cbc0b187-25a2-43aa-bfa5-9cf54f675e08 (api:31) 2023-10-16 18:23:04,142+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=cbc0b187-25a2-43aa-bfa5-9cf54f675e08 (api:37) 2023-10-16 18:23:04,147+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:04,541+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=77176abe-b0e4-4f22-bce7-2f26f571743d (api:31) 2023-10-16 18:23:04,541+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=77176abe-b0e4-4f22-bce7-2f26f571743d (api:37) 2023-10-16 18:23:05,942+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:05,954+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=99736bd8-b5b6-4e29-a345-cec8db30f976 (api:31) 2023-10-16 18:23:05,954+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=99736bd8-b5b6-4e29-a345-cec8db30f976 (api:37) 2023-10-16 18:23:05,955+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=286f88d9-373c-42c6-b990-b70d2eb73137 (api:31) 2023-10-16 18:23:05,955+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=286f88d9-373c-42c6-b990-b70d2eb73137 (api:37) 2023-10-16 18:23:05,960+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:11,687+1100 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:11,687+1100 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:11,925+1100 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'jsonrpc': '2.0', 'method': 'StoragePool.connectStorageServer', 'params': {'storagepoolID': '7bfaa890-f892-4b3b-bf18-33dabc13e646', 'domainType': 3, 'connectionParams': [{'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.24', 'ipv6_enabled': 'false', 'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'user': 'cephiscsichapinitiator', 'tpgt': '3'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.25', 'ipv6_enabled': 'false', 'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'user': 'cephiscsichapinitiator', 'tpgt': '2'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.23', 'ipv6_enabled': 'false', 'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'user': 'cephiscsich apinitiator', 'tpgt': '1'}]}, 'id': 'd19073fd-9420-4dee-ac37-509a5b321b58'} at 0x7f877a5fa128> timeout=60, duration=120.00 at 0x7f877a69d550> task#=1179 at 0x7f8778933240>, traceback: File: "/usr/lib64/python3.6/threading.py", line 905, in _bootstrap self._bootstrap_inner() File: "/usr/lib64/python3.6/threading.py", line 937, in _bootstrap_inner self.run() File: "/usr/lib64/python3.6/threading.py", line 885, in run self._target(*self._args, **self._kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 243, in run ret = func(*args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 284, in _run self._execute_task() File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 298, in _execute_task task() File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 374, in __call__ self._callable() File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 253, in __call__ self._handler(self._ctx, self._req) File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 296, in _serveRequest response = self._handle_request(req, ctx) File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 338, in _handle_request res = method(**params) File: "/usr/lib/python3.6/site-packages/vdsm/rpc/Bridge.py", line 182, in _dynamicMethod result = fn(*methodArgs) File: "/usr/lib/python3.6/site-packages/vdsm/API.py", line 1117, in connectStorageServer connectionParams) File: "/usr/lib/python3.6/site-packages/vdsm/storage/dispatcher.py", line 57, in wrapper result = ctask.prepare(func, *args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 93, in wrapper return m(self, *a, **kw) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 1163, in prepare result = self._run(func, *args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 867, in _run return fn(*args, **kargs) File: "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-117>", line 2, in connectStorageServer File: "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 33, in method ret = func(*args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2164, in connectStorageServer results = storageServer.connect(domType, conList) File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 914, in connect return con_class.connect_all(connections) File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 617, in connect_all for res in login_results: File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 207, in tmap yield results.get() File: "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File: "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() (executor:346) 2023-10-16 18:23:13,165+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=0cb1a91c-f096-4daa-bb08-ec407a86433a (api:31) 2023-10-16 18:23:13,165+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.3', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=0cb1a91c-f096-4daa-bb08-ec407a86433a (api:37) 2023-10-16 18:23:14,173+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:14,185+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=fbf38d2b-bc3e-49ad-afd9-de55861129db (api:31) 2023-10-16 18:23:14,185+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.3', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=fbf38d2b-bc3e-49ad-afd9-de55861129db (api:37) 2023-10-16 18:23:14,185+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=3e936d0b-7879-45c8-89d4-ae6bbae466a2 (api:31) 2023-10-16 18:23:14,186+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=3e936d0b-7879-45c8-89d4-ae6bbae466a2 (api:37) 2023-10-16 18:23:14,192+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:14,554+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=316fa299-920a-4c26-a829-9e93647ea804 (api:31) 2023-10-16 18:23:14,554+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=316fa299-920a-4c26-a829-9e93647ea804 (api:37) 2023-10-16 18:23:15,986+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=c7094132-9870-44f0-8274-54527984da18 (api:31) 2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=c7094132-9870-44f0-8274-54527984da18 (api:37) 2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=61b7e2c2-106d-42ff-b6cd-e91e557af60d (api:31) 2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=61b7e2c2-106d-42ff-b6cd-e91e557af60d (api:37) 2023-10-16 18:23:16,003+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:16,823+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:23:16,823+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:18,834+1100 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=5086291d-9172-41f1-9db9-7011a6a34abb (api:31) 2023-10-16 18:23:18,834+1100 INFO (periodic/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=5086291d-9172-41f1-9db9-7011a6a34abb (api:37) 2023-10-16 18:23:23,249+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=bf9f76c3-bead-4087-947b-ebd063ebab6d (api:31) 2023-10-16 18:23:23,249+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.3', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=bf9f76c3-bead-4087-947b-ebd063ebab6d (api:37) 2023-10-16 18:23:24,217+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=3296546b-846b-48f8-b2f3-9fa13e06864e (api:31) 2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.3', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=3296546b-846b-48f8-b2f3-9fa13e06864e (api:37) 2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=77d3abd5-6cf4-4a3f-874e-0a0a1adcf3d5 (api:31) 2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=77d3abd5-6cf4-4a3f-874e-0a0a1adcf3d5 (api:37) 2023-10-16 18:23:24,236+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:24,566+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=12ad1f12-b845-4107-9bc0-326453b99721 (api:31) 2023-10-16 18:23:24,566+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=12ad1f12-b845-4107-9bc0-326453b99721 (api:37) 2023-10-16 18:23:26,030+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=cf9b3b2a-8182-440a-82b8-0c2dd0e33cee (api:31) 2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=cf9b3b2a-8182-440a-82b8-0c2dd0e33cee (api:37) 2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=20a2b03c-eee3-4d1d-a47e-cc6e581df61f (api:31) 2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=20a2b03c-eee3-4d1d-a47e-cc6e581df61f (api:37) 2023-10-16 18:23:26,047+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:26,373+1100 INFO (iscsi-login/0) [storage.iscsi] Removing iscsi node for target 192.168.4.24:3260,3 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:227) 2023-10-16 18:23:26,388+1100 INFO (iscsi-login/1) [storage.iscsi] Removing iscsi node for target 192.168.4.25:3260,2 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:227) 2023-10-16 18:23:26,436+1100 ERROR (iscsi-login/0) [storage.storageServer] Could not login to target (storageServer:607) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 301, in node_login "-p", portal, "-l"]) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 104, in run_cmd out = commands.run(cmd, sudo=True) File "/usr/lib/python3.6/site-packages/vdsm/common/commands.py", line 85, in run raise cmdutils.Error(args, p.returncode, out, err) vdsm.common.cmdutils.Error: Command ['/sbin/iscsiadm', '-m', 'node', '-T', 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', '-I', 'default', '-p', '192.168.4.24:3260,3', '-l'] failed with rc=11 out=b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260]\n' err=b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 604, in iscsi_login iscsi.loginToIscsiNode(con.iface, con.target) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsi.py", line 216, in loginToIscsiNode iscsiadm.node_login(iface.name, target.address, target.iqn) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 321, in node_login raise IscsiNodeError(e.rc, e.out, e.err) vdsm.storage.iscsiadm.IscsiNodeError: (11, b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260]\n', b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n') 2023-10-16 18:23:26,497+1100 ERROR (iscsi-login/1) [storage.storageServer] Could not login to target (storageServer:607) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 301, in node_login "-p", portal, "-l"]) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 104, in run_cmd out = commands.run(cmd, sudo=True) File "/usr/lib/python3.6/site-packages/vdsm/common/commands.py", line 85, in run raise cmdutils.Error(args, p.returncode, out, err) vdsm.common.cmdutils.Error: Command ['/sbin/iscsiadm', '-m', 'node', '-T', 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', '-I', 'default', '-p', '192.168.4.25:3260,2', '-l'] failed with rc=11 out=b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260]\n' err=b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 604, in iscsi_login iscsi.loginToIscsiNode(con.iface, con.target) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsi.py", line 216, in loginToIscsiNode iscsiadm.node_login(iface.name, target.address, target.iqn) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 321, in node_login raise IscsiNodeError(e.rc, e.out, e.err) vdsm.storage.iscsiadm.IscsiNodeError: (11, b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260]\n', b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n') 2023-10-16 18:23:26,514+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Refreshing storage domain cache (resize=True) (sdc:63) 2023-10-16 18:23:26,514+1100 INFO (jsonrpc/3) [storage.iscsi] Scanning iSCSI devices (iscsi:445) 2023-10-16 18:23:26,609+1100 INFO (jsonrpc/3) [storage.iscsi] Scanning iSCSI devices: 0.09 seconds (utils:373) 2023-10-16 18:23:26,609+1100 INFO (jsonrpc/3) [storage.hba] Scanning FC devices (hba:42) 2023-10-16 18:23:26,663+1100 INFO (jsonrpc/3) [storage.hba] Scanning FC devices: 0.05 seconds (utils:373) 2023-10-16 18:23:26,663+1100 INFO (jsonrpc/3) [storage.multipath] Waiting until multipathd is ready (multipath:95) 2023-10-16 18:23:26,690+1100 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:26,690+1100 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:28,705+1100 INFO (jsonrpc/3) [storage.multipath] Waited 2.04 seconds for multipathd (tries=2, ready=2) (multipath:122) 2023-10-16 18:23:28,705+1100 INFO (jsonrpc/3) [storage.multipath] Resizing multipath devices (multipath:223) 2023-10-16 18:23:28,706+1100 INFO (jsonrpc/3) [storage.multipath] Resizing multipath devices: 0.00 seconds (utils:373) 2023-10-16 18:23:28,706+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Refreshing storage domain cache: 2.19 seconds (utils:373) 2023-10-16 18:23:28,796+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Removing domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 from storage domain cache (sdc:194) 2023-10-16 18:23:28,796+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57) 2023-10-16 18:23:28,796+1100 INFO (jsonrpc/3) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'status': 0}, {'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'status': 465}, {'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'status': 465}]} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=32063392-d746-458c-b143-919fbf9df35b (api:37) 2023-10-16 18:23:28,797+1100 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer took more than 1.00 seconds to succeed: 136.87 (__init__:305) 2023-10-16 18:23:28,847+1100 INFO (jsonrpc/5) [vdsm.api] START connectStoragePool(spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', hostID=1, msdUUID='ee5b0268-8141-43e0-995e-8b4586217123', masterVersion=1, domainsMap={'e311ddf1-7f2c-49ef-a618-050d9a2b947f': 'active', 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': 'active', '30775056-38ef-483a-889c-70453e9aa5f9': 'active', 'ee5b0268-8141-43e0-995e-8b4586217123': 'active', '3e33b7ce-7279-4765-a203-cca37eff8cb0': 'active', '4412e492-f752-4a6c-98ff-f289043618da': 'active', '0dde39a4-42e4-4fba-8068-e63e42dd3334': 'active', 'd44ec7f2-3629-4110-93a5-7d8d57230efc': 'active'}) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=22550292-d106-45d1-b5b7-17ff11f6f8a7 (api:31) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagepoolmemorybackend] new storage pool master version 1 and domains map {'e311ddf1-7f2c-49ef-a618-050d9a2b947f': 'Active', 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': 'Active', '30775056-38ef-483a-889c-70453e9aa5f9': 'Active', 'ee5b0268-8141-43e0-995e-8b4586217123': 'Active', '3e33b7ce-7279-4765-a203-cca37eff8cb0': 'Active', '4412e492-f752-4a6c-98ff-f289043618da': 'Active', '0dde39a4-42e4-4fba-8068-e63e42dd3334': 'Active', 'd44ec7f2-3629-4110-93a5-7d8d57230efc': 'Active'} (spbackends:417) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagepool] updating pool 7bfaa890-f892-4b3b-bf18-33dabc13e646 backend from type NoneType instance 0x7f87cc236b70 to type StoragePoolMemoryBackend instance 0x7f877a7cdcc8 (sp:149) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagepool] Connect host #1 to the storage pool 7bfaa890-f892-4b3b-bf18-33dabc13e646 with master domain: ee5b0268-8141-43e0-995e-8b4586217123 (ver = 1) (sp:699) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Clearing storage domain cache (sdc:182) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Refreshing storage domain cache (resize=True) (sdc:63) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.iscsi] Scanning iSCSI devices (iscsi:445) 2023-10-16 18:23:28,946+1100 INFO (jsonrpc/5) [storage.iscsi] Scanning iSCSI devices: 0.10 seconds (utils:373) 2023-10-16 18:23:28,946+1100 INFO (jsonrpc/5) [storage.hba] Scanning FC devices (hba:42) 2023-10-16 18:23:28,996+1100 INFO (jsonrpc/5) [storage.hba] Scanning FC devices: 0.05 seconds (utils:373) 2023-10-16 18:23:28,996+1100 INFO (jsonrpc/5) [storage.multipath] Waiting until multipathd is ready (multipath:95) 2023-10-16 18:23:31,033+1100 INFO (jsonrpc/5) [storage.multipath] Waited 2.04 seconds for multipathd (tries=2, ready=2) (multipath:122) 2023-10-16 18:23:31,033+1100 INFO (jsonrpc/5) [storage.multipath] Resizing multipath devices (multipath:223) 2023-10-16 18:23:31,034+1100 INFO (jsonrpc/5) [storage.multipath] Resizing multipath devices: 0.00 seconds (utils:373) 2023-10-16 18:23:31,034+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Refreshing storage domain cache: 2.19 seconds (utils:373) 2023-10-16 18:23:31,035+1100 INFO (tmap-757/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Starting client (__init__:340) 2023-10-16 18:23:31,043+1100 INFO (tmap-757/1) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting client (__init__:340) 2023-10-16 18:23:31,045+1100 INFO (ioprocess/87928) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Starting ioprocess (__init__:465) 2023-10-16 18:23:31,052+1100 INFO (tmap-757/2) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting client (__init__:340) 2023-10-16 18:23:31,054+1100 INFO (ioprocess/87936) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting ioprocess (__init__:465) 2023-10-16 18:23:31,059+1100 INFO (tmap-757/3) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting client (__init__:340) 2023-10-16 18:23:31,062+1100 INFO (ioprocess/87940) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting ioprocess (__init__:465) 2023-10-16 18:23:31,068+1100 INFO (ioprocess/87948) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting ioprocess (__init__:465) 2023-10-16 18:23:31,842+1100 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:23:31,843+1100 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:23:32,194+1100 INFO (jsonrpc/5) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:23:32,195+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 0dde39a4-42e4-4fba-8068-e63e42dd3334 (monitor:176) 2023-10-16 18:23:32,195+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 3e33b7ce-7279-4765-a203-cca37eff8cb0 (monitor:176) 2023-10-16 18:23:32,201+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring e311ddf1-7f2c-49ef-a618-050d9a2b947f (monitor:176) 2023-10-16 18:23:32,206+1100 INFO (jsonrpc/5) [storage.monitor] Attaching monitor for ee5b0268-8141-43e0-995e-8b4586217123 to the pool (monitor:172) 2023-10-16 18:23:32,207+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 4412e492-f752-4a6c-98ff-f289043618da (monitor:176) 2023-10-16 18:23:32,208+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring fa8d922a-df3c-4f5b-adf6-a77c3689c397 (monitor:176) 2023-10-16 18:23:32,210+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 30775056-38ef-483a-889c-70453e9aa5f9 (monitor:176) 2023-10-16 18:23:32,211+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154) 2023-10-16 18:23:32,218+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring d44ec7f2-3629-4110-93a5-7d8d57230efc (monitor:176) 2023-10-16 18:23:32,219+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154) 2023-10-16 18:23:32,230+1100 INFO (monitor/3e33b7c) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Starting client (__init__:340) 2023-10-16 18:23:32,234+1100 INFO (jsonrpc/5) [storage.storagepool] Creating pool directory '/rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646' (sp:1323) 2023-10-16 18:23:32,234+1100 INFO (jsonrpc/5) [storage.fileutils] Creating directory: /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646 mode: None (fileUtils:214) 2023-10-16 18:23:32,239+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154) 2023-10-16 18:23:32,260+1100 INFO (monitor/e311ddf) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Starting client (__init__:340) 2023-10-16 18:23:32,263+1100 INFO (ioprocess/87989) [IOProcess] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Starting ioprocess (__init__:465) 2023-10-16 18:23:32,268+1100 INFO (monitor/4412e49) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) Starting client (__init__:340) 2023-10-16 18:23:32,271+1100 INFO (ioprocess/88003) [IOProcess] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Starting ioprocess (__init__:465) 2023-10-16 18:23:32,278+1100 INFO (ioprocess/88012) [IOProcess] (4412e492-f752-4a6c-98ff-f289043618da) Starting ioprocess (__init__:465) 2023-10-16 18:23:32,323+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.11 seconds (utils:373) 2023-10-16 18:23:32,323+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',) 2023-10-16 18:23:32,323+1100 INFO (monitor/fa8d922) [storage.monitor] Domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 became INVALID (monitor:465) 2023-10-16 18:23:32,334+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.11 seconds (utils:373) 2023-10-16 18:23:32,334+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',) 2023-10-16 18:23:32,334+1100 INFO (monitor/3077505) [storage.monitor] Domain 30775056-38ef-483a-889c-70453e9aa5f9 became INVALID (monitor:465) 2023-10-16 18:23:32,339+1100 WARN (jsonrpc/5) [storage.lvm] Marked vgs=[0dde39a4-42e4-4fba-8068-e63e42dd3334] as Unreadable due to reload failure (lvm:679) 2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.storagepool] Creating domain metadata directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md' (sp:1354) 2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md mode: None (fileUtils:214) 2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.storagepool] Creating domain images directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images' (sp:1357) 2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images mode: None (fileUtils:214) 2023-10-16 18:23:32,340+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.10 seconds (utils:373) 2023-10-16 18:23:32,340+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',) 2023-10-16 18:23:32,340+1100 INFO (monitor/d44ec7f) [storage.monitor] Domain d44ec7f2-3629-4110-93a5-7d8d57230efc became INVALID (monitor:465) 2023-10-16 18:23:32,345+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/0dde39a4-42e4-4fba-8068-e63e42dd3334 (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/0dde39a4-42e4-4fba-8068-e63e42dd3334 (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/e311ddf1-7f2c-49ef-a618-050d9a2b947f (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/e311ddf1-7f2c-49ef-a618-050d9a2b947f (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/3e33b7ce-7279-4765-a203-cca37eff8cb0 (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/3e33b7ce-7279-4765-a203-cca37eff8cb0 (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/4412e492-f752-4a6c-98ff-f289043618da (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/4412e492-f752-4a6c-98ff-f289043618da (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/ee5b0268-8141-43e0-995e-8b4586217123 (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/ee5b0268-8141-43e0-995e-8b4586217123 (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/mastersd (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/mastersd (sp:1295) 2023-10-16 18:23:32,383+1100 INFO (jsonrpc/5) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=22550292-d106-45d1-b5b7-17ff11f6f8a7 (api:37) 2023-10-16 18:23:32,384+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect took more than 1.00 seconds to succeed: 3.54 (__init__:305) 2023-10-16 18:23:32,385+1100 INFO (monitor/0dde39a) [storage.lvm] Activating lvs: vg=0dde39a4-42e4-4fba-8068-e63e42dd3334 lvs=['ids'] (lvm:1839) 2023-10-16 18:23:32,390+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.2.28,43698, flow_id=706708f9 (api:31) 2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=ac4e0a4f-b2d6-4d28-b765-6ec18d5ecb94 (api:31) 2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.2', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '0.1', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actua l': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '0.1', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '0.1', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=ac4e0a4f-b2d6-4d28-b765-6ec18d5ecb94 (api:37) 2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=c006784d-0604-489e-a8a8-8be4c44e3bec (api:31) 2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=c006784d-0604-489e-a8a8-8be4c44e3bec (api:37) 2023-10-16 18:23:32,413+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698, flow_id=706708f9 (api:37) 2023-10-16 18:23:32,572+1100 INFO (monitor/0dde39a) [storage.lvm] Activating lvs: vg=0dde39a4-42e4-4fba-8068-e63e42dd3334 lvs=['leases'] (lvm:1839) 2023-10-16 18:23:32,771+1100 INFO (monitor/0dde39a) [storage.lvm] Activating lvs: vg=0dde39a4-42e4-4fba-8068-e63e42dd3334 lvs=['metadata', 'inbox', 'outbox', 'xleases', 'master'] (lvm:1839) 2023-10-16 18:23:32,975+1100 INFO (monitor/0dde39a) [storage.blocksd] Setting up domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 (blockSD:973) 2023-10-16 18:23:33,056+1100 INFO (monitor/0dde39a) [storage.check] Start checking '/dev/0dde39a4-42e4-4fba-8068-e63e42dd3334/metadata' (check:104) 2023-10-16 18:23:33,143+1100 INFO (monitor/0dde39a) [storage.monitor] Domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 became VALID (monitor:465) 2023-10-16 18:23:33,144+1100 INFO (monitor/0dde39a) [storage.sanlock] Acquiring host id for domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 (id=1, wait=False) (clusterlock:359) 2023-10-16 18:23:33,145+1100 INFO (event/81) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:23:33,145+1100 INFO (event/82) [storage.storagedomainmanifest] Creating domain images directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images' (blockSD:766) 2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images mode: None (fileUtils:214) 2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.storagedomainmanifest] Creating domain special volumes directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md' (blockSD:771) 2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md mode: None (fileUtils:214) 2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.storagepool] Linking /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/0dde39a4-42e4-4fba-8068-e63e42dd3334 (sp:1278) 2023-10-16 18:23:33,338+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=50cf6835-08a1-4b31-8a61-6e18b85fb86e (api:31) 2023-10-16 18:23:33,338+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.1', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=50cf6835-08a1-4b31-8a61-6e18b85fb86e (api:37) 2023-10-16 18:23:33,470+1100 INFO (monitor/3e33b7c) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:23:33,471+1100 INFO (monitor/3e33b7c) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0/dom_md/metadata' (check:104) 2023-10-16 18:23:33,474+1100 INFO (monitor/4412e49) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:23:33,475+1100 INFO (monitor/4412e49) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da/dom_md/metadata' (check:104) 2023-10-16 18:23:33,519+1100 INFO (check/loop) [storage.monitor] Domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 became VALID (monitor:465) 2023-10-16 18:23:33,521+1100 INFO (event/84) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:23:33,522+1100 INFO (check/loop) [storage.monitor] Domain 4412e492-f752-4a6c-98ff-f289043618da became VALID (monitor:465) 2023-10-16 18:23:33,523+1100 INFO (event/85) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/3e33b7ce-7279-4765-a203-cca37eff8cb0 (sp:1278) 2023-10-16 18:23:33,524+1100 INFO (monitor/e311ddf) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:23:33,524+1100 INFO (monitor/e311ddf) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f/dom_md/metadata' (check:104) 2023-10-16 18:23:33,528+1100 INFO (event/87) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:23:33,534+1100 INFO (event/88) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/4412e492-f752-4a6c-98ff-f289043618da (sp:1278) 2023-10-16 18:23:33,542+1100 INFO (check/loop) [storage.monitor] Domain e311ddf1-7f2c-49ef-a618-050d9a2b947f became VALID (monitor:465) 2023-10-16 18:23:33,543+1100 INFO (event/90) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:23:33,543+1100 INFO (event/91) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/e311ddf1-7f2c-49ef-a618-050d9a2b947f (sp:1278) 2023-10-16 18:23:33,856+1100 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=67318041-086e-4992-a2c3-dc0b7e5ebd4e (api:31) 2023-10-16 18:23:33,856+1100 INFO (periodic/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '0.7', 'delay': '0.00174884', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00116006', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00604746', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00091358', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '1.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '1.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '1.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=internal, task_id=67318041-086e-4992-a2c3-dc0b7e5ebd4e (api:37) 2023-10-16 18:23:34,263+1100 INFO (jsonrpc/3) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=8cf2bf3c-b4ac-4f24-9179-29054ce22a46 (api:31) 2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '1.1', 'delay': '0.00174884', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.8', 'delay': '0.00116006', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.7', 'delay': '0.00604746', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.8', 'delay': '0.00091358', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '2.0', 'delay': '0', 'valid': False, 'version': - 1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=8cf2bf3c-b4ac-4f24-9179-29054ce22a46 (api:37) 2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::1,50158, task_id=dfd9aafe-1cbf-445f-afa9-6b19287c9955 (api:31) 2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=dfd9aafe-1cbf-445f-afa9-6b19287c9955 (api:37) 2023-10-16 18:23:34,280+1100 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:34,572+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=efc38225-ad10-4a67-97d8-1f7f2854456d (api:31) 2023-10-16 18:23:34,572+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.4', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=efc38225-ad10-4a67-97d8-1f7f2854456d (api:37) 2023-10-16 18:23:36,073+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:36,084+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=e1664ede-6677-4756-97d9-d1b033ebb9b6 (api:31) 2023-10-16 18:23:36,085+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00174884', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.6', 'delay': '0.00116006', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.5', 'delay': '0.00604746', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.6', 'delay': '0.00091358', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.8', 'delay': '0', 'valid': False, 'version': - 1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.8', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=e1664ede-6677-4756-97d9-d1b033ebb9b6 (api:37) 2023-10-16 18:23:36,085+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=592b6419-5402-4f74-83ae-b088d9b4b978 (api:31) 2023-10-16 18:23:36,085+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=592b6419-5402-4f74-83ae-b088d9b4b978 (api:37) 2023-10-16 18:23:36,090+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:41,693+1100 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:41,694+1100 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:42,324+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154) 2023-10-16 18:23:42,335+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154) 2023-10-16 18:23:42,341+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154) 2023-10-16 18:23:42,423+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.09 seconds (utils:373) 2023-10-16 18:23:42,423+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',) 2023-10-16 18:23:42,426+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.10 seconds (utils:373) 2023-10-16 18:23:42,426+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',) 2023-10-16 18:23:42,428+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=139f761e-b9f7-4451-8864-f96a498176ed (api:31) 2023-10-16 18:23:42,428+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.2', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=139f761e-b9f7-4451-8864-f96a498176ed (api:37) 2023-10-16 18:23:42,432+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.09 seconds (utils:373) 2023-10-16 18:23:42,432+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',) 2023-10-16 18:23:43,488+1100 INFO (monitor/4412e49) [storage.sanlock] Acquiring host id for domain 4412e492-f752-4a6c-98ff-f289043618da (id=1, wait=False) (clusterlock:359) 2023-10-16 18:23:43,488+1100 INFO (monitor/3e33b7c) [storage.sanlock] Acquiring host id for domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 (id=1, wait=False) (clusterlock:359) 2023-10-16 18:23:43,541+1100 INFO (monitor/e311ddf) [storage.sanlock] Acquiring host id for domain e311ddf1-7f2c-49ef-a618-050d9a2b947f (id=1, wait=False) (clusterlock:359) 2023-10-16 18:23:44,308+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=0d42d65f-0534-47c0-aaf9-7dbac3067a91 (api:31) 2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '1.2', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.8', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.7', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.8', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=0d42d65f-0534-47c0-aaf9-7dbac3067a91 (api:37) 2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=dad656d9-6a05-443b-82ca-c4b7632334dd (api:31) 2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=dad656d9-6a05-443b-82ca-c4b7632334dd (api:37) 2023-10-16 18:23:44,328+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:44,587+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ae63dc53-56a4-4d9a-8bc1-63c6f3c2f160 (api:31) 2023-10-16 18:23:44,587+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.4', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=ae63dc53-56a4-4d9a-8bc1-63c6f3c2f160 (api:37) 2023-10-16 18:23:46,096+1100 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:46,107+1100 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, task_id=9d13808b-d90d-449b-9f9c-85e91b946eeb (api:31) 2023-10-16 18:23:46,107+1100 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.5', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::ffff:192.168.2.28,43698, task_id=9d13808b-d90d-449b-9f9c-85e91b946eeb (api:37) 2023-10-16 18:23:46,108+1100 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, task_id=53f94ff6-5532-41b5-ae7a-3271108c20d2 (api:31) 2023-10-16 18:23:46,108+1100 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, task_id=53f94ff6-5532-41b5-ae7a-3271108c20d2 (api:37) 2023-10-16 18:23:46,113+1100 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:46,121+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=19f3fa69-f932-4edd-96f6-76ba56449cf2 (api:31) 2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=19f3fa69-f932-4edd-96f6-76ba56449cf2 (api:37) 2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=78db57fe-4ab9-4256-b239-2fb8fa2c6a61 (api:31) 2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=78db57fe-4ab9-4256-b239-2fb8fa2c6a61 (api:37) 2023-10-16 18:23:46,137+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:46,860+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:23:46,860+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:23:47,219+1100 INFO (jsonrpc/4) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:47,220+1100 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:47,220+1100 INFO (jsonrpc/4) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:47,220+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:48,878+1100 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=56a338bf-2d23-4dfe-8065-e8e5acc5736a (api:31) 2023-10-16 18:23:48,878+1100 INFO (periodic/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.7', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '5.7', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '5.4', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '5.3', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '5.3', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '6.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '6.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '6.4', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=internal, task_id=56a338bf-2d23-4dfe-8065-e8e5acc5736a (api:37) 2023-10-16 18:23:52,424+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154) 2023-10-16 18:23:52,428+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154) 2023-10-16 18:23:52,433+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154) 2023-10-16 18:23:52,516+1100 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=a77c47c6-03ba-40fd-9bfa-29375ea32ae4 (api:31) 2023-10-16 18:23:52,516+1100 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=a77c47c6-03ba-40fd-9bfa-29375ea32ae4 (api:37) 2023-10-16 18:23:52,542+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.12 seconds (utils:373) 2023-10-16 18:23:52,543+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',) 2023-10-16 18:23:52,546+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.11 seconds (utils:373) 2023-10-16 18:23:52,546+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',) 2023-10-16 18:23:52,551+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.12 seconds (utils:373) 2023-10-16 18:23:52,551+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',) 2023-10-16 18:23:53,353+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=72f4cb81-c54d-4153-9425-0a5766308ed4 (api:31) 2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.1', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '0.2', 'delay': '0.00305946', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '9.9', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '9.8', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '9.8', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '0.8', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '0.8', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '0.8', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=72f4cb81-c54d-4153-9425-0a5766308ed4 (api:37) 2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=65572c4a-c042-49c8-834d-dee8d879896a (api:31) 2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=65572c4a-c042-49c8-834d-dee8d879896a (api:37) 2023-10-16 18:23:53,373+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:54,605+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=1ba074d9-5e35-4527-93ca-0423ed3187f8 (api:31) 2023-10-16 18:23:54,605+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.4', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=1ba074d9-5e35-4527-93ca-0423ed3187f8 (api:37) 2023-10-16 18:23:56,164+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=cdb6b35e-ab67-48e1-a210-2f4ab737a00f (api:31) 2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00305946', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.7', 'delay': '0.00109704', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.6', 'delay': '0.00924556', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.7', 'delay': '0.000824828', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=cdb6b35e-ab67-48e1-a210-2f4ab737a00f (api:37) 2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=9099f061-8efd-4c2e-9669-ff848b5aa791 (api:31) 2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=9099f061-8efd-4c2e-9669-ff848b5aa791 (api:37) 2023-10-16 18:23:56,182+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:56,696+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:56,697+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:24:01,111+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:24:01,123+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, task_id=49598180-2b1c-493c-a4ff-f49a3e988b19 (api:31) 2023-10-16 18:24:01,123+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.4', 'delay': '0.00137321', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '8.0', 'delay': '0.00305946', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '7.6', 'delay': '0.00109704', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '7.6', 'delay': '0.00924556', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '7.6', 'delay': '0.000824828', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '8.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '8.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '8.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::ffff:192.168.2.28,43698, task_id=49598180-2b1c-493c-a4ff-f49a3e988b19 (api:37) 2023-10-16 18:24:01,124+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, task_id=ed16e0f7-5985-4608-94dd-28a587cc92d2 (api:31) 2023-10-16 18:24:01,124+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, task_id=ed16e0f7-5985-4608-94dd-28a587cc92d2 (api:37) 2023-10-16 18:24:01,130+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:24:01,879+1100 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:24:01,879+1100 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:24:02,543+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154) 2023-10-16 18:24:02,547+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154) 2023-10-16 18:24:02,552+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154) 2023-10-16 18:24:02,610+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=6b15ba3d-d2fe-4568-b235-25ded159d77c (api:31) 2023-10-16 18:24:02,610+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00137321', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=6b15ba3d-d2fe-4568-b235-25ded159d77c (api:37) 2023-10-16 18:24:02,744+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.20 seconds (utils:373) 2023-10-16 18:24:02,744+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',) 2023-10-16 18:24:02,755+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.21 seconds (utils:373) 2023-10-16 18:24:02,755+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',) 2023-10-16 18:24:02,755+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.20 seconds (utils:373) 2023-10-16 18:24:02,755+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',) 2023-10-16 18:24:02,828+1100 INFO (jsonrpc/3) [vds] Setting Hosted Engine HA local maintenance to True (API:1697) ~~~ Further info can be provided if required (I don't know what else to include) Thanks in advance for any and all help.

1 0

Ansible - export_ova
by duparchy＠esrf.fr 16 Oct '23

16 Oct '23

We're using a script to export some VMs. This involves export_ova task - Is there a way to parallelize OVA exports (running on different hosts) ? When exporting to OVA from the GUI, we have to select a host, but ansible export_ova has no "hosts" parameter.

3 2

What is the status of the whole Ovirt Project?
by Arman Khalatyan 15 Oct '23

15 Oct '23

Hello everybody, What is the status of the ovirt project, would be continued it in the Rocky Linunx9? the last message is so sad: https://lists.ovirt.org/archives/list/users@ovirt.org/thread/HEKKBM6MZEKBEA… Any good news/progress there? As a happy ovirt user since 2016 should we move to an another system? Thank you beforehand, Arman.

9 14

Ansible - export_ova
by duparchy＠esrf.fr 13 Oct '23

13 Oct '23

We're using a script to export some VMs. This involves export_ova task - Is there a way to parallelize OVA exports (running on different hosts) ? When exporting to OVA from the GUI, we have to select a host, but ansible export_ova has no "hosts" parameter.

1 0

How do i to drive Intel 82599ES 10-Gigabit SFP+ Adapter
by ejzhang＠hotmail.com 13 Oct '23

13 Oct '23

I can find the Adapter. [root@localhost ~]# lspci | grep 82599 06:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) But cannot to install the driver, I try to upgrade kernel to 6.1.57, but the problem is remains. How can i to solve the problem? Thanks very much! [root@localhost tmp]# dnf localinstall kmod-ixgbe-5.19.6-1.rhel9u1.x86_64.rpm Last metadata expiration check: 0:24:26 ago on Thu 12 Oct 2023 05:25:27 PM CST. Error: Problem: conflicting requests - nothing provides kernel(__SCK__tp_func_xdp_exception) = 0xc15ae5f5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(___pskb_trim) = 0x80ed79e7 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__alloc_pages) = 0x26458776 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__alloc_skb) = 0xbc60587f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__dev_kfree_skb_any) = 0x8d62db77 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__free_pages) = 0x7ddd045c needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__hw_addr_sync_dev) = 0x84c17451 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__hw_addr_unsync_dev) = 0xb05b216e needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__napi_alloc_skb) = 0x5f802418 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__napi_schedule_irqoff) = 0xd022b30c needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__netif_napi_del) = 0x310d1afd needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__page_frag_cache_drain) = 0xb6ccef70 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__pci_register_driver) = 0x1cb89f6b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__skb_pad) = 0x16cd8a38 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__tracepoint_xdp_exception) = 0xad42e583 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(_dev_err) = 0x3a3ec6c6 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(_dev_info) = 0xff4f021a needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(_dev_warn) = 0x1e5fac62 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(alloc_etherdev_mqs) = 0x665d1b75 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(bpf_prog_put) = 0xa1b79421 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(bpf_warn_invalid_xdp_action) = 0xb05e54b3 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(build_skb) = 0x5c034c8f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(consume_skb) = 0x014a9f1b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dca3_get_tag) = 0x258e3aac needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dca_add_requester) = 0x14542454 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dca_remove_requester) = 0x7e094b21 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dcb_getapp) = 0x535a6190 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dcb_ieee_delapp) = 0xc6496dd5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dcb_ieee_setapp) = 0x68aabfb5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dcb_setapp) = 0x0bf4f0bb needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(debugfs_create_dir) = 0x11b4f7a8 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(debugfs_create_file) = 0xa6202222 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(debugfs_remove) = 0x20033468 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dev_addr_add) = 0x96486dae needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dev_addr_del) = 0xe222c9e8 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dev_driver_string) = 0xcdafdb2a needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dev_get_stats) = 0x9a9222bd needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(device_set_wakeup_enable) = 0xac4a0980 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(devm_hwmon_device_register_with_groups) = 0x70a8831d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(devm_kmalloc) = 0xe9ad0580 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_alloc_attrs) = 0xed08591d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_free_attrs) = 0x5c4b9170 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_map_page_attrs) = 0xe2dc579b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_set_coherent_mask) = 0x97c42987 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_set_mask) = 0xf4110cd1 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_sync_single_for_cpu) = 0x51b1550f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_sync_single_for_device) = 0x9980b0a6 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_unmap_page_attrs) = 0xa41d1497 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(driver_for_each_device) = 0x9465cb85 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(eth_get_headlen) = 0x98601bb3 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(eth_type_trans) = 0x25c1aba0 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(eth_validate_addr) = 0xfc0d5980 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ethtool_op_get_link) = 0xc3aa9108 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ethtool_op_get_ts_info) = 0x7eb31781 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(flow_block_cb_setup_simple) = 0x4dd5fafe needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(free_netdev) = 0x52c68cbc needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ipv6_find_hdr) = 0x9c90a5b6 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(kfree_skb_reason) = 0xc87ace65 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(kmalloc_caches) = 0x5f73fb3c needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(kmem_cache_alloc_trace) = 0xa3cddcae needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(module_layout) = 0xa7f1f8eb needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(napi_complete_done) = 0x40402bca needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(napi_consume_skb) = 0x10e9a3dc needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(napi_disable) = 0x284fd3bf needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(napi_enable) = 0xaf328093 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(napi_gro_receive) = 0x4bd5d8dd needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(napi_schedule_prep) = 0x76bae8ab needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ndo_dflt_bridge_getlink) = 0x089345b5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ndo_dflt_fdb_add) = 0x77dcc759 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_crit) = 0x6e6d1264 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_err) = 0xc66472cf needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_info) = 0xa71c2c40 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_notice) = 0x07efd02f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_printk) = 0x2c04b9df needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_reset_tc) = 0x8849ff8a needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_set_num_tc) = 0xd82a15a6 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_set_tc_queue) = 0x9d6a1782 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_warn) = 0xfe96cfed needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_carrier_off) = 0x14111c62 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_carrier_on) = 0x96781ba1 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_device_attach) = 0x6d7af90c needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_device_detach) = 0xf3103b00 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_napi_add) = 0xec77a489 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_schedule_queue) = 0xe191e05f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_set_real_num_rx_queues) = 0x8379688b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_set_real_num_tx_queues) = 0xa1e2fa4b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_set_xps_queue) = 0x2813f1cd needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_tx_stop_all_queues) = 0x13392855 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_tx_wake_queue) = 0x24e11830 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(param_array_ops) = 0x7a0b1ed0 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(param_ops_int) = 0xaad18ecd needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_aer_clear_nonfatal_status) = 0x254f8d5e needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_dev_driver) = 0x45689a09 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_dev_get) = 0xdd00d2c6 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_dev_put) = 0x516c634d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_disable_device) = 0x0921c2a1 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_disable_link_state) = 0xc5408d7d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_disable_msi) = 0xe3ee7e3c needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_disable_msix) = 0x5d88de56 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_disable_pcie_error_reporting) = 0xf4167337 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_disable_sriov) = 0xae1adbfa needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_enable_device_mem) = 0x74ebcbd1 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_enable_msi) = 0x01bdea04 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_enable_msix_range) = 0x98ff98d5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_enable_pcie_error_reporting) = 0x717d4b61 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_enable_sriov) = 0x68c478f7 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_find_capability) = 0x3b06b1de needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_find_ext_capability) = 0xce3f4b59 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_get_device) = 0x7e3c03c5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_num_vf) = 0xbf36cc3e needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_prepare_to_sleep) = 0x4d75b46f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_read_config_byte) = 0x595813a6 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_read_config_dword) = 0x5d999172 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_read_config_word) = 0x3911c65c needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_release_selected_regions) = 0x7a3abda2 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_request_selected_regions) = 0xfab14d9f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_restore_state) = 0x24bbcb8b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_save_state) = 0xfd2da512 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_select_bars) = 0x1191f7a3 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_set_master) = 0x2f6d2bb5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_set_power_state) = 0x54621869 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_sriov_set_totalvfs) = 0xc0efcf8f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_unregister_driver) = 0xa972d633 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_vfs_assigned) = 0x6bdb5a77 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_wake_from_d3) = 0x2464b789 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_write_config_word) = 0x767cdb47 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pcie_capability_read_word) = 0xd51ae9d7 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pcie_print_link_status) = 0x26df835b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(printk) = 0xc5850110 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pskb_expand_head) = 0x0d526b2b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ptp_clock_event) = 0xe7d77369 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ptp_clock_index) = 0x3876beca needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ptp_clock_register) = 0x0ed9b290 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ptp_clock_unregister) = 0xb77a6063 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(register_netdev) = 0xba3ff097 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(simple_open) = 0xf9fc90f3 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(skb_add_rx_frag) = 0x21ea980a needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(skb_checksum_help) = 0x08d53501 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(skb_clone_tx_timestamp) = 0x5b57e627 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(skb_copy_bits) = 0x632061f3 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(skb_put) = 0xe6c6ba90 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(skb_tstamp_tx) = 0xc6a9499d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(softnet_data) = 0x3bf76a4e needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(udp_tunnel_nic_ops) = 0x9b1d2e6b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(unregister_netdev) = 0x01d13501 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_convert_zc_to_xdp_frame) = 0xcdb7b8cd needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_do_redirect) = 0x635f6fef needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_master_redirect) = 0xd895cc1b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_return_frame) = 0x5b1c3ac9 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_return_frame_rx_napi) = 0x4742580d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_rxq_info_reg) = 0xc5f644e9 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_rxq_info_reg_mem_model) = 0xcabfedb2 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_rxq_info_unreg) = 0x0d8f96fc needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_rxq_info_unreg_mem_model) = 0x610cb9e1 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_alloc) = 0xa0815c99 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_dma_map) = 0x0198092e needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_dma_sync_for_cpu_slow) = 0x0cac6528 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_dma_sync_for_device_slow) = 0xe91374d9 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_dma_unmap) = 0xb4f7ca78 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_free) = 0xa353e08d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_raw_get_dma) = 0x0eb88c7f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_set_rxq_info) = 0x86b2db4e needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xsk_tx_completed) = 0x5205b281 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xsk_tx_peek_desc) = 0xe67282ce needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xsk_tx_release) = 0xb016016f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

1 0

Missing data for some VM in Grafana Monitoring Portal
by Guillaume Pavese 12 Oct '23

12 Oct '23

Hi all. We are using oVirt 4.5.4 with the provided grafana monitoring tool. Everything seemed to be working as expected, however we have discovered that for some vm (not all), there is no available Disk or Ethernet data (while CPU and Memory data is available) : [image: image.png] While for some other VMs running on the same host, all data is available: [image: image.png] Does anyone have an idea what might be going wrong? -- Ce message et toutes les pièces jointes (ci-après le “message”) sont établis à l’intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse. L’internet ne permettant pas d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse ou il aurait été modifié. IT, ES, UK. <https://interactiv-group.com/disclaimer.html>

1 0

PKIX path validation failed:
by Marcelo Vieira 11 Oct '23

11 Oct '23

Prezados, boa tarde! Como faço para atualizar o certificado Java? Estou recebendo o seguinte erro na console do engine : PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed e recebendo erro na console do root" This web browser is too old to run the Web Console (missing selector(:is():where())) [root@node1 ~]# cat /etc/centos-release CentOS Stream release 8 [root@node1 ~]# uname -a Linux node1 4.18.0-383.el8.x86_64 #1 SMP Wed Apr 20 15:38:08 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux Att., Marcelo

1 0

After upgrade to 4.5.4: host reboot fails and api-user unable to log in
by Thyen, Niko 11 Oct '23

11 Oct '23

Hi all, i just upgraded a cluster from 4.3.10 to 4.5.4. 3 hosts, hosted-engine, engine deployment via shell and with restore from backup. Engine update worked fine, but host update from 4.4.10 to 4.5.4 failed because the engine could not initiate a host reboot: 2023-10-10 13:37:27,802+02 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (EE-ManagedThreadFactory-engine-Thread-2) [7d4d6 a58] The message key 'SshHostReboot' is missing from 'bundles/ExecutionMessages' 2023-10-10 13:37:27,823+02 INFO [org.ovirt.engine.core.bll.SshHostRebootCommand] (EE-ManagedThreadFactory-engine-Thread-2) [7d4d6a58] Running command: SshHostRebootCommand internal: true. Entities affected : ID: b578519d-7f34-4509-8b8b-dbab1b2cf6d2 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2023-10-10 13:37:27,827+02 INFO [org.ovirt.engine.core.bll.SshHostRebootCommand] (EE-ManagedThreadFactory-engine-Thread-2) [7d4d6a58] Opening SSH reboot session on host <host fqdn> 2023-10-10 13:37:28,030+02 ERROR [org.ovirt.engine.core.bll.SshHostRebootCommand] (EE-ManagedThreadFactory-engine-Thread-2) [7d4d6a58] SSH reboot command failed on host '<host fqdn>': SSH session closed during connection 'root@<host fqdn>' Stdout: Stderr: 2023-10-10 13:37:28,038+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-2) [7d4d6a58] EVENT_ID: SYSTEM_FAILED_SSH_HOST_RESTART(198), A restart using SSH initiated by the engine to Host <host fqdn> has failed. This also happens when a host is reinstalled via engine webinterface. How can i fix this? The only Workaround i found is rebooting the host via SSH and reactivating it via engine webinterface. Also, i constantly find error messages like these in the engine.log: 2023-10-11 13:54:11,338+02 INFO [org.ovirt.engine.extension.aaa.jdbc.core.Authentication] (default task-815) [] locking user: api-user due to interval failures 2023-10-11 13:54:14,932+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-812) [] EVENT_ID: USER_VDC_LOGIN_FAILED(114), User api-user@internal connecting from '<engine-ip>' failed to log in : 'Unable to log in because the password has expired. Please change the password to proceed.'. 2023-10-11 13:54:14,933+02 ERROR [org.ovirt.engine.core.sso.service.SsoService] (default task-815) [] OAuthException access_denied: Cannot authenticate user 'api-user@internal': Unable to log in because the password has expired. Please change the password to proceed.. 2023-10-11 13:54:14,934+02 ERROR [org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default task-814) [] Cannot authenticate using authentication Headers: access_denied: Cannot authenticate user 'api-user@internal': Unable to log in because the password has expired. Please change the password to proceed.. This only affects the user "api-user". I am still new to oVirt and have no idea what the api-user is trying to do and where. I found https://www.ovirt.org/develop/release-management/features/infra/aaa-jdbc.ht… but havent tried resetting yet because i dont know what will be affected by that. Can anyone help to point me in the right direction? Many thanks in advance :) Regards, Niko

1 0

Impossible to snaphot certain virtual machines
by nolhan.bertille＠unige.ch 10 Oct '23

10 Oct '23

Hello all, When performing snapshots I recieve the following error 2023-10-09 16:09:49,951: !!! No snapshot found !!! 2023-10-09 16:09:49,951: All backups done 2023-10-09 16:09:49,951: Backup failure for: 2023-10-09 16:09:49,951: ******* 2023-10-09 16:09:49,951: Some errors occurred during the backup, please check the log file Where can I look for more information? Where can I rise the verbosity ? What process should I look at ? Thanks all

1 0

Getting error when performing snapshots
by nolhan.bertille＠unige.ch 09 Oct '23

09 Oct '23

Hello Guys, I'm repeatly getting the following error when running snapshots with the script backup.py 2023-10-06 14:54:09,773: !!! No snapshot found !!! 2023-10-06 14:54:09,773: All backups done 2023-10-06 14:54:09,773: Backup failure for: 2023-10-06 14:54:09,773: ******* 2023-10-06 14:54:09,773: Some errors occurred during the backup, please check the log file I have no idea what/where to lok at ? Could somebody help me ?

1 0

Hosted engine VM not coming up after Storage rebuild.
by Sumit Basu 06 Oct '23

06 Oct '23

Hi, We are running Ovirt 4.3 on IBM x3650, x3550 servers with our SAN on IBM Midrange storage DS5300. All the storage domains are separate LUN's in the storage with a dedicated LUN for the HostedEngine VM. The storage had a failure due to a major power issue. One of the storage array's that have the storage domains had to be re-constructed using IBM's StorageManager tool. After booting the hosts, i can see all the storage domains as LUN's with the multipath -ll from all the hosts, but on starting the hosted engine with #hosted-engine --vm-start, and checking with --vm-status i get "Engine status : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}'. The global maintenance mode is enabled and i tried "hosted-engine --console" - after logging in i checked the boot.log, i find [FAILED] Failed to mount /var/log. See 'systemctl status var-log.mount' for details. [DEPEND] Dependency failed for Update UTMP about System Boot/Shutdown. [DEPEND] Dependency failed for Update UTMP about System Runlevel Changes. [DEPEND] Dependency failed for Flush Journal to Persistent Storage. [DEPEND] Dependency failed for /var/log/audit. [DEPEND] Dependency failed for Local File Systems. [DEPEND] Dependency failed for Mark the need to relabel after reboot. [DEPEND] Dependency failed for Migrate local... structure to the new structure. [DEPEND] Dependency failed for Relabel all filesystems, if necessary. the "systemctl status var-log.mount" shows ● var-log.mount - /var/log Loaded: loaded (/etc/fstab; bad; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2023-10-06 14:38:58 IST; 10min ago Where: /var/log What: /dev/mapper/ovirt-log Docs: man:fstab(5) man:systemd-fstab-generator(8) Process: 691 ExecMount=/bin/mount /dev/mapper/ovirt-log /var/log -t xfs -o nodev (code=exited, status=32) Oct 06 14:38:57 ovman systemd[1]: Mounting /var/log... Oct 06 14:38:58 ovman mount[691]: mount: mount /dev/mapper/ovirt-...g Oct 06 14:38:58 ovman systemd[1]: var-log.mount mount process exi...2 Oct 06 14:38:58 ovman systemd[1]: Failed to mount /var/log. Oct 06 14:38:58 ovman systemd[1]: Unit var-log.mount entered fail.... Hint: Some lines were ellipsized, use -l to show in full. I need to recover from this quick. Sumit Basu

1 0

keycloak Active Directory BufferOverFlowException
by karl.morgan＠gmail.com 05 Oct '23

05 Oct '23

been trying to get a configuration working with ovirt 4.5.4 and keycloak and windows Active Directory I have had partial success in that i can with a little fiddling use ldap authentication for users. The fiddling involves going in and adding specific permissions to each ID after the first attempt has been made. I am now trying to get groups working such that I can have an AD user group automatically determine a users capabilities dependent on inclusion in specific groups. I have keycloak configured where, from within keycloak I can list users, And i can see which groups a user is a member of. It looks like is should be working. However, in Ovirt im getting invalid data error in the web and logs print the following ==> error_log <== [Thu Oct 05 09:22:58.037130 2023] [proxy_ajp:error] [pid 75583:tid 140622648301312] AH03229: ajp_msg_append_cvt_string(): BufferOverflowException 4 5694 ==> ssl_error_log <== [Thu Oct 05 09:22:58.037162 2023] [proxy_ajp:error] [pid 75583:tid 140622648301312] [client 172.28.96.180:43980] AH00971: ajp_marshal_into_msgb: Error appending the header value, referer: https://ov2ctl01-mn.internal.shutterfly.com/ovirt-engine/ [Thu Oct 05 09:22:58.037171 2023] [proxy_ajp:error] [pid 75583:tid 140622648301312] [client 172.28.96.180:43980] AH00988: ajp_send_header: ajp_marshal_into_msgb failed, referer: https://ov2ctl01-mn.internal.shutterfly.com/ovirt-engine/ [Thu Oct 05 09:22:58.037176 2023] [proxy_ajp:error] [pid 75583:tid 140622648301312] (120001)APR does not understand this error code: [client 172.28.96.180:43980] AH00868: request failed to 127.0.0.1:8702 (127.0.0.1), referer: https://ov2ctl01-mn.internal.shutterfly.com/ovirt-engine/ Pretty sure i need to be able to adjust a buffer size between keycloak and ajp module but am clueless how to proceed. Any help would be appreciated

1 0

How to obtain vm snapshots status
by anton.alymov＠cyberprotect.ru 04 Oct '23

04 Oct '23

Hi! I use ovirt rest api to start vm, backup vm and then remove vm. I start vm, wait for vmstatus up, then start backup, wait for starting, finalize, wait for succeeded, wait for disk unlock. Looks like backup is finished here from my side.Because ovirt repost succeed status and unlocks disk. But if i try shutdown and remove vm ovirt will throw error Cannot remove VM. The VM is performing an operation on a Snapshot. Please wait for the operation to finish, and try again. Ok, ovirt is right here, I see from web interface that operation hasn't finished yet. How can I obtain correct status where vm can be removed? I also tried to get info about vm snapshots but all of them had Status: ok

4 3

Cannot remove template because a disk is based on it
by nicolas＠devels.es 04 Oct '23

04 Oct '23

Hi, We're running oVirt 4.5. We have a template which we'd like to get rid of, there are no VMs based on it. However, trying to remove it oVirt states that: Cannot remove Template. The following Disk(s) are based on it: (b54ee1cb-ed64-4db4-bd3d-eac8b22ea095) . When opening the 'Disks' subtab in the template, it appears a disk (screenshot attached). However, when opening the oVirt Storage->Disks option, I cannot find the disk by the ID, not even ordering the ID column and trying to find it visually. Can anyone point to the problem and a possible solution/workaround? Thanks.

2 4

Image upload paused by system
by muchiri.maina＠gmail.com 03 Oct '23

03 Oct '23

I recently renewed my certs for the manager and hosts in ovirt 4.4 now when i try to upload an iso it gets paused by system then the error ,Unable to upload image to disk d840c8a3-21f5-4ad9-bd64-f569a93b4e74 due to a network error. Ensure ovirt-engine's CA certificate is registered as a trusted CA in the browser. i have registered the CA certificate but still wont upload

1 0

What certs need monitoring for expiration for Ovirt hosts ?
by morgan cox 03 Oct '23

03 Oct '23

Hi. I am aware of the following certs - "/etc/pki/vdsm/certs/vdsmcert.pem" - "/etc/pki/vdsm/libvirt-spice/server-cert.pem" - "/etc/pki/vdsm/libvirt-vnc/server-cert.pem" - "/etc/pki/libvirt/clientcert.pem" - "/etc/pki/vdsm/libvirt-migrate/server-cert.pem" And are monitoring them to avoid certs being expired. We have a 3rd party cert/ca - do I also need to monitor the following - "/etc/pki/ovirt-vmconsole/ca.pub" - "/etc/pki/vdsm/certs/cacert.pem" - "/etc/pki/vdsm/libvirt-migrate/ca-cert.pem" - "/etc/pki/vdsm/libvirt-spice/ca-cert.pem" - "/etc/pki/vdsm/libvirt-vnc/ca-cert.pem" - "/etc/pki/CA/cacert.pem" If the CA is updated on the engine do the above ca certs get updated with an update or re-enroll ? Thanks

1 0

Re: host deploy problem
by David Slaine 27 Sep '23

27 Sep '23

installed python3.11-netaddr the following error appeared: FAILED! => {\"msg\": \"The conditional check 'elasticsearch_host == None or elasticsearch_host is undefined' failed. The error was: error while evaluating conditional (elasticsearch_host == None or elasticsearch_host is undefined): 'elasticsearch_host' is undefined. 'elasticsearch_host' is undefined\\n\\nThe error appears to be in '/usr/share/ansible/roles/oVirt.metrics/roles/ovirt_initial_validations/tasks/check_logging_collectors.yml': line 4, column 3, but may\\nbe elsewhere in the file depending on the exact syntax problem.\\n\\nThe offending line appears to be:\\n\\n\\n- name: If output plugin is elasticsearch, validate host address is set\\n ^ here\\n\"}", ср, 27 сент. 2023 г. в 10:54, David Slaine <david.slaine(a)gmail.com>: > Thanks for your answer > I installed python38-netaddr-0.8.0-2.el8.noarch on both servers(host and > engine), but it didn't help and the same error appears > > ср, 27 сент. 2023 г. в 08:56, David Slaine <david.slaine(a)gmail.com>: > >> Thanks for your answer >> I installed python38-netaddr-0.8.0-2.el8.noarch on both servers(host and >> engine), but it didn't help and the same error appears >> >> вт, 26 сент. 2023 г. в 22:12, Strahil Nikolov <hunter86_bg(a)yahoo.com>: >> >>> Install python3-netaddr or python38-netaddr package. >>> >>> Best Regards, >>> Strahil Nikolov >>> >>> >>> On Tuesday, September 26, 2023, 9:04 PM, David Slaine < >>> david.slaine(a)gmail.com> wrote: >>> >>> Ovirt-engine 4.5.4-1 >>> Rocky 8.8 >>> >>> Hello! >>> >>> problem with adding a host to the cluster, it fails with the following >>> message in /var/log/ovirt-engine/host-deploy/: >>> >>> FAILED! => {\"msg\": \"The conditional check 'cluster_switch == >>> \\\"ovs\\\" or (ovn_central is defined and ovn_central | ipaddr)' failed. >>> The error was: The ipaddr filter requires python's netaddr be installed on >>> the ansible controller\\n\\nThe error appears to be in >>> '/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-provider-ovn-driver/tasks/configure.yml': >>> line 3, column 5, but may\\nbe elsewhere in the file depending on the exact >>> syntax problem.\\n\\nThe offending line appears to be:\\n\\n- block:\\n - >>> name: Install ovs\\n ^ here\\n\"} >>> >>> how to fix this issue ? >>> thanks >>> _______________________________________________ >>> Users mailing list -- users(a)ovirt.org >>> To unsubscribe send an email to users-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/FDEAQXI7XHUL6… >>> >>>

1 0

host deploy problem
by David Slaine 26 Sep '23

26 Sep '23

Ovirt-engine 4.5.4-1 Rocky 8.8 Hello! problem with adding a host to the cluster, it fails with the following message in /var/log/ovirt-engine/host-deploy/: FAILED! => {\"msg\": \"The conditional check 'cluster_switch == \\\"ovs\\\" or (ovn_central is defined and ovn_central | ipaddr)' failed. The error was: The ipaddr filter requires python's netaddr be installed on the ansible controller\\n\\nThe error appears to be in '/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-provider-ovn-driver/tasks/configure.yml': line 3, column 5, but may\\nbe elsewhere in the file depending on the exact syntax problem.\\n\\nThe offending line appears to be:\\n\\n- block:\\n - name: Install ovs\\n ^ here\\n\"} how to fix this issue ? thanks

2 1

Unable to add KVM (ubuntu flavour)hypervisors on Ovirt engine.
by spp16121991＠gmail.com 26 Sep '23

26 Sep '23

Hi all, I am a noob in KVM. I am a VMware engineer. I have created a lab environment to where i have hosted KVM as nested hypervisors within esxi. I have a vsan environment. I want to create a similar KVM environment. hence, i have deployed 3 kvm hypervisors (nested) i have deployed Ovirt-manager ( equivalent to vcenter) to host/manage my KVM hypervisors. to deploy Ovirt-manager, i installed ovirt packages in centos VM. Because ovirt pacjages are not available in ubuntu, hence choose centos. next is i want to add kvm hypervisors on ovirt manager similar to how we add esxi hosts to vcenter. That is where i am failing. From my limited understanding in KVM, ovirt is looking for yum package manager , KVM are ubuntu flavoured, hence it has apt package manager. What my ask is, can this be fixed? is there a file i need to edit in ovirt-engine.? Can Ovirt-manager host KVM of ubuntu flavour? Please advise. Checked the ovirt logs: Here is the log excerpts: ---------------------------------------------- } ], "python3-oauthlib" : [ { "name" : "python3-oauthlib", "version" : "3.1.0-1ubuntu2", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-openssl" : [ { "name" : "python3-openssl", "version" : "19.0.0-1build1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-pexpect" : [ { "name" : "python3-pexpect", "version" : "4.6.0-1build1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-pkg-resources" : [ { "name" : "python3-pkg-resources", "version" : "45.2.0-1ubuntu0.1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-problem-report" : [ { "name" : "python3-problem-report", "version" : "2.20.11-0ubuntu27.27", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-ptyprocess" : [ { "name" : "python3-ptyprocess", "version" : "0.6.0-1ubuntu1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-pyasn1" : [ { "name" : "python3-pyasn1", "version" : "0.4.2-3build1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-pyasn1-modules" : [ { "name" : "python3-pyasn1-modules", "version" : "0.2.1-0.2build1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-pymacaroons" : [ { "name" : "python3-pymacaroons", "version" : "0.13.0-3", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-pyrsistent" : [ { "name" : "python3-pyrsistent", "version" : "0.15.5-1build1", "arch" : "amd64", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-requests" : [ { "name" : "python3-requests", "version" : "2.22.0-2ubuntu1.1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-requests-unixsocket" : [ { "name" : "python3-requests-unixsocket", "version" : "0.2.0-2", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-secretstorage" : [ { "name" : "python3-secretstorage", "version" : "2.3.1-2ubuntu1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-serial" : [ { "name" : "python3-serial", "version" : "3.4-5.1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-service-identity" : [ { "name" : "python3-service-identity", "version" : "18.1.0-5build1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-setuptools" : [ { "name" : "python3-setuptools", "version" : "45.2.0-1ubuntu0.1", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-simplejson" : [ { "name" : "python3-simplejson", "version" : "3.16.0-2ubuntu2", "arch" : "amd64", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-six" : [ { "name" : "python3-six", "version" : "1.14.0-2", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-software-properties" : [ { "name" : "python3-software-properties", "version" : "0.99.9.12", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-systemd" : [ { "name" : "python3-systemd", "version" : "234-3build2", "arch" : "amd64", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-twisted" : [ { "name" : "python3-twisted", "version" : "18.9.0-11ubuntu0.20.04.2", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-twisted-bin" : [ { "name" : "python3-twisted-bin", "version" : "18.9.0-11ubuntu0.20.04.2", "arch" : "amd64", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], "python3-update-manager" : [ { "name" : "python3-update-manager", "version" : "1:20.04.10.11", "arch" : "all", "category" : "python", "origin" : "Ubuntu", "source" : "apt" } ], ============================== "duration" : 2.05353, "event_loop" : null, "uuid" : "f72c96ea-2491-4612-807e-37da9d563d4d" } } } 2023-09-15 03:18:41 EDT - TASK [ovirt-host-deploy-facts : Check if vdsm is preinstalled] ***************** 2023-09-15 03:18:41 EDT - ok: [10.125.0.110] 2023-09-15 03:18:41 EDT - { "status" : "OK", "msg" : "", "data" : { "uuid" : "89475143-4bb1-41ec-806d-53def5431773", "counter" : 23, "stdout" : "ok: [10.125.0.110]", "start_line" : 18, "end_line" : 19, "runner_ident" : "137dc514-5398-11ee-8c1e-00505686a372", "event" : "runner_on_ok", "pid" : 498377, "created" : "2023-09-15T07:18:39.085171", "parent_uuid" : "00505686-a372-da37-7bda-00000000016b", ============================================================ "task_action" : "lineinfile", "task_args" : "", "task_path" : "/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-host-deploy-facts/tasks/host-os.yml:19", "role" : "ovirt-host-deploy-facts", "host" : "10.125.0.110", "remote_addr" : "10.125.0.110", "res" : { "changed" : true, "msg" : "line added", "backup" : "", "diff" : [ { "before" : "", "after" : "", "before_header" : "/etc/os-release (content)", "after_header" : "/etc/os-release (content)" }, { "before_header" : "/etc/os-release (file attributes)", "after_header" : "/etc/os-release (file attributes)" } ], "invocation" : { "module_args" : { ============================================================= "duration" : 0.094784, "event_loop" : "items", "uuid" : "c29c53ba-c732-4237-bb6b-decac24d15b1" } } } 2023-09-15 03:18:41 EDT - TASK [ovirt-host-deploy-vdsm : Install ovirt-hosted-engine-setup package] ****** 2023-09-15 03:18:41 EDT - fatal: [10.125.0.110]: FAILED! => {"changed": false, "msg": ["Could not detect which major revision of yum is in use, which is required to determine module backend.", "You can manually specify use_backend to tell the module whether to use the yum (yum3) or dnf (yum4) backend})"]} 2023-09-15 03:18:41 EDT - { "status" : "OK", "msg" : "", "data" : { "uuid" : "f730ac20-fe5b-4240-8074-f5c04dddeeec", "counter" : 78, "stdout" : "fatal: [10.125.0.110]: FAILED! => {\"changed\": false, \"msg\": [\"Could not detect which major revision of yum is in use, which is required to determine module backend.\", \"You can manually specify use_backend to tell the module whether to use the yum (yum3) or dnf (yum4) backend})\"]}", "start_line" : 67, "end_line" : 68, "runner_ident" : "137dc514-5398-11ee-8c1e-00505686a372", "event" : "runner_on_failed", "pid" : 498377, "created" : "2023-09-15T07:18:41.351584", "parent_uuid" : "00505686-a372-da37-7bda-0000000001ba", "event_data" : { "playbook" : "ovirt-host-deploy.yml", "playbook_uuid" : "ba8a6834-12cf-4837-ae46-f70182bbad00", "play" : "all", "play_uuid" : "00505686-a372-da37-7bda-000000000008", "play_pattern" : "all", "task" : "Install ovirt-hosted-engine-setup package", "task_uuid" : "00505686-a372-da37-7bda-0000000001ba", "task_action" : "yum", "task_args" : "", "task_path" : "/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-host-deploy-vdsm/tasks/packages.yml:6", "role" : "ovirt-host-deploy-vdsm", "host" : "10.125.0.110", "remote_addr" : "10.125.0.110", "res" : { "msg" : [ "Could not detect which major revision of yum is in use, which is required to determine module backend.", "You can manually specify use_backend to tell the module whether to use the yum (yum3) or dnf (yum4) backend})" ], "_ansible_no_log" : false, "changed" : false

2 1

Impossibile to install a new system
by Andrea Chierici 22 Sep '23

22 Sep '23

Dear all, I've been struggling with a new ovirt installation for several weeks but I seem to have no luck at all. I've tried both starting from a centos8 stream system and a ovirt-node, but the engine installation always breaks here: TASK [ovirt.ovirt.hosted_engine_setup : Wait for the host to be up] I keep getting the same problem, it looks like the engine is not "migrated" to the public ip and is stuck on the natted address that the installation process is using during the preliminary phase. Here is a message that seems to confirm my theory: 2023-09-19 15:43:29,259+0200 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'msg': "The task includes an option with an undefined variable. The error was: 'local_vm_ip' is undefined\n\nThe error appears to be in '/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_engine_setup/tasks/sync_on_engine_machine.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Set the name for add_host\n ^ here\n", '_ansible_no_log': False} 2 I know the problem the development of ovirt is facing lately, but I am simply wondering: is there a way to install ovirt 4.5.4 today or is it all already lost??! Any kind of help would be really appreciated. Thanks, Andrea -- Andrea Chierici - INFN-CNAF Viale Berti Pichat 6/2, 40127 BOLOGNA Office Tel: +39 051 2095463 SkypeID ataruz --

3 4

ovirt self-hosted engine 4.5 with cluster version 4.6
by Николаев Алексей 20 Sep '23

20 Sep '23

1 0

In-place upgrade of Windows OS
by Hean-Seng Tan 19 Sep '23

19 Sep '23

We are unable to do an in-place upgrade of the Windows OS for a vm from Windows Server 2012r2 to version 2016. Has anyone ever performed successfully an in-place upgrade? Can you please share the steps?

1 0

oVirt 4.5.4-1: engine-setup does not renew certificates
by Lars Stolpe 15 Sep '23

15 Sep '23

Hi, I wanted to add short names to the webgui apache certificate. I followed these instructions: „Engine can be accessed using alternate host names (or IP addresses) that can configured by adding a new configuration file (for example /etc/ovirt-engine/engine.conf.d/99-custom-sso-setup.conf) with following content: SSO_ALTERNATE_ENGINE_FQDNS="alias1.example.com alias2.example.com" The list of alternate names has to be listed separated by spaces. It's possible to add also IP addresses of engine host, but using IP addresses instead of DNS names is not considered to be a good practise. Run engine-setup to generate new certificates.“ Certificates where not renewed. How do i get engine-setup to renew the apache certificate? oVirt: 4.5.4-1 Regards, Lars

2 2

How to turn off this checks?
by Jorge Visentini 13 Sep '23

13 Sep '23

Hi guys, Is it possible to disable these checks? *1 - Failed to verify Power Management configuration for Host ovirtHost.domain.* *2 - Available swap memory of host ovirtHost.domain [2 MB] is under defined threshold [1024 MB].* I know that is not a good practice, but I don't need it for now. Thanks all! -- Att, Jorge Visentini +55 55 98432-9868

1 0

Version 4.5. How should I properly replace a failed host?
by ziyi Liu 12 Sep '23

12 Sep '23

gluster uses 2+1 arbitration mode My host is damaged and I need to reinstall the system. The host fqdn still uses the old fqdn. What should I do? I found some historical information. Is it still usable for the new version? Found gluster-ansible in GitHub https://github.com/gluster/gluster-ansible/blob/master/playbooks/hc-ansible… See the instructions and configure node_prep_inventory.yml and node_replace_inventory.yml to restore. Execute ansible-playbook -i node_prep_inventory.yml -i node_replace_inventory.yml tasks/replace_node.yml My data disk /dev/sdb has also been initialized by me. Can this ansible script help me create a new partition, create a mount, and complete the recovery of the gluster node?

1 0

ssl stomp reactor timeout
by Chris Smith 12 Sep '23

12 Sep '23

Hi all, I'm using ovirt 4.2 so I'm a little behind, but I'm trying to understand what this error message means. ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connection timeout for host 'x.x.x.x', last response arrived 1549 ms ago. Is this initiated by the engine server or the ovirt host ? Is there a way for me to replicate what ovirt is doing manually via a telnet or curl type command ? Thanks, -- Chris -- Squiz named market leader (second year) in Omdia's Selecting a Digital Experience Management Solution report, 2022-23. Download the report today! <https://www.squiz.net/omdia-report-2022-23?utm_source=Website&utm_medium=Do…>

1 0

engine python sdk ImportError: cannot import name 'Client'
by parallax 11 Sep '23

11 Sep '23

I try to use the Python sdk to download vm disk but get the error: # python3.9 download_disk.py f556403f-2a32-4e35-9f17-dc67864e0ae1 /home/librenms.raw --format raw --config engine2 Traceback (most recent call last): File "download_disk.py", line 31, in <module> from ovirt_imageio import client ModuleNotFoundError: No module named 'ovirt_imageio' or when run it with python3.6: # python3.6 download_disk.py f556403f-2a32-4e35-9f17-dc67864e0ae1 /home/librenms.raw --format raw --config engine2 Traceback (most recent call last): File "download_disk.py", line 31, in <module> from ovirt_imageio import client ImportError: cannot import name 'client' Environment: Rocky Linux release 8.8 Python 3.6.8, Python 3.9.16 python39-ovirt-engine-sdk4-4.6.0-1.el8.x86_64, python3-ovirt-engine-sdk4-4.6.2-1.el8.x86_64 ovirt-release44-4.4.10.3-1.el8.noarch

1 1

Testing oVirt backup and Restore operations 4.3
by Mikhail Po 07 Sep '23

07 Sep '23

There was a desire to conduct a test backup and restore operation ovirt-node-ng 4.3.2-2019031908 Operations performed: - Deployed two test hosts, configured Hosted Engine. Created two test VMs. - Created a backup copy # engine-backup --mode=backup --file=ovirt-full-backup_2023-09-04.tar --log=ovirt-full-backup_2023-09-04.log - Turned off the original Self-hosted Engine and disconnected from Self-hosted Domain Storage # hosted-engine --vm-shutdown # hosted-engine --disconnect-storage - Created a new host using a previously created backup # hosted-engine --deploy --restore-from-file=ovirt-full-backup_2023-09-04.tar Although an error occurred at the end of the installation (after the start of copying to the iSCSI shared storage), but oVirt is accessible via the Web. The error that occurred [ INFO ] TASK [ovirt.hosted_engine_setup : Check for the local bootstrap VM] [ ERROR ] Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The response content type 'text/html; charset=iso-8859-1' isn't the expected XML"} [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook ... [ INFO ] Stage: Clean up [ INFO ] Cleaning temporary resources [ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch. Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20230907201859-ovmmky.log In the Web interface - all Hosts are green - all VMs are green. There is a virtual machine "external-HostedEngineLocal". - Storage Domain "hosted_storage_old_20230907T202958" is present. Any questions: 1. The instructions say that at the end you need to reinstall the hosts. Is it necessary to do this? I ask, because in my case, all hosts are available to the manager, virtual machines are working. What is this operation necessary for? 2. What to do with the "external-HostedEngineLocal" virtual machine? 3. Why the "Error" error occurred: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML"? Can it be related to the lack of access to the Internet (repository) have a VM Engine? When installing hosts, access to the Internet is required (otherwise the installation does not occur), I register access through a proxy in yum.conf

1 0

oVirt 4.5.4-1 OVA export generates incomplete XML tag in vm.ovf
by lars.stolpe＠bvg.de 07 Sep '23

07 Sep '23

Hi, we can't import some OVA VMs into oVirt, that was previously exported in oVirt. Further investigations and testing showed, that in some vm.ovf files the last closing tag is missing ( </ovf:Envelope instead of </ovf:Envelope> ). Intense testing showed the following occurrence: on test DC (4.4.10.7-1): - about half of the VMs have that problem (about 9 VMs) - VMs where the export is working properly are ok, no matter how often i do export - VMs who's vm.ovf is wrong stay that way, no matter how often i do export on production DC (4.4.10.7-1): - all VMs have that problem (49 VMs) - an imported "good" VM formerly exported on test DC, now have a wrong vm.ovf too after re-exporting on production DC - vm.ovf of all VMs stay incorrect, no matter how often i do export on develop DC (4.5.4-1): - the only VM have that problem (1 VM) - vm.ovf of VMs stay incorrect, no matter how often i do export Environment: all DC have independent hardware and are built exactly the same way with the same components oVirt is with hosted-engine test and production: oVirt: 4.4.10.7-1 engine OS: Cent OS Stream 8 host OS: RHEL 8.7 develop: oVirt: 4.5.4-1 engine OS: Cent OS Stream 8 (latest updates till September 1st) host OS: RHEL 8.8 (latest updates till September 1st) Is that a known problem? Is there a way to fix this? regards, Lars

2 3

GLPI agent on oVirt node 4.5.3.2
by Olivier 05 Sep '23

05 Sep '23

I try to install GLPI agent 1.5 (same result with 1.4) from [https://github.com/glpi-project/glpi-agent/releases] on oVirt 4.5 nodes. I get this error: dnf install glpi-agent-1.5-1.noarch.rpm Last metadata expiration check: 2:18:03 ago on Tue 05 Sep 2023 12:25:28 PM CEST. Error: Problem: conflicting requests - nothing provides perl(IO::Socket::SSL) needed by glpi-agent-1.5-1.noarch - nothing provides perl(base) needed by glpi-agent-1.5-1.noarch - nothing provides perl(File::Find) needed by glpi-agent-1.5-1.noarch - nothing provides perl(lib) needed by glpi-agent-1.5-1.noarch - nothing provides perl(Time::HiRes) needed by glpi-agent-1.5-1.noarch - nothing provides perl(threads) needed by glpi-agent-1.5-1.noarch - nothing provides perl(English) needed by glpi-agent-1.5-1.noarch - nothing provides perl(threads::shared) needed by glpi-agent-1.5-1.noarch - nothing provides perl(Net::SSLeay) needed by glpi-agent-1.5-1.noarch - nothing provides perl(URI::Escape) needed by glpi-agent-1.5-1.noarch - nothing provides perl(Digest::SHA) needed by glpi-agent-1.5-1.noarch - nothing provides perl(HTTP::Request) needed by glpi-agent-1.5-1.noarch - nothing provides perl(HTTP::Status) needed by glpi-agent-1.5-1.noarch - nothing provides perl(LWP::UserAgent) needed by glpi-agent-1.5-1.noarch - nothing provides perl(Sys::Syslog) needed by glpi-agent-1.5-1.noarch - nothing provides perl(URI) needed by glpi-agent-1.5-1.noarch - nothing provides perl(Compress::Zlib) needed by glpi-agent-1.5-1.noarch - nothing provides perl(LWP::Protocol::https) needed by glpi-agent-1.5-1.noarch - nothing provides perl(XML::LibXML) needed by glpi-agent-1.5-1.noarch - nothing provides perl(File::Which) needed by glpi-agent-1.5-1.noarch - nothing provides perl(HTTP::Headers) needed by glpi-agent-1.5-1.noarch - nothing provides perl(LWP) needed by glpi-agent-1.5-1.noarch - nothing provides perl(Net::Domain) needed by glpi-agent-1.5-1.noarch - nothing provides perl(HTTP::Cookies) needed by glpi-agent-1.5-1.noarch - nothing provides perl(Net::hostent) needed by glpi-agent-1.5-1.noarch - nothing provides perl(Text::Template) needed by glpi-agent-1.5-1.noarch - nothing provides perl(Net::HTTPS) needed by glpi-agent-1.5-1.noarch - nothing provides perl(Thread::Semaphore) needed by glpi-agent-1.5-1.noarch (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) I can't install the perl modules via dnf install on theses node. I guess is due to the restricted repost : # dnf repolist repo id repo name centos-ovirt45 CentOS Stream 9 - oVirt 4.5 centos-ovirt45-debug CentOS Stream 9 - oVirt 4.5 - Debug centos-ovirt45-source CentOS Stream 9 - oVirt 4.5 - Source crb CentOS Stream 9 - CRB epel Extra Packages for Enterprise Linux 9 - x86_64 epel-cisco-openh264 Extra Packages for Enterprise Linux 9 openh264 (From Cisco) - x86_64 onn-baseos oVirt Node Optional packages from CentOS Stream 9 - BaseOS onn-sap oVirt Node Optional packages from CentOS Stream 9 - SAP ovirt-45-upstream oVirt upstream for CentOS Stream 9 - oVirt 4.5 But I wonder if I can, without risk for my oVirt installation, add new repo ? Any advices ? Thank's

1 0

Install oVirt hosted-engine 4.5.2 problem
by josepascual＠telfy.com 05 Sep '23

05 Sep '23

Hello, I have installed ovirt hosted-entine 4.5.2 and i would like to install other with the same version, but it seems that there is an error with repositories [root@engine9backup ~]# dnf module reset virt CentOS Stream 9 - oVirt 4.5 642 kB/s | 887 kB 00:01 CentOS Stream 9 - oVirt 4.5 - Debug 33 kB/s | 32 kB 00:00 CentOS Stream 9 - oVirt 4.5 - Source 77 kB/s | 76 kB 00:00 determining the fastest mirror (3 hosts).. done. [ === ] 101 B/s | 204 B --:-- ETA oVirt upstream for CentOS Stream 9 - oVirt 4.5 459 B/s | 7.1 kB 00:15 oVirt Node Optional packages from CentOS Stream 9 - BaseOS 7.3 MB/s | 7.8 MB 00:01 oVirt Node Optional packages from CentOS Stream 9 - SAP 4.6 kB/s | 3.9 kB 00:00 Unable to resolve argument virt Error: Problems in request: missing groups or modules: virt [root@engine9backup yum.repos.d]# dnf module list virt Last metadata expiration check: 0:15:28 ago on Tue Sep 5 12:10:58 2023. Error: No matching Modules to list How can i install this version? if i have a backup of virtual machine with 4.5.2 can i restore in ovirt 4.5.4 without problems? Thanks, José Pascual Best Regards

1 0

Install Ovirt
by stevenmendoza2103＠gmail.com 04 Sep '23

04 Sep '23

Hello People; Could somebody help me with the installation of Ovirt , i am newbie , already i tried many times do the proccess but i cant goal it

2 1

error
by hillaryzunigacastro1123＠gmail.com 31 Aug '23

31 Aug '23

what would be the solution ? Errors during downloading metadata for repository 'centos-ovirt45': - Status code: 503 for http://mirrorlist.centos.org/?release-8-stream&arch-x86_64&repo-virt-ovirt- 45 (IP: 18.225.36.18) - Status code: 503 for http://mirrorlist.centos.org/?release-8-stream&arch=x86_64&repo-virt-ovirt- 45 (IP: 35.180.43.213) Error: Failed to download metadata for repo 'centos-ovirt45': Cannot prepare internal mirrorlist: St atus code: 503 for http://mirrorlist.centos.org/?release-8-stream&arch-x86_64&repo-virt-ovirt-… (IP 35.180.43.213)

2 1

I login web interface using IP with Internal Server Error
by javin158＠163.com 31 Aug '23

31 Aug '23

Hi, I’m using ovirt version 4.5.4, and I want to log in to the web interface using an IP address. After modifying the field SSO_ALTERNATE_ENGINE_FQDNS in the configuration file /etc/ovirt-engine/engine.conf.d/11-setup-sso.conf to an IP address, I was able to open the URL https://172.20.52.245/ovirt-engine/. However, when I tried to log in as a user, I encountered the following error: " Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error. More information about this error may be available in the server error log. " I got error message from file /var/log/httpd/ssl_error_log: " [Wed Aug 30 10:42:48.825547 2023] [auth_openidc:error] [pid 64405:tid 139916067432192] [client 172.20.50.110:60017] oidc_authenticate_user: the URL hostname (ovirt-engine-01.labl.local) of the configured OIDCRedirectURI does not match the URL hostname of the URL being accessed (172.20.52.245): the "state" and " session" cookies will not be shared between the two!, referer: https://172.20.52.245/ovirt-engine/" Can you please point me in some direction to try to fix this? Thanks, and best regards cph

1 0

Re: Separate NFS Domain Storage for each Domain Storage
by Mikhail Po 30 Aug '23

30 Aug '23

For the initial installation of the operating system in a virtual machine, you need to mount an ISO image that is located in the storage domain of the NFS type (ISO function). If the NFS server is located within a high-speed LAN (one data center), then there are no problems, if you need to connect an image to a remote site (second data center) with slow channels, then difficulties arise.

3 3

Is there any way to provide only url (noVNC) without ovirt account?
by rak kim 29 Aug '23

29 Aug '23

I would like to provide the vm's console to my customers. I want to provide only url without ovirt account, is there any way? I usally use novnc but if other ways can, ill use

1 0

Separate NFS Domain Storage for each Domain Storage
by Mikhail Po 28 Aug '23

28 Aug '23

Is it possible in oVirt 4.3 for each Data Center to create a separate ISO Domain Storage, with NFS type, on one of the hosts belonging to this Data Center? For example, in the case of connecting a remote site in a separate Data Center to an existing Engine through slow network links.

2 1

engine-config -s UserSessionTimeOutInterval=X problem
by marek 28 Aug '23

28 Aug '23

ovirt 4.5.4, standalone engine, centos 8 stream [root@ovirt ~]# engine-config -g UserSessionTimeOutInterval Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false UserSessionTimeOutInterval: 30 version: general [root@ovirt ~]# engine-config -s UserSessionTimeOutInterval=60 Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false Cannot set value 60 to key UserSessionTimeOutInterval. any ideas where is the problem? Marek

5 6

Node Intel I7 13700 Raptor Lake
by sarpi17＠gmail.com 22 Aug '23

22 Aug '23

Hello, how are you? I have an Ovirt Engine Version 4.4.10 installed. I installed Ovirt Node on a computer with an Intel I7 13700 Raptor Lake processor. When I want to link that node with the Engine I get the error that the CPU family is not correct, I tried with several of those that appear, but none is correct and Raptor Lake does not appear. What can I do? Thanks

1 0

oVirt 4.5.5 VDSM error
by cmicarelli＠fdmservices.it 22 Aug '23

22 Aug '23

HI guys, I've installed a new ovirt stack 4.5 and everything works fine. After the upgrade to 4.5.5 VDSM doesn't work correctly --------------------------------------------------------------------------------------- [root@lce8cdpnode2 opt]# systemctl status vdsmd.service ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2023-08-22 14:34:44 CEST; 19min ago Process: 3332 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) Main PID: 3413 (vdsmd) Tasks: 56 (limit: 823896) Memory: 312.3M CGroup: /system.slice/vdsmd.service ├─3413 /usr/bin/python3 /usr/libexec/vdsm/vdsmd └─4704 /usr/libexec/ioprocess --read-pipe-fd 59 --write-pipe-fd 58 --max-threads 10 --max-queued-requests 10 Aug 22 14:34:44 lce8cdpnode2..local vdsmd_init_common.sh[3332]: vdsm: Running test_space Aug 22 14:34:44 lce8cdpnode2..local vdsmd_init_common.sh[3332]: vdsm: Running test_lo Aug 22 14:34:44 lce8cdpnode2..local systemd[1]: Started Virtual Desktop Server Manager. Aug 22 14:34:47 lce8cdpnode2..local vdsm[3413]: WARN MOM not available. Error: [Errno 2] No such file or directory Aug 22 14:34:47 lce8cdpnode2..local vdsm[3413]: WARN MOM not available, KSM stats will be missing. Error: Aug 22 14:41:30 lce8cdpnode2..local vdsm[3413]: WARN unhandled write event Aug 22 14:41:40 lce8cdpnode2..local vdsm[3413]: WARN unhandled write event Aug 22 14:41:43 lce8cdpnode2..local vdsm[3413]: ERROR Error getting managedvolume connector info: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc> Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/host/caps.py", line 161, in get caps["connector_info"] = managedvolume.connector_info() File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 63, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 78, in connector_info return run_helper("connector_info") File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 184, in run_helper sub_cmd, cmd_input=cmd_input) File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 38, in __call__ return callMethod() File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 36, in <lambda> **kwargs) File "<string>", line 2, in managedvolume_run_helper File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod raise convert_to_error(kind, result) vdsm.storage.exception.ManagedVolumeHelperFailed: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc> Aug 22 14:41:43 lce8cdpnode2..local vdsm[3413]: ERROR Error getting managedvolume connector info: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc> Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/host/caps.py", line 161, in get caps["connector_info"] = managedvolume.connector_info() File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 63, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 78, in connector_info return run_helper("connector_info") File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 184, in run_helper sub_cmd, cmd_input=cmd_input) File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 38, in __call__ return callMethod() File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 36, in <lambda> **kwargs) File "<string>", line 2, in managedvolume_run_helper File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod raise convert_to_error(kind, result) vdsm.storage.exception.ManagedVolumeHelperFailed: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc> Aug 22 14:41:45 lce8cdpnode2..local vdsm[3413]: ERROR Error getting managedvolume connector info: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc> Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/host/caps.py", line 161, in get caps["connector_info"] = managedvolume.connector_info() File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 63, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 78, in connector_info return run_helper("connector_info") File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 184, in run_helper sub_cmd, cmd_input=cmd_input) File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 38, in __call__ return callMethod() File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 36, in <lambda> **kwargs) File "<string>", line 2, in managedvolume_run_helper File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod raise convert_to_error(kind, result) vdsm.storage.exception.ManagedVolumeHelperFailed: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc> lines 7-72/72 (END) ------------------------------ Someone have had same problem Regards C

2 1

ovirt.hosted_engine_setup : Obtain SSO token using username/password credentials
by cmicarelli＠fdmservices.it 22 Aug '23

22 Aug '23

Hi Guys, I'm tried to install ovirt 4.5 on centos 8 stream and everytime file with this error ovirt.hosted_engine_setup : Obtain SSO token using username/password credentials Someone could help me? Regrards

2 2

Re: Active Active Stretched Node cluster not working as expected.
by Gianluca Cecchi 22 Aug '23

22 Aug '23

On Tue, Aug 22, 2023 at 8:52 AM Gaurang Patel <gaurang.patel(a)allotgroup.com> wrote: > Suggest any other solution if any. > > > Your subject says "not working as expected" and you refer to the RHV DR guides. My comment is that in my opinion HCI architecture is not expected to work / be supported for active active DR scenarios as described there. But you can follow general GlusterFS indications for solving split brain scenarios, like this one: https://docs.gluster.org/en/main/Troubleshooting/resolving-splitbrain/ and / or submit your question also to GlusterFS mailing lists or see if here you find useful information for supported configurations, quorum setup to prevent split brain conditions, ecc: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/h… https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/h… HIH, Gianluca

1 0

Active Active Stretched Node cluster not working as expected.
by gaurang.patel＠allotgroup.com 22 Aug '23

22 Aug '23

Hi, i have a Six node Stretched HCI cluster with 2 x 3 =6 distributed Replicated bricks, want to achieve Active - Active Disaster Recovery , if my three hosts in DC power Down or Failed my DR Site Three node should take care of running virtual machines. During Split-brain scenario Ovirt-Hosted-Engine VM goes in PAUSE mode and not restarting on DR hosts and waited for long time, due to this my all production VMs are not restarting on DR Site. please guide to achieve Active-Active Disaster Recovery with 6 node stretched cluster. i referred document hosted official site but did not get success, it would be great help of anybody can extend help or guide me for the same. https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/pd… My Setup insight. root@phnode06 /]# gluster volume status Status of volume: data Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 172.16.1.1:/gluster_bricks/data/data 58744 0 Y 1746524 Brick 172.16.1.2:/gluster_bricks/data/data 58732 0 Y 2933385 Brick 172.16.1.3:/gluster_bricks/data/data 60442 0 Y 6158 Brick phnode04..local:/gluster_bricks /data/data1 51661 0 Y 5905 Brick phnode05..local:/gluster_bricks /data/data1 52177 0 Y 6178 Brick phnode06..local:/gluster_bricks /data/data1 59910 0 Y 6208 Self-heal Daemon on localhost N/A N/A Y 5878 Self-heal Daemon on phnode03..local N/A N/A Y 6133 Self-heal Daemon on gluster01..local N/A N/A Y 6072 Self-heal Daemon on phnode05..local N/A N/A Y 5751 Self-heal Daemon on gluster02..local N/A N/A Y 5746 Self-heal Daemon on phnode04..local N/A N/A Y 4734 Task Status of Volume data ------------------------------------------------------------------------------ Task : Rebalance ID : 1c30fbfa-2707-457e-b844-ccaf2e07aa7f Status : completed Status of volume: engine Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 172.16.1.1:/gluster_bricks/engine/eng ine 53723 0 Y 1746535 Brick 172.16.1.2:/gluster_bricks/engine/eng ine 49937 0 Y 2933395 Brick 172.16.1.3:/gluster_bricks/engine/eng ine 53370 0 Y 6171 Brick phnode04..local:/gluster_bricks /engine/engine1 52024 0 Y 5916 Brick phnode05..local:/gluster_bricks /engine/engine1 51785 0 Y 6189 Brick phnode06..local:/gluster_bricks /engine/engine1 55443 0 Y 6221 Self-heal Daemon on localhost N/A N/A Y 5878 Self-heal Daemon on gluster02..local N/A N/A Y 5746 Self-heal Daemon on gluster01..local N/A N/A Y 6072 Self-heal Daemon on phnode05..local N/A N/A Y 5751 Self-heal Daemon on phnode03..local N/A N/A Y 6133 Self-heal Daemon on phnode04..local N/A N/A Y 4734 Task Status of Volume engine ------------------------------------------------------------------------------ Task : Rebalance ID : 21304de4-b100-4860-b408-5e58103080a2 Status : completed Status of volume: vmstore Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 172.16.1.1:/gluster_bricks/vmstore/vm store 50244 0 Y 1746546 Brick 172.16.1.2:/gluster_bricks/vmstore/vm store 51607 0 Y 2933405 Brick 172.16.1.3:/gluster_bricks/vmstore/vm store 49835 0 Y 6182 Brick phnode04..local:/gluster_bricks /vmstore/vmstore1 54098 0 Y 5927 Brick phnode05..local:/gluster_bricks /vmstore/vmstore1 56565 0 Y 6200 Brick phnode06..local:/gluster_bricks /vmstore/vmstore1 58653 0 Y 1853456 Self-heal Daemon on localhost N/A N/A Y 5878 Self-heal Daemon on gluster02..local N/A N/A Y 5746 Self-heal Daemon on gluster01..local N/A N/A Y 6072 Self-heal Daemon on phnode05..local N/A N/A Y 5751 Self-heal Daemon on phnode03..local N/A N/A Y 6133 Self-heal Daemon on phnode04..local N/A N/A Y 4734 Task Status of Volume vmstore ------------------------------------------------------------------------------ Task : Rebalance ID : 7d1849f8-433a-43b1-93bc-d674a7b0aef3 Status : completed [root@phnode06 /]# hosted-engine --vm-status --== Host phnode01..local (id: 1) status ==-- Host ID : 1 Host timestamp : 9113939 Score : 3400 Engine status : {"vm": "up", "health": "good", "detail": "Up"} Hostname : phnode01..local Local maintenance : False stopped : False crc32 : 00979124 conf_on_shared_storage : True local_conf_timestamp : 9113939 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=9113939 (Sat Aug 19 14:39:41 2023) host-id=1 score=3400 vm_conf_refresh_time=9113939 (Sat Aug 19 14:39:41 2023) conf_on_shared_storage=True maintenance=False state=EngineUp stopped=False --== Host phnode03..local (id: 2) status ==-- Host ID : 2 Host timestamp : 405629 Score : 3400 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : phnode03..local Local maintenance : False stopped : False crc32 : 0b1c7489 conf_on_shared_storage : True local_conf_timestamp : 405630 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=405629 (Sat Aug 19 14:39:43 2023) host-id=2 score=3400 vm_conf_refresh_time=405630 (Sat Aug 19 14:39:43 2023) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host phnode02..local (id: 3) status ==-- Host ID : 3 Host timestamp : 9069311 Score : 3400 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : phnode02..local Local maintenance : False stopped : False crc32 : b84baa31 conf_on_shared_storage : True local_conf_timestamp : 9069311 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=9069311 (Sat Aug 19 14:39:41 2023) host-id=3 score=3400 vm_conf_refresh_time=9069311 (Sat Aug 19 14:39:41 2023) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host phnode04..local (id: 4) status ==-- Host ID : 4 Host timestamp : 1055893 Score : 3400 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : phnode04..local Local maintenance : False stopped : False crc32 : 58a9b84c conf_on_shared_storage : True local_conf_timestamp : 1055893 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=1055893 (Sat Aug 19 14:39:38 2023) host-id=4 score=3400 vm_conf_refresh_time=1055893 (Sat Aug 19 14:39:38 2023) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host phnode05..local (id: 5) status ==-- Host ID : 5 Host timestamp : 404187 Score : 3400 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : phnode05..local Local maintenance : False stopped : False crc32 : 48cf9186 conf_on_shared_storage : True local_conf_timestamp : 404187 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=404187 (Sat Aug 19 14:39:40 2023) host-id=5 score=3400 vm_conf_refresh_time=404187 (Sat Aug 19 14:39:40 2023) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host phnode06..local (id: 6) status ==-- Host ID : 6 Host timestamp : 1055902 Score : 3400 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : phnode06..local Local maintenance : False stopped : False crc32 : 913ff6c2 conf_on_shared_storage : True local_conf_timestamp : 1055902 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=1055902 (Sat Aug 19 14:39:48 2023) host-id=6 score=3400 vm_conf_refresh_time=1055902 (Sat Aug 19 14:39:48 2023) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False [root@phnode06 /]#

3 2

Issue - Bonding 802.3ad
by Anthony Bustillos 21 Aug '23

21 Aug '23

Hello team, I have this warning when I tried, to configure two ports with boonding 802.3ad. Warning: Bond is in link aggregation mode (Mode 4 ) but no partner mac has been reported for it. At least one slave has a different aggregator id. how I can to solved this warning? best Regards

2 2