- Users - oVirt List Archives

CentOS Stream 9 Ovirt 4.5
by eevans＠digitaldatatechs.com 13 Jan '25

13 Jan '25

I have a small lab set up with 3 ovirt Centos 9 nodes and one Centos 9 Ovirt "controller node" that manages. 2 problems: It will no sync networks on hosts. It will no launch a vm unless you go to system and choose an instance size. It will boot and the network card has to be hot added. I'm not sure what I'm missing, I can give log entries showing several bad lines in diggerent modules.' Engine log while updating hosts: tail -n 100 -f engine.log ' 2025-01-09 02:43:15,722-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName = kvm02 2025-01-09 02:43:15,722-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm02, VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2025-01-09 02:43:15,729-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'}), log id: 5e145d4b 2025-01-09 02:43:15,729-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:18,057-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:18,057-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName = kvm04 2025-01-09 02:43:18,057-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:18,057-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 5e145d4b 2025-01-09 02:43:18,057-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:22,344-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'SpmStatusVDSCommand(HostName = kvm04, SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:22,344-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Irs placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover 2025-01-09 02:43:22,349-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'}), log id: 28df8c85 2025-01-09 02:43:22,349-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:25,267-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm01 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:25,267-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:25,267-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName = kvm01 2025-01-09 02:43:25,267-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:25,267-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 28df8c85 2025-01-09 02:43:25,267-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:25,275-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Irs placed on server 'acf66bf6-e159-4705-be47-484c99e746dd' failed. Proceed Failover 2025-01-09 02:43:25,279-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'}), log id: 20ec709f 2025-01-09 02:43:25,279-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:27,675-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm02 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:27,675-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:27,675-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName = kvm02 2025-01-09 02:43:27,676-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:27,676-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 20ec709f 2025-01-09 02:43:27,676-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:27,741-05 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-572) [6727848e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage 2025-01-09 02:43:27,765-05 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-572) [6727848e] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2025-01-09 02:43:27,789-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (EE-ManagedThreadFactory-engine-Thread-571) [] Finished reconstruct for pool '39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue 2025-01-09 02:43:30,154-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] START, GlusterServersListVDSCommand(HostName = kvm04, VdsIdVDSCommandParametersBase:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 770c0b37 2025-01-09 02:43:30,362-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] FINISH, GlusterServersListVDSCommand, return: [192.168.254.108/24:CONNECTED, ovirt.digitaldatatechs.com:CONNECTED, kvm02.digitaldatatechs.com:CONNECTED, kvm01.digitaldatatechs.com:CONNECTED], log id: 770c0b37 2025-01-09 02:43:30,381-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] START, AddGlusterServerVDSCommand(HostName = kvm04, AddGlusterServerVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 3b2ce207 2025-01-09 02:43:33,630-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] FINISH, AddGlusterServerVDSCommand, return: , log id: 3b2ce207 2025-01-09 02:43:33,640-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] EVENT_ID: GLUSTER_SERVER_ADD_FAILED(4,436), Failed to add host kvm04 into Cluster dds_cluster. Add host failed: rc=107 out=() err=['Probe returned with Transport endpoint is not connected'] 2025-01-09 02:43:33,641-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] START, GlusterVolumesListVDSCommand(HostName = kvm04, GlusterVolumesListVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 6a99b1ab 2025-01-09 02:43:33,748-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] FINISH, GlusterVolumesListVDSCommand, return: {b79a00d7-cc2a-48d8-b881-699991f3d930=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@205714d2}, log id: 6a99b1ab 2025-01-09 02:43:37,689-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'SpmStatusVDSCommand(HostName = kvm02, SpmStatusVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:37,691-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' TaskStatusListReturn:{status='Status [code=654, message=Not SPM: ()]'} ' 2025-01-09 02:43:37,691-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName = kvm02 2025-01-09 02:43:37,691-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm02, VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2025-01-09 02:43:37,698-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'}), log id: 201774c9 2025-01-09 02:43:37,698-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:38,381-05 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-10) [] There is no host with more than 10 running guests, no balancing is needed 2025-01-09 02:43:38,400-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-59) [] START, GlusterTasksListVDSCommand(HostName = kvm02, VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'}), log id: 702ab66f 2025-01-09 02:43:38,515-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-59) [] FINISH, GlusterTasksListVDSCommand, return: [], log id: 702ab66f 2025-01-09 02:43:40,598-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:40,598-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName = kvm01 2025-01-09 02:43:40,598-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:40,598-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 201774c9 2025-01-09 02:43:40,598-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:40,606-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Irs placed on server 'acf66bf6-e159-4705-be47-484c99e746dd' failed. Proceed Failover 2025-01-09 02:43:40,611-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'}), log id: 164d36cb 2025-01-09 02:43:40,611-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:42,950-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm04 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:42,950-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:42,950-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName = kvm04 2025-01-09 02:43:42,950-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:42,950-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 164d36cb 2025-01-09 02:43:42,950-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:42,960-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'SpmStatusVDSCommand(HostName = kvm04, SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:42,960-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Irs placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover 2025-01-09 02:43:42,964-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'}), log id: 863c329 2025-01-09 02:43:42,965-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:45,328-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:45,328-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName = kvm02 2025-01-09 02:43:45,328-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:45,328-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 863c329 2025-01-09 02:43:45,328-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:45,388-05 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-580) [2e23e34] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage 2025-01-09 02:43:45,412-05 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-580) [2e23e34] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2025-01-09 02:43:45,436-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (EE-ManagedThreadFactory-engine-Thread-579) [] Finished reconstruct for pool '39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue 2025-01-09 02:43:48,763-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] START, GlusterServersListVDSCommand(HostName = kvm04, VdsIdVDSCommandParametersBase:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: db1640e 2025-01-09 02:43:48,970-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] FINISH, GlusterServersListVDSCommand, return: [192.168.254.108/24:CONNECTED, ovirt.digitaldatatechs.com:CONNECTED, kvm02.digitaldatatechs.com:CONNECTED, kvm01.digitaldatatechs.com:CONNECTED], log id: db1640e 2025-01-09 02:43:48,983-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] START, AddGlusterServerVDSCommand(HostName = kvm04, AddGlusterServerVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 717f84ae 2025-01-09 02:43:52,255-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] FINISH, AddGlusterServerVDSCommand, return: , log id: 717f84ae 2025-01-09 02:43:52,277-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] EVENT_ID: GLUSTER_SERVER_ADD_FAILED(4,436), Failed to add host kvm04 into Cluster dds_cluster. Add host failed: rc=107 out=() err=['Probe returned with Transport endpoint is not connected'] 2025-01-09 02:43:52,278-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] START, GlusterVolumesListVDSCommand(HostName = kvm04, GlusterVolumesListVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 6a43f9e6 2025-01-09 02:43:52,384-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] FINISH, GlusterVolumesListVDSCommand, return: {b79a00d7-cc2a-48d8-b881-699991f3d930=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@205714d2}, log id: 6a43f9e6 2025-01-09 02:43:55,341-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'SpmStatusVDSCommand(HostName = kvm02, SpmStatusVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:55,343-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' TaskStatusListReturn:{status='Status [code=654, message=Not SPM: ()]'} ' 2025-01-09 02:43:55,343-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName = kvm02 2025-01-09 02:43:55,343-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm02, VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2025-01-09 02:43:55,350-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'}), log id: 35cf220b 2025-01-09 02:43:55,350-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:57,661-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:57,661-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName = kvm04 2025-01-09 02:43:57,661-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:57,661-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 35cf220b 2025-01-09 02:43:57,661-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:57,671-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'SpmStatusVDSCommand(HostName = kvm04, SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:57,671-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Irs placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover 2025-01-09 02:43:57,675-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'}), log id: 2ff24c19 2025-01-09 02:43:57,676-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:00,065-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm02 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:00,065-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:00,065-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName = kvm02 2025-01-09 02:44:00,065-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:00,065-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 2ff24c19 2025-01-09 02:44:00,065-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:04,367-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'SpmStatusVDSCommand(HostName = kvm02, SpmStatusVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:44:04,367-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Irs placed on server 'cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6' failed. Proceed Failover 2025-01-09 02:44:04,372-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'}), log id: 23ed7788 2025-01-09 02:44:04,372-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:07,292-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm01 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:07,292-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:07,292-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName = kvm01 2025-01-09 02:44:07,292-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:07,292-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 23ed7788 2025-01-09 02:44:07,292-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:07,375-05 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-594) [6e7f60bb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage 2025-01-09 02:44:07,412-05 INFO [org.ovirt.engine.core.bll.network.host.SyncAllHostNetworksCommand] (default task-7) [9913b205-9b2e-4f8a-86e3-048a9cdff2ba] Running command: SyncAllHostNetworksCommand internal: false. Entities affected : ID: acf66bf6-e159-4705-be47-484c99e746dd Type: VDSAction group CONFIGURE_HOST_NETWORK with role type ADMIN 2025-01-09 02:44:07,421-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, GlusterServersListVDSCommand(HostName = kvm04, VdsIdVDSCommandParametersBase:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 340c8f9d 2025-01-09 02:44:07,423-05 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-594) [6e7f60bb] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2025-01-09 02:44:07,423-05 INFO [org.ovirt.engine.core.bll.network.host.PersistentHostSetupNetworksCommand] (default task-7) [7814c7ae] Running command: PersistentHostSetupNetworksCommand internal: true. Entities affected : ID: acf66bf6-e159-4705-be47-484c99e746dd Type: VDS 2025-01-09 02:44:07,448-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (EE-ManagedThreadFactory-engine-Thread-593) [] Finished reconstruct for pool '39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue 2025-01-09 02:44:07,460-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [7814c7ae] EVENT_ID: PERSIST_HOST_SETUP_NETWORK_ON_HOST(10,902), (1/1): Applying network's changes on host kvm01. (User: admin@ovirt@internalkeycloak-authz) 2025-01-09 02:44:07,461-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Before acquiring lock-timeout 'EngineLock:{exclusiveLocks='[HOST_NETWORKacf66bf6-e159-4705-be47-484c99e746dd=HOST_NETWORK]', sharedLocks=''}' 2025-01-09 02:44:07,461-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Lock-timeout acquired to object 'EngineLock:{exclusiveLocks='[HOST_NETWORKacf66bf6-e159-4705-be47-484c99e746dd=HOST_NETWORK]', sharedLocks=''}' 2025-01-09 02:44:07,497-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Running command: HostSetupNetworksCommand internal: true. Entities affected : ID: acf66bf6-e159-4705-be47-484c99e746dd Type: VDSAction group CONFIGURE_HOST_NETWORK with role type ADMIN 2025-01-09 02:44:07,497-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Before acquiring lock in order to prevent monitoring for host 'kvm01' from data-center 'DDS_Data_Center' 2025-01-09 02:44:07,497-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Lock acquired, from now a monitoring of host will be skipped for host 'kvm01' from data-center 'DDS_Data_Center' 2025-01-09 02:44:07,498-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-7) [7814c7ae] START, HostSetupNetworksVDSCommand(HostName = kvm01, HostSetupNetworksVdsCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vds='Host[kvm01,acf66bf6-e159-4705-be47-484c99e746dd]', rollbackOnFailure='true', commitOnSuccess='true', connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='false', bonding='false', networkName='vlan3', vdsmName='vlan3', nicName='eno2', vlan='3', vmNetwork='true', stp='false', properties='[]', ipv4BootProtocol='STATIC_IP', ipv4Address='172.50.26.252', ipv4Netmask='255.255.255.0', ipv4Gateway='172.50.26.1', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}, HostNetwork:{defaultRoute='true', bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt', nicName='eno1', vlan='null', vmNetwork='true', stp='false', properties='[]', ipv4BootProtocol='STAT IC_IP', ipv4Address='192.168.254.252', ipv4Netmask='255.255.255.0', ipv4Gateway='192.168.254.1', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='[NameServer:{address='192.168.254.222'}, NameServer:{address='192.168.254.232'}]'}]', removedNetworks='[]', bonds='[]', removedBonds='[]', clusterSwitchType='LEGACY', managementNetworkChanged='true'}), log id: 56a27c18 2025-01-09 02:44:07,499-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-7) [7814c7ae] FINISH, HostSetupNetworksVDSCommand, return: , log id: 56a27c18 2025-01-09 02:44:07,631-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, GlusterServersListVDSCommand, return: [192.168.254.108/24:CONNECTED, ovirt.digitaldatatechs.com:CONNECTED, kvm02.digitaldatatechs.com:CONNECTED, kvm01.digitaldatatechs.com:CONNECTED], log id: 340c8f9d 2025-01-09 02:44:07,642-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, AddGlusterServerVDSCommand(HostName = kvm04, AddGlusterServerVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 7c25cce3 2025-01-09 02:44:10,879-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, AddGlusterServerVDSCommand, return: , log id: 7c25cce3 2025-01-09 02:44:10,923-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] EVENT_ID: GLUSTER_SERVER_ADD_FAILED(4,436), Failed to add host kvm04 into Cluster dds_cluster. Add host failed: rc=107 out=() err=['Probe returned with Transport endpoint is not connected'] 2025-01-09 02:44:10,923-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, GlusterVolumesListVDSCommand(HostName = kvm04, GlusterVolumesListVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: effd1e8 2025-01-09 02:44:11,033-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, GlusterVolumesListVDSCommand, return: {b79a00d7-cc2a-48d8-b881-699991f3d930=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@205714d2}, log id: effd1e8 2025-01-09 02:44:17,314-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' TaskStatusListReturn:{status='Status [code=654, message=Not SPM: ()]'} ' 2025-01-09 02:44:17,315-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName = kvm01 2025-01-09 02:44:17,315-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm01, VdsIdVDSCommandParametersBase:{hostId='acf66bf6-e159-4705-be47-484c99e746dd'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2025-01-09 02:44:17,331-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] START, ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'}), log id: bcbf59e 2025-01-09 02:44:17,331-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:18,272-05 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [7814c7ae] EVENT_ID: VDS_NETWORKS_OUT_OF_SYNC(1,110), Host kvm01's following network(s) are not synchronized with their Logical Network configuration: ovirtmgmt,vlan3. 2025-01-09 02:44:18,283-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Host setup networks finished. Lock released. Monitoring can run now for host 'kvm01' from data-center 'DDS_Data_Center' 2025-01-09 02:44:18,285-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Lock freed to object 'EngineLock:{exclusiveLocks='[HOST_NETWORKacf66bf6-e159-4705-be47-484c99e746dd=HOST_NETWORK]', sharedLocks=''}' 2025-01-09 02:44:18,305-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [7814c7ae] EVENT_ID: PERSIST_SETUP_NETWORK_ON_HOST_FINISHED(10,903), (1/1): Successfully applied changes on host kvm01. (User: admin@ovirt@internalkeycloak-authz) 2025-01-09 02:44:18,316-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [7814c7ae] EVENT_ID: HOST_SYNC_ALL_NETWORKS_FINISHED(10,901), Managed to sync all host kvm01 networks. 2025-01-09 02:44:19,668-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm04 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:19,668-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:19,668-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName = kvm04 2025-01-09 02:44:19,668-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:19,668-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: bcbf59e 2025-01-09 02:44:19,668-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:19,679-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'SpmStatusVDSCommand(HostName = kvm04, SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:44:19,679-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Irs placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover 2025-01-09 02:44:19,684-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] START, ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'}), log id: 251dbd22 2025-01-09 02:44:19,684-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:22,181-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:22,181-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName = kvm01 2025-01-09 02:44:22,181-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:22,181-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 251dbd22 2025-01-09 02:44:22,181-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:22,189-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Irs placed on server 'acf66bf6-e159-4705-be47-484c99e746dd' failed. Proceed Failover 2025-01-09 02:44:22,198-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] START, ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'}), log id: 1f60ebb1 2025-01-09 02:44:22,198-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:24,581-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:24,581-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName = kvm02 2025-01-09 02:44:24,581-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:24,582-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 1f60ebb1 2025-01-09 02:44:24,582-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:24,658-05 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-610) [2af75e62] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage 2025-01-09 02:44:24,691-05 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-610) [2af75e62] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2025-01-09 02:44:24,715-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (EE-ManagedThreadFactory-engine-Thread-609) [] Finished reconstruct for pool '39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue While attempting to start vm will be sent after but I think the net allocation issue with hosts is the center of the vm issues,,

2 1

Restart oVirt-Engine
by Jeremey Wise 09 Jan '25

09 Jan '25

How ,without reboot of hosting system, do I restart the oVirt engine? # I tried below but do not seem to effect the virtual machine [root@thor iso]# systemctl restart ov ovirt-ha-agent.service ovirt-imageio.service ovn-controller.service ovs-delete-transient-ports.service ovirt-ha-broker.service ovirt-vmconsole-host-sshd.service ovsdb-server.service ovs-vswitchd.service [root@thor iso]# # You cannot restart the VM " HostedEngine " as it responses: Error while executing action: HostedEngine: - Cannot restart VM. This VM is not managed by the engine. Reason is I had to do some work on a node. Reboot it.. it is back up.. network is all fine.. Cockpit working fine... and gluster fine.. But oVirt-Engine refuses to accept the node is up. -- p <jeremey.wise(a)gmail.com>enguinpages

4 4

Host deploy failure: Configure OVN for oVirt
by stephan.badenhorst＠fnb.co.za 05 Jan '25

05 Jan '25

Good day, I am running into a problem during host deploy via the oVirt engine GUI since upgrading to ovirt-engine-4.5.5-1.el8. The "Configure OVN for oVirt" task seem to fail when trying to run the vdsm-tool ovn-config command. Host deploy used to work fine when the engine was on version 4.5.4. Anyone that can guide me on the right path to get past this issue? Does not seem to be a new problem - https://lists.ovirt.org/archives/list/users@ovirt.org/thread/IDLGSBQFX35EHH… Log extract: 2024-02-01 14:48:19 SAST - TASK [ovirt-provider-ovn-driver : Configure OVN for oVirt] ********************* . . . "stdout" : "fatal: [mob-r1-l-ovirt-aa-1-23.x.fnb.co.za]: FAILED! => {\"changed\": true, \"cmd\": [\"vdsm-tool\", \"ovn-config\", \"192.168.2.100\", \"host23.mydomain.com\"], \"delta\": \"0:00:00.538143\", \"end \": \"2024-02-01 14:48:20.596823\", \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2024-02-01 14:48:20.058680\", \"stderr\": \"Traceback (most recent call last):\\n File \\\"/usr/lib/python3.6/site-packages/vdsm/t ool/ovn_config.py\\\", line 117, in get_network\\n return networks[net_name]\\nKeyError: 'host23.mydomain.com'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most rec ent call last):\\n File \\\"/usr/bin/vdsm-tool\\\", line 195, in main\\n return tool_command[cmd][\\\"command\\\"](*args)\\n File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\\n ip_address = get_ip_addr(get_network(network_caps(), net_name))\\n File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 119, in get_network\\n raise NetworkNotFoundError(net_name)\\nvdsm.tool.ovn _config.NetworkNotFoundError: host23.mydomain.com\", \"stderr_lines\": [\"Traceback (most recent call last):\", \" File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 117, in get_network \", \" return networks[net_name]\", \"KeyError: 'host23.mydomain.com'\", \"\", \"During handling of the above exception, another exception occurred:\", \"\", \"Traceback (most recent call last):\", \" File \ \\"/usr/bin/vdsm-tool\\\", line 195, in main\", \" return tool_command[cmd][\\\"command\\\"](*args)\", \" File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\", \" ip_address = get_ip_addr(get_network(network_caps(), net_name))\", \" File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 119, in get_network\", \" raise NetworkNotFoundError(net_name)\", \"vdsm.tool.ovn_config. NetworkNotFoundError: host23.mydomain.com\"], \"stdout\": \"\", \"stdout_lines\": []}", Thanks in advance!! Stephan

5 5

Re: Host deploy failure: Configure OVN for oVirt
by Andrea Cavegn 03 Jan '25

03 Jan '25

We use ovirt-engine 4.5.5-1.el8 and for us the following did the trick: 1. First you have to decide, if you need the ovn/ovs functionality. This is a cluster level decision! * If needed, you need to have a functional (You can test it) external network provider configured (Administration -> Providers -> External Network Provider) ?? As far as I know, a default "ovirt-provider-ovn" is already present in each installation * Then in the cluster option you choose the "Default Network Provider". If you don't need OVS/OVN, then select "No Default Provider", else select the one you need * If you change here anything, ovirt want you to reinstall all your hosts in this cluster! 2. Then, replace the ansible file in your ovirt-engine: * /usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-provider-ovn-driver/tasks/configure.yml * The new version can be found here: https://github.com/oVirt/ovirt-engine/blob/master/packaging/ansible-runner-… as far as I can tell, only the last line (when statement) changes Probably, only ppl who don't need ovn/ovs had (Setting: "No Default Provider") had this problem. 1. Reinstall the hosts As always, know what you do, before you anything.

1 0

Enroll Certificate
by Денис Панкратьев 03 Jan '25

03 Jan '25

Hello! My ovirt host (4.5.5) certificate is about to expire (in 46 days, Jan 20, 2025). d.pankratiev@zabbix ~]$ echo | openssl s_client -servername ovirt1.gtc.local -connect XXX.XXX.XXX.XXX.XXX:9090 2>/dev/null | openssl x509 -noout -dates 2>/dev/null notBefore=Dec 22 05:27:04 2023 GMT notAfter=Jan 20 05:27:04 2025 GMT 1. I have migrated all VMs to another host 2. I put the host in maintenance mode (Managеment -> Maintenance) 3. Do enroll certificate (Installation -> Enroll Certificate) 4. Everything went successfully, without errors (Enrolling certificate for host ovirt1.gtc.local was completed successfully). But the certificate date did not change, still expire (after 46 days, Jan 20 2025). d.pankratiev@zabbix ~]$ echo | openssl s_client -servername ovirt1.gtc.local -connect XXX.XXX.XXX.XXX.XXX:9090 2>/dev/null | openssl x509 -noout -dates 2>/dev/null notBefore=Dec 22 05:27:04 2023 GMT notAfter=Jan 20 05:27:04 2025 GMT What did I do wrong? Is there anything else I need to do?

5 5

oVirt and TPM
by Fabrice Bacchella 30 Dec '24

30 Dec '24

Did some one use TPM in oVirt ? I tried to active it but failed to use. The VM is configured to use an EFI secure boot. The VM (a RHEL 9.5 OS) see it, but I can’t communicate with it: $ dmesg | grep -i tpm [ 0.000000] efi: SMBIOS=0x7ddd5000 TPMFinalLog=0x7dfe7000 ACPI=0x7df7e000 ACPI 2.0=0x7df7e014 MEMATTR=0x7cbd9018 MOKvar=0x7dd82000 RNG=0x7df71018 TPMEventLog=0x7c544018 [ 0.018626] ACPI: TPM2 0x000000007DF76000 00004C (v04 BOCHS BXPC 00000001 BXPC 00000001) [ 0.018657] ACPI: Reserving TPM2 table memory at [mem 0x7df76000-0x7df7604b] [ 5.312318] systemd[1]: TPM2 PCR Machine ID Measurement was skipped because of an unmet condition check (ConditionPathExists=/sys/firmware/efi/efivars/StubPcrKernelImage-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f). pcsc_scan fails to read it: $ sudo pcsc_scan -s -v PC/SC device scanner V 1.6.2 (c) 2001-2022, Ludovic Rousseau <ludovic.rousseau(a)free.fr> SCardEstablishContext: RPC transport error.

2 2

Curl error (7): Couldn't connect to server for https://mirrorlist.ovirt.org/mirrorlist-ovirt-4.5-el8
by eyup ustaoglu 29 Dec '24

29 Dec '24

Hello. I keep getting this error during the ovirt PREPARE VM phase. Is there a site where I can get the current repositories under /etc/yum.repos.d? Also, have ovirt updates stopped? Why are repo addresses that have been unreachable for a long time still in the ovirt node?

1 0

Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for http://mirrorlist.centos.org/?release=8-stream&arch=x86_64&repo=virt-ovirt-45 [Could not resolve host: mirrorlist.centos.org]
by onr onr 26 Dec '24

26 Dec '24

I am in the Prepare VM step in ovirt installation. and I keep getting the above error. Why is the unused repo address defined even though I downloaded the latest ovirt-node? Where can I find repo addresses for el8 that can be accessed for installation? Is ovirt a discontinued system?

1 0

How to stop backup operation on vm
by marco.pegoraro＠univr.it 23 Dec '24

23 Dec '24

Hi, I have the following problem. In the db engine I have the following backups: engine=# select * from vm_backups; backup_id | from_checkpoint_id | to_checkpoint_id | vm_id | phase | _create_date | host_id | description | _update_date | backup_type | snapshot_id | is_stopped --------------------------------------+--------------------+--------------------------------------+------------------------------------ --+-------+----------------------------+---------+------------------------------------------------------------------------------------- -----------------------------------+----------------------------+-------------+--------------------------------------+------------ 864beee4-c87d-447c-a249-83a38fd1d4e7 | | 0ecf0a56-e7bb-4133-9ac7-441aa0d84545 | e1465242-3724-405b-9750-250d1eeb974 2 | Ready | 2024-12-18 16:42:54.717+01 | | {"type":"veeam","job_uuid":"0c3a6b0a-b46e-4ca2-8ed1-3d29c4b221ab","session_id":"4706 6400-4f1f-456e-ab94-6c3af7043809"} | 2024-12-18 16:43:06.736+01 | hybrid | 234de52d-8739-499a-8031-b8d6f6db0422 | f ace9cdf3-ad51-45b9-af6f-1728b3941187 | | e5b1ed24-4781-43dc-a1fd-b71173fd1602 | f2e2a274-71f7-4e4a-87ae-942ce0b78dd 2 | Ready | 2024-12-19 11:35:16.67+01 | | {"type":"veeam","job_uuid":"0c3a6b0a-b46e-4ca2-8ed1-3d29c4b221ab","session_id":"8d89 018d-ad81-4857-a4e3-42cf0b38a41d"} | 2024-12-19 11:35:40.907+01 | hybrid | c9108e74-8bdc-401a-b27d-adab99f28682 | f (2 rows) I want to stop the backups and delete them. I ran the following commands on the KVM nodes to stop the backups: #vdsm-client VM stop_backup vmID=e1465242-3724-405b-9750-250d1eeb9742 backup_id=864beee4-c87d-447c-a249-83a38fd1d4e7 { "code": 0, "message": "Done" } #vdsm-client VM stop_backup vmID=e1465242-3724-405b-9750-250d1eeb9742 backup_id=864beee4-c87d-447c-a249-83a38fd1d4e7 { "code": 0, "message": "Done" } but in the db there are still: engine=# select * from vm_backups; backup_id | from_checkpoint_id | to_checkpoint_id | vm_id | phase | _create_date | host_id | description | _update_date | backup_type | snapshot_id | is_stopped --------------------------------------+--------------------+--------------------------------------+------------------------------------ --+-------+----------------------------+---------+------------------------------------------------------------------------------------- -----------------------------------+----------------------------+-------------+--------------------------------------+------------ 864beee4-c87d-447c-a249-83a38fd1d4e7 | | 0ecf0a56-e7bb-4133-9ac7-441aa0d84545 | e1465242-3724-405b-9750-250d1eeb974 2 | Ready | 2024-12-18 16:42:54.717+01 | | {"type":"veeam","job_uuid":"0c3a6b0a-b46e-4ca2-8ed1-3d29c4b221ab","session_id":"4706 6400-4f1f-456e-ab94-6c3af7043809"} | 2024-12-18 16:43:06.736+01 | hybrid | 234de52d-8739-499a-8031-b8d6f6db0422 | f ace9cdf3-ad51-45b9-af6f-1728b3941187 | | e5b1ed24-4781-43dc-a1fd-b71173fd1602 | f2e2a274-71f7-4e4a-87ae-942ce0b78dd 2 | Ready | 2024-12-19 11:35:16.67+01 | | {"type":"veeam","job_uuid":"0c3a6b0a-b46e-4ca2-8ed1-3d29c4b221ab","session_id":"8d89 018d-ad81-4857-a4e3-42cf0b38a41d"} | 2024-12-19 11:35:40.907+01 | hybrid | c9108e74-8bdc-401a-b27d-adab99f28682 | f (2 rows) Do you have any idea how I can accomplish this? Thank you very much.

1 0

"Master" Storage Domain
by duparchy＠esrf.fr 22 Dec '24

22 Dec '24

Hi What is the role of the "Master Storage Domain" ? (and what consequences if it is unresponsive for a while)

2 1