I have a small lab set up with 3 ovirt Centos 9 nodes and one Centos 9 Ovirt "controller node" that manages. 2 problems: It will no sync networks on hosts. It will no launch a vm unless you go to system and choose an instance size. It will boot and the network card has to be hot added. I'm not sure what I'm missing, I can give log entries showing several bad lines in diggerent modules.' Engine log while updating hosts: tail -n 100 -f engine.log ' 2025-01-09 02:43:15,722-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName = kvm02 2025-01-09 02:43:15,722-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm02, VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2025-01-09 02:43:15,729-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'}), log id: 5e145d4b 2025-01-09 02:43:15,729-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:18,057-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:18,057-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName = kvm04 2025-01-09 02:43:18,057-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:18,057-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 5e145d4b 2025-01-09 02:43:18,057-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:22,344-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'SpmStatusVDSCommand(HostName = kvm04, SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:22,344-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Irs placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover 2025-01-09 02:43:22,349-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'}), log id: 28df8c85 2025-01-09 02:43:22,349-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:25,267-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm01 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:25,267-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:25,267-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName = kvm01 2025-01-09 02:43:25,267-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:25,267-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 28df8c85 2025-01-09 02:43:25,267-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:25,275-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Irs placed on server 'acf66bf6-e159-4705-be47-484c99e746dd' failed. Proceed Failover 2025-01-09 02:43:25,279-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'}), log id: 20ec709f 2025-01-09 02:43:25,279-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:27,675-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm02 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:27,675-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:27,675-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName = kvm02 2025-01-09 02:43:27,676-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:27,676-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 20ec709f 2025-01-09 02:43:27,676-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:27,741-05 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-572) [6727848e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage 2025-01-09 02:43:27,765-05 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-572) [6727848e] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2025-01-09 02:43:27,789-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (EE-ManagedThreadFactory-engine-Thread-571) [] Finished reconstruct for pool '39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue 2025-01-09 02:43:30,154-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] START, GlusterServersListVDSCommand(HostName = kvm04, VdsIdVDSCommandParametersBase:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 770c0b37 2025-01-09 02:43:30,362-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] FINISH, GlusterServersListVDSCommand, return: [192.168.254.108/24:CONNECTED, ovirt.digitaldatatechs.com:CONNECTED, kvm02.digitaldatatechs.com:CONNECTED, kvm01.digitaldatatechs.com:CONNECTED], log id: 770c0b37 2025-01-09 02:43:30,381-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] START, AddGlusterServerVDSCommand(HostName = kvm04, AddGlusterServerVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 3b2ce207 2025-01-09 02:43:33,630-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] FINISH, AddGlusterServerVDSCommand, return: , log id: 3b2ce207 2025-01-09 02:43:33,640-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] EVENT_ID: GLUSTER_SERVER_ADD_FAILED(4,436), Failed to add host kvm04 into Cluster dds_cluster. Add host failed: rc=107 out=() err=['Probe returned with Transport endpoint is not connected'] 2025-01-09 02:43:33,641-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] START, GlusterVolumesListVDSCommand(HostName = kvm04, GlusterVolumesListVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 6a99b1ab 2025-01-09 02:43:33,748-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] FINISH, GlusterVolumesListVDSCommand, return: {b79a00d7-cc2a-48d8-b881-699991f3d930=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@205714d2}, log id: 6a99b1ab 2025-01-09 02:43:37,689-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'SpmStatusVDSCommand(HostName = kvm02, SpmStatusVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:37,691-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' TaskStatusListReturn:{status='Status [code=654, message=Not SPM: ()]'} ' 2025-01-09 02:43:37,691-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName = kvm02 2025-01-09 02:43:37,691-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm02, VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2025-01-09 02:43:37,698-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'}), log id: 201774c9 2025-01-09 02:43:37,698-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:38,381-05 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-10) [] There is no host with more than 10 running guests, no balancing is needed 2025-01-09 02:43:38,400-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-59) [] START, GlusterTasksListVDSCommand(HostName = kvm02, VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'}), log id: 702ab66f 2025-01-09 02:43:38,515-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-59) [] FINISH, GlusterTasksListVDSCommand, return: [], log id: 702ab66f 2025-01-09 02:43:40,598-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:40,598-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName = kvm01 2025-01-09 02:43:40,598-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:40,598-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 201774c9 2025-01-09 02:43:40,598-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:40,606-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Irs placed on server 'acf66bf6-e159-4705-be47-484c99e746dd' failed. Proceed Failover 2025-01-09 02:43:40,611-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'}), log id: 164d36cb 2025-01-09 02:43:40,611-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:42,950-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm04 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:42,950-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:42,950-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName = kvm04 2025-01-09 02:43:42,950-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:42,950-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 164d36cb 2025-01-09 02:43:42,950-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:42,960-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'SpmStatusVDSCommand(HostName = kvm04, SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:42,960-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Irs placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover 2025-01-09 02:43:42,964-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'}), log id: 863c329 2025-01-09 02:43:42,965-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:45,328-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:45,328-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName = kvm02 2025-01-09 02:43:45,328-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:45,328-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 863c329 2025-01-09 02:43:45,328-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:45,388-05 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-580) [2e23e34] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage 2025-01-09 02:43:45,412-05 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-580) [2e23e34] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2025-01-09 02:43:45,436-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (EE-ManagedThreadFactory-engine-Thread-579) [] Finished reconstruct for pool '39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue 2025-01-09 02:43:48,763-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] START, GlusterServersListVDSCommand(HostName = kvm04, VdsIdVDSCommandParametersBase:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: db1640e 2025-01-09 02:43:48,970-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] FINISH, GlusterServersListVDSCommand, return: [192.168.254.108/24:CONNECTED, ovirt.digitaldatatechs.com:CONNECTED, kvm02.digitaldatatechs.com:CONNECTED, kvm01.digitaldatatechs.com:CONNECTED], log id: db1640e 2025-01-09 02:43:48,983-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] START, AddGlusterServerVDSCommand(HostName = kvm04, AddGlusterServerVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 717f84ae 2025-01-09 02:43:52,255-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] FINISH, AddGlusterServerVDSCommand, return: , log id: 717f84ae 2025-01-09 02:43:52,277-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] EVENT_ID: GLUSTER_SERVER_ADD_FAILED(4,436), Failed to add host kvm04 into Cluster dds_cluster. Add host failed: rc=107 out=() err=['Probe returned with Transport endpoint is not connected'] 2025-01-09 02:43:52,278-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] START, GlusterVolumesListVDSCommand(HostName = kvm04, GlusterVolumesListVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 6a43f9e6 2025-01-09 02:43:52,384-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] FINISH, GlusterVolumesListVDSCommand, return: {b79a00d7-cc2a-48d8-b881-699991f3d930=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@205714d2}, log id: 6a43f9e6 2025-01-09 02:43:55,341-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'SpmStatusVDSCommand(HostName = kvm02, SpmStatusVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:55,343-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' TaskStatusListReturn:{status='Status [code=654, message=Not SPM: ()]'} ' 2025-01-09 02:43:55,343-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName = kvm02 2025-01-09 02:43:55,343-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm02, VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2025-01-09 02:43:55,350-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'}), log id: 35cf220b 2025-01-09 02:43:55,350-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:57,661-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:57,661-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName = kvm04 2025-01-09 02:43:57,661-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:57,661-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 35cf220b 2025-01-09 02:43:57,661-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:57,671-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'SpmStatusVDSCommand(HostName = kvm04, SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:57,671-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Irs placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover 2025-01-09 02:43:57,675-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'}), log id: 2ff24c19 2025-01-09 02:43:57,676-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:00,065-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm02 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:00,065-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:00,065-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName = kvm02 2025-01-09 02:44:00,065-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:00,065-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 2ff24c19 2025-01-09 02:44:00,065-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:04,367-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'SpmStatusVDSCommand(HostName = kvm02, SpmStatusVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:44:04,367-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Irs placed on server 'cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6' failed. Proceed Failover 2025-01-09 02:44:04,372-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'}), log id: 23ed7788 2025-01-09 02:44:04,372-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:07,292-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm01 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:07,292-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:07,292-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName = kvm01 2025-01-09 02:44:07,292-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:07,292-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 23ed7788 2025-01-09 02:44:07,292-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:07,375-05 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-594) [6e7f60bb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage 2025-01-09 02:44:07,412-05 INFO [org.ovirt.engine.core.bll.network.host.SyncAllHostNetworksCommand] (default task-7) [9913b205-9b2e-4f8a-86e3-048a9cdff2ba] Running command: SyncAllHostNetworksCommand internal: false. Entities affected : ID: acf66bf6-e159-4705-be47-484c99e746dd Type: VDSAction group CONFIGURE_HOST_NETWORK with role type ADMIN 2025-01-09 02:44:07,421-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, GlusterServersListVDSCommand(HostName = kvm04, VdsIdVDSCommandParametersBase:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 340c8f9d 2025-01-09 02:44:07,423-05 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-594) [6e7f60bb] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2025-01-09 02:44:07,423-05 INFO [org.ovirt.engine.core.bll.network.host.PersistentHostSetupNetworksCommand] (default task-7) [7814c7ae] Running command: PersistentHostSetupNetworksCommand internal: true. Entities affected : ID: acf66bf6-e159-4705-be47-484c99e746dd Type: VDS 2025-01-09 02:44:07,448-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (EE-ManagedThreadFactory-engine-Thread-593) [] Finished reconstruct for pool '39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue 2025-01-09 02:44:07,460-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [7814c7ae] EVENT_ID: PERSIST_HOST_SETUP_NETWORK_ON_HOST(10,902), (1/1): Applying network's changes on host kvm01. (User: admin@ovirt@internalkeycloak-authz) 2025-01-09 02:44:07,461-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Before acquiring lock-timeout 'EngineLock:{exclusiveLocks='[HOST_NETWORKacf66bf6-e159-4705-be47-484c99e746dd=HOST_NETWORK]', sharedLocks=''}' 2025-01-09 02:44:07,461-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Lock-timeout acquired to object 'EngineLock:{exclusiveLocks='[HOST_NETWORKacf66bf6-e159-4705-be47-484c99e746dd=HOST_NETWORK]', sharedLocks=''}' 2025-01-09 02:44:07,497-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Running command: HostSetupNetworksCommand internal: true. Entities affected : ID: acf66bf6-e159-4705-be47-484c99e746dd Type: VDSAction group CONFIGURE_HOST_NETWORK with role type ADMIN 2025-01-09 02:44:07,497-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Before acquiring lock in order to prevent monitoring for host 'kvm01' from data-center 'DDS_Data_Center' 2025-01-09 02:44:07,497-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Lock acquired, from now a monitoring of host will be skipped for host 'kvm01' from data-center 'DDS_Data_Center' 2025-01-09 02:44:07,498-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-7) [7814c7ae] START, HostSetupNetworksVDSCommand(HostName = kvm01, HostSetupNetworksVdsCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vds='Host[kvm01,acf66bf6-e159-4705-be47-484c99e746dd]', rollbackOnFailure='true', commitOnSuccess='true', connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='false', bonding='false', networkName='vlan3', vdsmName='vlan3', nicName='eno2', vlan='3', vmNetwork='true', stp='false', properties='[]', ipv4BootProtocol='STATIC_IP', ipv4Address='172.50.26.252', ipv4Netmask='255.255.255.0', ipv4Gateway='172.50.26.1', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}, HostNetwork:{defaultRoute='true', bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt', nicName='eno1', vlan='null', vmNetwork='true', stp='false', properties='[]', ipv4BootProtocol='STAT IC_IP', ipv4Address='192.168.254.252', ipv4Netmask='255.255.255.0', ipv4Gateway='192.168.254.1', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='[NameServer:{address='192.168.254.222'}, NameServer:{address='192.168.254.232'}]'}]', removedNetworks='[]', bonds='[]', removedBonds='[]', clusterSwitchType='LEGACY', managementNetworkChanged='true'}), log id: 56a27c18 2025-01-09 02:44:07,499-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-7) [7814c7ae] FINISH, HostSetupNetworksVDSCommand, return: , log id: 56a27c18 2025-01-09 02:44:07,631-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, GlusterServersListVDSCommand, return: [192.168.254.108/24:CONNECTED, ovirt.digitaldatatechs.com:CONNECTED, kvm02.digitaldatatechs.com:CONNECTED, kvm01.digitaldatatechs.com:CONNECTED], log id: 340c8f9d 2025-01-09 02:44:07,642-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, AddGlusterServerVDSCommand(HostName = kvm04, AddGlusterServerVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 7c25cce3 2025-01-09 02:44:10,879-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, AddGlusterServerVDSCommand, return: , log id: 7c25cce3 2025-01-09 02:44:10,923-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] EVENT_ID: GLUSTER_SERVER_ADD_FAILED(4,436), Failed to add host kvm04 into Cluster dds_cluster. Add host failed: rc=107 out=() err=['Probe returned with Transport endpoint is not connected'] 2025-01-09 02:44:10,923-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, GlusterVolumesListVDSCommand(HostName = kvm04, GlusterVolumesListVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: effd1e8 2025-01-09 02:44:11,033-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, GlusterVolumesListVDSCommand, return: {b79a00d7-cc2a-48d8-b881-699991f3d930=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@205714d2}, log id: effd1e8 2025-01-09 02:44:17,314-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' TaskStatusListReturn:{status='Status [code=654, message=Not SPM: ()]'} ' 2025-01-09 02:44:17,315-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName = kvm01 2025-01-09 02:44:17,315-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm01, VdsIdVDSCommandParametersBase:{hostId='acf66bf6-e159-4705-be47-484c99e746dd'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2025-01-09 02:44:17,331-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] START, ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'}), log id: bcbf59e 2025-01-09 02:44:17,331-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:18,272-05 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [7814c7ae] EVENT_ID: VDS_NETWORKS_OUT_OF_SYNC(1,110), Host kvm01's following network(s) are not synchronized with their Logical Network configuration: ovirtmgmt,vlan3. 2025-01-09 02:44:18,283-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Host setup networks finished. Lock released. Monitoring can run now for host 'kvm01' from data-center 'DDS_Data_Center' 2025-01-09 02:44:18,285-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Lock freed to object 'EngineLock:{exclusiveLocks='[HOST_NETWORKacf66bf6-e159-4705-be47-484c99e746dd=HOST_NETWORK]', sharedLocks=''}' 2025-01-09 02:44:18,305-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [7814c7ae] EVENT_ID: PERSIST_SETUP_NETWORK_ON_HOST_FINISHED(10,903), (1/1): Successfully applied changes on host kvm01. (User: admin@ovirt@internalkeycloak-authz) 2025-01-09 02:44:18,316-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [7814c7ae] EVENT_ID: HOST_SYNC_ALL_NETWORKS_FINISHED(10,901), Managed to sync all host kvm01 networks. 2025-01-09 02:44:19,668-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm04 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:19,668-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:19,668-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName = kvm04 2025-01-09 02:44:19,668-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:19,668-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: bcbf59e 2025-01-09 02:44:19,668-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:19,679-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'SpmStatusVDSCommand(HostName = kvm04, SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:44:19,679-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Irs placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover 2025-01-09 02:44:19,684-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] START, ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'}), log id: 251dbd22 2025-01-09 02:44:19,684-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:22,181-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:22,181-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName = kvm01 2025-01-09 02:44:22,181-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:22,181-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 251dbd22 2025-01-09 02:44:22,181-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:22,189-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Irs placed on server 'acf66bf6-e159-4705-be47-484c99e746dd' failed. Proceed Failover 2025-01-09 02:44:22,198-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] START, ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'}), log id: 1f60ebb1 2025-01-09 02:44:22,198-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:24,581-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:24,581-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName = kvm02 2025-01-09 02:44:24,581-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:24,582-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 1f60ebb1 2025-01-09 02:44:24,582-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:24,658-05 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-610) [2af75e62] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage 2025-01-09 02:44:24,691-05 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-610) [2af75e62] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2025-01-09 02:44:24,715-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (EE-ManagedThreadFactory-engine-Thread-609) [] Finished reconstruct for pool '39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue While attempting to start vm will be sent after but I think the net allocation issue with hosts is the center of the vm issues,,