9:04 a.m.
I have a small lab set up with 3 ovirt Centos 9 nodes and one Centos 9 Ovirt
"controller node" that manages.
2 problems:
It will no sync networks on hosts.
It will no launch a vm unless you go to system and choose an instance size. It will boot
and the network card has to be hot added.
I'm not sure what I'm missing,
I can give log entries showing several bad lines in diggerent modules.'
Engine log while updating hosts:
tail -n 100 -f engine.log
'
2025-01-09 02:43:15,722-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName =
kvm02
2025-01-09 02:43:15,722-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command
'HSMGetAllTasksStatusesVDSCommand(HostName = kvm02,
VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'})'
execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not
SPM: ()
2025-01-09 02:43:15,729-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START,
ConnectStoragePoolVDSCommand(HostName = kvm04,
ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='765'}), log id: 5e145d4b
2025-01-09 02:43:15,729-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Executing with
domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:43:18,057-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:43:18,057-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName =
kvm04
2025-01-09 02:43:18,057-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command
'ConnectStoragePoolVDSCommand(HostName = kvm04,
ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='765'})' execution failed: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:18,057-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH,
ConnectStoragePoolVDSCommand, return: , log id: 5e145d4b
2025-01-09 02:43:18,057-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:22,344-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command
'SpmStatusVDSCommand(HostName = kvm04,
SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed:
IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating
master storage domain: ('MD read error',)
2025-01-09 02:43:22,344-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Irs placed on
server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover
2025-01-09 02:43:22,349-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START,
ConnectStoragePoolVDSCommand(HostName = kvm01,
ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd',
vdsId='acf66bf6-e159-4705-be47-484c99e746dd',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='765'}), log id: 28df8c85
2025-01-09 02:43:22,349-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Executing with
domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:43:25,267-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm01 command ConnectStoragePoolVDS failed:
Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:25,267-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:43:25,267-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName =
kvm01
2025-01-09 02:43:25,267-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command
'ConnectStoragePoolVDSCommand(HostName = kvm01,
ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd',
vdsId='acf66bf6-e159-4705-be47-484c99e746dd',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='765'})' execution failed: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:25,267-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH,
ConnectStoragePoolVDSCommand, return: , log id: 28df8c85
2025-01-09 02:43:25,267-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:25,275-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Irs placed on
server 'acf66bf6-e159-4705-be47-484c99e746dd' failed. Proceed Failover
2025-01-09 02:43:25,279-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START,
ConnectStoragePoolVDSCommand(HostName = kvm02,
ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='765'}), log id: 20ec709f
2025-01-09 02:43:25,279-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Executing with
domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:43:27,675-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm02 command ConnectStoragePoolVDS failed:
Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:27,675-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:43:27,675-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName =
kvm02
2025-01-09 02:43:27,676-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command
'ConnectStoragePoolVDSCommand(HostName = kvm02,
ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='765'})' execution failed: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:27,676-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH,
ConnectStoragePoolVDSCommand, return: , log id: 20ec709f
2025-01-09 02:43:27,676-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:27,741-05 INFO
[org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-572) [6727848e] Running command:
ReconstructMasterDomainCommand internal: true. Entities affected : ID:
bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage
2025-01-09 02:43:27,765-05 WARN
[org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-572) [6727848e] skipping connect and refresh for
all hosts, last master 'true', command status 'true'
2025-01-09 02:43:27,789-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor]
(EE-ManagedThreadFactory-engine-Thread-571) [] Finished reconstruct for pool
'39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue
2025-01-09 02:43:30,154-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] START,
GlusterServersListVDSCommand(HostName = kvm04,
VdsIdVDSCommandParametersBase:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}),
log id: 770c0b37
2025-01-09 02:43:30,362-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] FINISH,
GlusterServersListVDSCommand, return: [192.168.254.108/24:CONNECTED,
ovirt.digitaldatatechs.com:CONNECTED, kvm02.digitaldatatechs.com:CONNECTED,
kvm01.digitaldatatechs.com:CONNECTED], log id: 770c0b37
2025-01-09 02:43:30,381-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] START,
AddGlusterServerVDSCommand(HostName = kvm04,
AddGlusterServerVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}),
log id: 3b2ce207
2025-01-09 02:43:33,630-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] FINISH,
AddGlusterServerVDSCommand, return: , log id: 3b2ce207
2025-01-09 02:43:33,640-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] EVENT_ID:
GLUSTER_SERVER_ADD_FAILED(4,436), Failed to add host kvm04 into Cluster dds_cluster. Add
host failed: rc=107 out=() err=['Probe returned with Transport endpoint is not
connected']
2025-01-09 02:43:33,641-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] START,
GlusterVolumesListVDSCommand(HostName = kvm04,
GlusterVolumesListVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}),
log id: 6a99b1ab
2025-01-09 02:43:33,748-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] FINISH,
GlusterVolumesListVDSCommand, return:
{b79a00d7-cc2a-48d8-b881-699991f3d930=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@205714d2},
log id: 6a99b1ab
2025-01-09 02:43:37,689-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command
'SpmStatusVDSCommand(HostName = kvm02,
SpmStatusVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed:
IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating
master storage domain: ('MD read error',)
2025-01-09 02:43:37,691-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand'
return value '
TaskStatusListReturn:{status='Status [code=654, message=Not SPM: ()]'}
'
2025-01-09 02:43:37,691-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName =
kvm02
2025-01-09 02:43:37,691-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command
'HSMGetAllTasksStatusesVDSCommand(HostName = kvm02,
VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'})'
execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not
SPM: ()
2025-01-09 02:43:37,698-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] START,
ConnectStoragePoolVDSCommand(HostName = kvm01,
ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd',
vdsId='acf66bf6-e159-4705-be47-484c99e746dd',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='766'}), log id: 201774c9
2025-01-09 02:43:37,698-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Executing with
domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:43:38,381-05 INFO
[org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-10) [] There is no
host with more than 10 running guests, no balancing is needed
2025-01-09 02:43:38,400-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-59) [] START,
GlusterTasksListVDSCommand(HostName = kvm02,
VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'}),
log id: 702ab66f
2025-01-09 02:43:38,515-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-59) [] FINISH,
GlusterTasksListVDSCommand, return: [], log id: 702ab66f
2025-01-09 02:43:40,598-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:43:40,598-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName =
kvm01
2025-01-09 02:43:40,598-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command
'ConnectStoragePoolVDSCommand(HostName = kvm01,
ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd',
vdsId='acf66bf6-e159-4705-be47-484c99e746dd',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='766'})' execution failed: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:40,598-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] FINISH,
ConnectStoragePoolVDSCommand, return: , log id: 201774c9
2025-01-09 02:43:40,598-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:40,606-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Irs placed on
server 'acf66bf6-e159-4705-be47-484c99e746dd' failed. Proceed Failover
2025-01-09 02:43:40,611-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] START,
ConnectStoragePoolVDSCommand(HostName = kvm04,
ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='766'}), log id: 164d36cb
2025-01-09 02:43:40,611-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Executing with
domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:43:42,950-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm04 command ConnectStoragePoolVDS failed:
Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:42,950-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:43:42,950-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName =
kvm04
2025-01-09 02:43:42,950-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command
'ConnectStoragePoolVDSCommand(HostName = kvm04,
ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='766'})' execution failed: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:42,950-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] FINISH,
ConnectStoragePoolVDSCommand, return: , log id: 164d36cb
2025-01-09 02:43:42,950-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:42,960-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command
'SpmStatusVDSCommand(HostName = kvm04,
SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed:
IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating
master storage domain: ('MD read error',)
2025-01-09 02:43:42,960-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Irs placed on
server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover
2025-01-09 02:43:42,964-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] START,
ConnectStoragePoolVDSCommand(HostName = kvm02,
ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='766'}), log id: 863c329
2025-01-09 02:43:42,965-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Executing with
domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:43:45,328-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:43:45,328-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName =
kvm02
2025-01-09 02:43:45,328-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command
'ConnectStoragePoolVDSCommand(HostName = kvm02,
ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='766'})' execution failed: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:45,328-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] FINISH,
ConnectStoragePoolVDSCommand, return: , log id: 863c329
2025-01-09 02:43:45,328-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:45,388-05 INFO
[org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-580) [2e23e34] Running command:
ReconstructMasterDomainCommand internal: true. Entities affected : ID:
bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage
2025-01-09 02:43:45,412-05 WARN
[org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-580) [2e23e34] skipping connect and refresh for all
hosts, last master 'true', command status 'true'
2025-01-09 02:43:45,436-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor]
(EE-ManagedThreadFactory-engine-Thread-579) [] Finished reconstruct for pool
'39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue
2025-01-09 02:43:48,763-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] START,
GlusterServersListVDSCommand(HostName = kvm04,
VdsIdVDSCommandParametersBase:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}),
log id: db1640e
2025-01-09 02:43:48,970-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] FINISH,
GlusterServersListVDSCommand, return: [192.168.254.108/24:CONNECTED,
ovirt.digitaldatatechs.com:CONNECTED, kvm02.digitaldatatechs.com:CONNECTED,
kvm01.digitaldatatechs.com:CONNECTED], log id: db1640e
2025-01-09 02:43:48,983-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] START,
AddGlusterServerVDSCommand(HostName = kvm04,
AddGlusterServerVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}),
log id: 717f84ae
2025-01-09 02:43:52,255-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] FINISH,
AddGlusterServerVDSCommand, return: , log id: 717f84ae
2025-01-09 02:43:52,277-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] EVENT_ID:
GLUSTER_SERVER_ADD_FAILED(4,436), Failed to add host kvm04 into Cluster dds_cluster. Add
host failed: rc=107 out=() err=['Probe returned with Transport endpoint is not
connected']
2025-01-09 02:43:52,278-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] START,
GlusterVolumesListVDSCommand(HostName = kvm04,
GlusterVolumesListVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}),
log id: 6a43f9e6
2025-01-09 02:43:52,384-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] FINISH,
GlusterVolumesListVDSCommand, return:
{b79a00d7-cc2a-48d8-b881-699991f3d930=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@205714d2},
log id: 6a43f9e6
2025-01-09 02:43:55,341-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command
'SpmStatusVDSCommand(HostName = kvm02,
SpmStatusVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed:
IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating
master storage domain: ('MD read error',)
2025-01-09 02:43:55,343-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand'
return value '
TaskStatusListReturn:{status='Status [code=654, message=Not SPM: ()]'}
'
2025-01-09 02:43:55,343-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName =
kvm02
2025-01-09 02:43:55,343-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command
'HSMGetAllTasksStatusesVDSCommand(HostName = kvm02,
VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'})'
execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not
SPM: ()
2025-01-09 02:43:55,350-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] START,
ConnectStoragePoolVDSCommand(HostName = kvm04,
ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='767'}), log id: 35cf220b
2025-01-09 02:43:55,350-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Executing with
domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:43:57,661-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:43:57,661-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName =
kvm04
2025-01-09 02:43:57,661-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command
'ConnectStoragePoolVDSCommand(HostName = kvm04,
ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='767'})' execution failed: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:57,661-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] FINISH,
ConnectStoragePoolVDSCommand, return: , log id: 35cf220b
2025-01-09 02:43:57,661-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:43:57,671-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command
'SpmStatusVDSCommand(HostName = kvm04,
SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed:
IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating
master storage domain: ('MD read error',)
2025-01-09 02:43:57,671-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Irs placed on
server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover
2025-01-09 02:43:57,675-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] START,
ConnectStoragePoolVDSCommand(HostName = kvm02,
ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='767'}), log id: 2ff24c19
2025-01-09 02:43:57,676-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Executing with
domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:44:00,065-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm02 command ConnectStoragePoolVDS failed:
Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:00,065-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:44:00,065-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName =
kvm02
2025-01-09 02:44:00,065-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command
'ConnectStoragePoolVDSCommand(HostName = kvm02,
ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='767'})' execution failed: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:00,065-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] FINISH,
ConnectStoragePoolVDSCommand, return: , log id: 2ff24c19
2025-01-09 02:44:00,065-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:04,367-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command
'SpmStatusVDSCommand(HostName = kvm02,
SpmStatusVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed:
IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating
master storage domain: ('MD read error',)
2025-01-09 02:44:04,367-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Irs placed on
server 'cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6' failed. Proceed Failover
2025-01-09 02:44:04,372-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] START,
ConnectStoragePoolVDSCommand(HostName = kvm01,
ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd',
vdsId='acf66bf6-e159-4705-be47-484c99e746dd',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='767'}), log id: 23ed7788
2025-01-09 02:44:04,372-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Executing with
domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:44:07,292-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm01 command ConnectStoragePoolVDS failed:
Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:07,292-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:44:07,292-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName =
kvm01
2025-01-09 02:44:07,292-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command
'ConnectStoragePoolVDSCommand(HostName = kvm01,
ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd',
vdsId='acf66bf6-e159-4705-be47-484c99e746dd',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='767'})' execution failed: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:07,292-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] FINISH,
ConnectStoragePoolVDSCommand, return: , log id: 23ed7788
2025-01-09 02:44:07,292-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:07,375-05 INFO
[org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-594) [6e7f60bb] Running command:
ReconstructMasterDomainCommand internal: true. Entities affected : ID:
bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage
2025-01-09 02:44:07,412-05 INFO
[org.ovirt.engine.core.bll.network.host.SyncAllHostNetworksCommand] (default task-7)
[9913b205-9b2e-4f8a-86e3-048a9cdff2ba] Running command: SyncAllHostNetworksCommand
internal: false. Entities affected : ID: acf66bf6-e159-4705-be47-484c99e746dd Type:
VDSAction group CONFIGURE_HOST_NETWORK with role type ADMIN
2025-01-09 02:44:07,421-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START,
GlusterServersListVDSCommand(HostName = kvm04,
VdsIdVDSCommandParametersBase:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}),
log id: 340c8f9d
2025-01-09 02:44:07,423-05 WARN
[org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-594) [6e7f60bb] skipping connect and refresh for
all hosts, last master 'true', command status 'true'
2025-01-09 02:44:07,423-05 INFO
[org.ovirt.engine.core.bll.network.host.PersistentHostSetupNetworksCommand] (default
task-7) [7814c7ae] Running command: PersistentHostSetupNetworksCommand internal: true.
Entities affected : ID: acf66bf6-e159-4705-be47-484c99e746dd Type: VDS
2025-01-09 02:44:07,448-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor]
(EE-ManagedThreadFactory-engine-Thread-593) [] Finished reconstruct for pool
'39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue
2025-01-09 02:44:07,460-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7)
[7814c7ae] EVENT_ID: PERSIST_HOST_SETUP_NETWORK_ON_HOST(10,902), (1/1): Applying
network's changes on host kvm01. (User: admin@ovirt@internalkeycloak-authz)
2025-01-09 02:44:07,461-05 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7)
[7814c7ae] Before acquiring lock-timeout
'EngineLock:{exclusiveLocks='[HOST_NETWORKacf66bf6-e159-4705-be47-484c99e746dd=HOST_NETWORK]',
sharedLocks=''}'
2025-01-09 02:44:07,461-05 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7)
[7814c7ae] Lock-timeout acquired to object
'EngineLock:{exclusiveLocks='[HOST_NETWORKacf66bf6-e159-4705-be47-484c99e746dd=HOST_NETWORK]',
sharedLocks=''}'
2025-01-09 02:44:07,497-05 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7)
[7814c7ae] Running command: HostSetupNetworksCommand internal: true. Entities affected :
ID: acf66bf6-e159-4705-be47-484c99e746dd Type: VDSAction group CONFIGURE_HOST_NETWORK with
role type ADMIN
2025-01-09 02:44:07,497-05 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7)
[7814c7ae] Before acquiring lock in order to prevent monitoring for host 'kvm01'
from data-center 'DDS_Data_Center'
2025-01-09 02:44:07,497-05 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7)
[7814c7ae] Lock acquired, from now a monitoring of host will be skipped for host
'kvm01' from data-center 'DDS_Data_Center'
2025-01-09 02:44:07,498-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-7)
[7814c7ae] START, HostSetupNetworksVDSCommand(HostName = kvm01,
HostSetupNetworksVdsCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd',
vds='Host[kvm01,acf66bf6-e159-4705-be47-484c99e746dd]',
rollbackOnFailure='true', commitOnSuccess='true',
connectivityTimeout='120',
networks='[HostNetwork:{defaultRoute='false', bonding='false',
networkName='vlan3', vdsmName='vlan3', nicName='eno2',
vlan='3', vmNetwork='true', stp='false', properties='[]',
ipv4BootProtocol='STATIC_IP', ipv4Address='172.50.26.252',
ipv4Netmask='255.255.255.0', ipv4Gateway='172.50.26.1',
ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null',
ipv6Gateway='null', nameServers='null'},
HostNetwork:{defaultRoute='true', bonding='false',
networkName='ovirtmgmt', vdsmName='ovirtmgmt', nicName='eno1',
vlan='null', vmNetwork='true', stp='false',
properties='[]', ipv4BootProtocol='STAT
IC_IP', ipv4Address='192.168.254.252', ipv4Netmask='255.255.255.0',
ipv4Gateway='192.168.254.1', ipv6BootProtocol='NONE',
ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null',
nameServers='[NameServer:{address='192.168.254.222'},
NameServer:{address='192.168.254.232'}]'}]', removedNetworks='[]',
bonds='[]', removedBonds='[]', clusterSwitchType='LEGACY',
managementNetworkChanged='true'}), log id: 56a27c18
2025-01-09 02:44:07,499-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-7)
[7814c7ae] FINISH, HostSetupNetworksVDSCommand, return: , log id: 56a27c18
2025-01-09 02:44:07,631-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH,
GlusterServersListVDSCommand, return: [192.168.254.108/24:CONNECTED,
ovirt.digitaldatatechs.com:CONNECTED, kvm02.digitaldatatechs.com:CONNECTED,
kvm01.digitaldatatechs.com:CONNECTED], log id: 340c8f9d
2025-01-09 02:44:07,642-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START,
AddGlusterServerVDSCommand(HostName = kvm04,
AddGlusterServerVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}),
log id: 7c25cce3
2025-01-09 02:44:10,879-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH,
AddGlusterServerVDSCommand, return: , log id: 7c25cce3
2025-01-09 02:44:10,923-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] EVENT_ID:
GLUSTER_SERVER_ADD_FAILED(4,436), Failed to add host kvm04 into Cluster dds_cluster. Add
host failed: rc=107 out=() err=['Probe returned with Transport endpoint is not
connected']
2025-01-09 02:44:10,923-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START,
GlusterVolumesListVDSCommand(HostName = kvm04,
GlusterVolumesListVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}),
log id: effd1e8
2025-01-09 02:44:11,033-05 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH,
GlusterVolumesListVDSCommand, return:
{b79a00d7-cc2a-48d8-b881-699991f3d930=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@205714d2},
log id: effd1e8
2025-01-09 02:44:17,314-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand'
return value '
TaskStatusListReturn:{status='Status [code=654, message=Not SPM: ()]'}
'
2025-01-09 02:44:17,315-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName =
kvm01
2025-01-09 02:44:17,315-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command
'HSMGetAllTasksStatusesVDSCommand(HostName = kvm01,
VdsIdVDSCommandParametersBase:{hostId='acf66bf6-e159-4705-be47-484c99e746dd'})'
execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not
SPM: ()
2025-01-09 02:44:17,331-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] START,
ConnectStoragePoolVDSCommand(HostName = kvm04,
ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='768'}), log id: bcbf59e
2025-01-09 02:44:17,331-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Executing with
domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:44:18,272-05 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7)
[7814c7ae] EVENT_ID: VDS_NETWORKS_OUT_OF_SYNC(1,110), Host kvm01's following
network(s) are not synchronized with their Logical Network configuration:
ovirtmgmt,vlan3.
2025-01-09 02:44:18,283-05 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7)
[7814c7ae] Host setup networks finished. Lock released. Monitoring can run now for host
'kvm01' from data-center 'DDS_Data_Center'
2025-01-09 02:44:18,285-05 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7)
[7814c7ae] Lock freed to object
'EngineLock:{exclusiveLocks='[HOST_NETWORKacf66bf6-e159-4705-be47-484c99e746dd=HOST_NETWORK]',
sharedLocks=''}'
2025-01-09 02:44:18,305-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7)
[7814c7ae] EVENT_ID: PERSIST_SETUP_NETWORK_ON_HOST_FINISHED(10,903), (1/1): Successfully
applied changes on host kvm01. (User: admin@ovirt@internalkeycloak-authz)
2025-01-09 02:44:18,316-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7)
[7814c7ae] EVENT_ID: HOST_SYNC_ALL_NETWORKS_FINISHED(10,901), Managed to sync all host
kvm01 networks.
2025-01-09 02:44:19,668-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm04 command ConnectStoragePoolVDS failed:
Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:19,668-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:44:19,668-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName =
kvm04
2025-01-09 02:44:19,668-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command
'ConnectStoragePoolVDSCommand(HostName = kvm04,
ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='768'})' execution failed: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:19,668-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] FINISH,
ConnectStoragePoolVDSCommand, return: , log id: bcbf59e
2025-01-09 02:44:19,668-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:19,679-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command
'SpmStatusVDSCommand(HostName = kvm04,
SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed:
IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating
master storage domain: ('MD read error',)
2025-01-09 02:44:19,679-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Irs placed on
server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover
2025-01-09 02:44:19,684-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] START,
ConnectStoragePoolVDSCommand(HostName = kvm01,
ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd',
vdsId='acf66bf6-e159-4705-be47-484c99e746dd',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='768'}), log id: 251dbd22
2025-01-09 02:44:19,684-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Executing with
domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:44:22,181-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:44:22,181-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName =
kvm01
2025-01-09 02:44:22,181-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command
'ConnectStoragePoolVDSCommand(HostName = kvm01,
ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd',
vdsId='acf66bf6-e159-4705-be47-484c99e746dd',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='768'})' execution failed: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:22,181-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] FINISH,
ConnectStoragePoolVDSCommand, return: , log id: 251dbd22
2025-01-09 02:44:22,181-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:22,189-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Irs placed on
server 'acf66bf6-e159-4705-be47-484c99e746dd' failed. Proceed Failover
2025-01-09 02:44:22,198-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] START,
ConnectStoragePoolVDSCommand(HostName = kvm02,
ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='768'}), log id: 1f60ebb1
2025-01-09 02:44:22,198-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Executing with
domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active,
ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active}
2025-01-09 02:44:24,581-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return
value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]'
2025-01-09 02:44:24,581-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName =
kvm02
2025-01-09 02:44:24,581-05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command
'ConnectStoragePoolVDSCommand(HostName = kvm02,
ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6',
storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3',
masterVersion='768'})' execution failed: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:24,582-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] FINISH,
ConnectStoragePoolVDSCommand, return: , log id: 1f60ebb1
2025-01-09 02:44:24,582-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3,
msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94'
2025-01-09 02:44:24,658-05 INFO
[org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-610) [2af75e62] Running command:
ReconstructMasterDomainCommand internal: true. Entities affected : ID:
bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage
2025-01-09 02:44:24,691-05 WARN
[org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-610) [2af75e62] skipping connect and refresh for
all hosts, last master 'true', command status 'true'
2025-01-09 02:44:24,715-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor]
(EE-ManagedThreadFactory-engine-Thread-609) [] Finished reconstruct for pool
'39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue
While attempting to start vm will be sent after but I think the net allocation issue with
hosts is the center of the vm issues,,