- Users - oVirt List Archives

Changing Disk Alocation Policy
by suporte＠logicworks.pt 13 Jan '25

13 Jan '25

Hi, It's possible to change a thin-provision disk to a preallocated disk? Storages are ISCSI Thanks -- Jose Ferradeira http://www.logicworks.pt

4 4

Re: [External] : Changing Disk Alocation Policy
by change_jeeringly679＠dralias.com 13 Jan '25

13 Jan '25

Hey, Jose is right, you can do this via the api. The server needs to be shutdown. I usually use the below curl command on the engine itself (login with ssh). You need 3 pieces of information: - You need the ce.pem of the engine (which you can download from the engine login page) and make it available. In the command below I assume it is in the local directory. - You need an administrator user in engine, in the below example I used admin. Replace "yourpassword" behind the : with your own. - You need the disk-id as it is know in oVirt. You need to replace "yourdiskid", with your own. You can find the id in Engine. Select Storage, then Disks, then find the disk you want to convert and click on it. The number behind ID is the one you want and has the following format: db7b649f-76b6-4fd1-9b90-aa6becc2a283 The command: curl --cacert './ca.pem' --user 'admin@internal:yourpassword!' --request POST --header 'Version: 4' --header 'Content-Type: application/xml' --header 'Accept: application/xml' --data ' <action> <disk> <sparse>true</sparse> <format>cow</format> <backup_mode>incremental</backup_mode> </disk> </action> ' https://engine.blueit.dk/ovirt-engine/api/disks/yourdiskid/convert The command will return immediately. In engine you will see it is working on the disk image. You get the ballon messages in the right bottom corner and you can see it working in the taskview on the webinterface. Once it is done, you can start the VM again, and the disk is now thin provisioned and incremental backups are allowed. Good luck

1 0

CentOS Stream 9 Ovirt 4.5
by eevans＠digitaldatatechs.com 13 Jan '25

13 Jan '25

I have a small lab set up with 3 ovirt Centos 9 nodes and one Centos 9 Ovirt "controller node" that manages. 2 problems: It will no sync networks on hosts. It will no launch a vm unless you go to system and choose an instance size. It will boot and the network card has to be hot added. I'm not sure what I'm missing, I can give log entries showing several bad lines in diggerent modules.' Engine log while updating hosts: tail -n 100 -f engine.log ' 2025-01-09 02:43:15,722-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName = kvm02 2025-01-09 02:43:15,722-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm02, VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2025-01-09 02:43:15,729-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'}), log id: 5e145d4b 2025-01-09 02:43:15,729-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:18,057-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:18,057-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName = kvm04 2025-01-09 02:43:18,057-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:18,057-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 5e145d4b 2025-01-09 02:43:18,057-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:22,344-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'SpmStatusVDSCommand(HostName = kvm04, SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:22,344-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Irs placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover 2025-01-09 02:43:22,349-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'}), log id: 28df8c85 2025-01-09 02:43:22,349-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:25,267-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm01 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:25,267-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:25,267-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName = kvm01 2025-01-09 02:43:25,267-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:25,267-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 28df8c85 2025-01-09 02:43:25,267-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:25,275-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Irs placed on server 'acf66bf6-e159-4705-be47-484c99e746dd' failed. Proceed Failover 2025-01-09 02:43:25,279-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'}), log id: 20ec709f 2025-01-09 02:43:25,279-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:27,675-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm02 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:27,675-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:27,675-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName = kvm02 2025-01-09 02:43:27,676-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:27,676-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 20ec709f 2025-01-09 02:43:27,676-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:27,741-05 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-572) [6727848e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage 2025-01-09 02:43:27,765-05 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-572) [6727848e] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2025-01-09 02:43:27,789-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (EE-ManagedThreadFactory-engine-Thread-571) [] Finished reconstruct for pool '39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue 2025-01-09 02:43:30,154-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] START, GlusterServersListVDSCommand(HostName = kvm04, VdsIdVDSCommandParametersBase:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 770c0b37 2025-01-09 02:43:30,362-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] FINISH, GlusterServersListVDSCommand, return: [192.168.254.108/24:CONNECTED, ovirt.digitaldatatechs.com:CONNECTED, kvm02.digitaldatatechs.com:CONNECTED, kvm01.digitaldatatechs.com:CONNECTED], log id: 770c0b37 2025-01-09 02:43:30,381-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] START, AddGlusterServerVDSCommand(HostName = kvm04, AddGlusterServerVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 3b2ce207 2025-01-09 02:43:33,630-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] FINISH, AddGlusterServerVDSCommand, return: , log id: 3b2ce207 2025-01-09 02:43:33,640-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] EVENT_ID: GLUSTER_SERVER_ADD_FAILED(4,436), Failed to add host kvm04 into Cluster dds_cluster. Add host failed: rc=107 out=() err=['Probe returned with Transport endpoint is not connected'] 2025-01-09 02:43:33,641-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] START, GlusterVolumesListVDSCommand(HostName = kvm04, GlusterVolumesListVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 6a99b1ab 2025-01-09 02:43:33,748-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] FINISH, GlusterVolumesListVDSCommand, return: {b79a00d7-cc2a-48d8-b881-699991f3d930=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@205714d2}, log id: 6a99b1ab 2025-01-09 02:43:37,689-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'SpmStatusVDSCommand(HostName = kvm02, SpmStatusVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:37,691-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' TaskStatusListReturn:{status='Status [code=654, message=Not SPM: ()]'} ' 2025-01-09 02:43:37,691-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName = kvm02 2025-01-09 02:43:37,691-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm02, VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2025-01-09 02:43:37,698-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'}), log id: 201774c9 2025-01-09 02:43:37,698-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:38,381-05 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-10) [] There is no host with more than 10 running guests, no balancing is needed 2025-01-09 02:43:38,400-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-59) [] START, GlusterTasksListVDSCommand(HostName = kvm02, VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'}), log id: 702ab66f 2025-01-09 02:43:38,515-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-59) [] FINISH, GlusterTasksListVDSCommand, return: [], log id: 702ab66f 2025-01-09 02:43:40,598-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:40,598-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName = kvm01 2025-01-09 02:43:40,598-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:40,598-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 201774c9 2025-01-09 02:43:40,598-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:40,606-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Irs placed on server 'acf66bf6-e159-4705-be47-484c99e746dd' failed. Proceed Failover 2025-01-09 02:43:40,611-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'}), log id: 164d36cb 2025-01-09 02:43:40,611-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:42,950-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm04 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:42,950-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:42,950-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName = kvm04 2025-01-09 02:43:42,950-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:42,950-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 164d36cb 2025-01-09 02:43:42,950-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:42,960-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'SpmStatusVDSCommand(HostName = kvm04, SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:42,960-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Irs placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover 2025-01-09 02:43:42,964-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'}), log id: 863c329 2025-01-09 02:43:42,965-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:45,328-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:45,328-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName = kvm02 2025-01-09 02:43:45,328-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:45,328-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 863c329 2025-01-09 02:43:45,328-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:45,388-05 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-580) [2e23e34] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage 2025-01-09 02:43:45,412-05 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-580) [2e23e34] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2025-01-09 02:43:45,436-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (EE-ManagedThreadFactory-engine-Thread-579) [] Finished reconstruct for pool '39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue 2025-01-09 02:43:48,763-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] START, GlusterServersListVDSCommand(HostName = kvm04, VdsIdVDSCommandParametersBase:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: db1640e 2025-01-09 02:43:48,970-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] FINISH, GlusterServersListVDSCommand, return: [192.168.254.108/24:CONNECTED, ovirt.digitaldatatechs.com:CONNECTED, kvm02.digitaldatatechs.com:CONNECTED, kvm01.digitaldatatechs.com:CONNECTED], log id: db1640e 2025-01-09 02:43:48,983-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] START, AddGlusterServerVDSCommand(HostName = kvm04, AddGlusterServerVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 717f84ae 2025-01-09 02:43:52,255-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] FINISH, AddGlusterServerVDSCommand, return: , log id: 717f84ae 2025-01-09 02:43:52,277-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] EVENT_ID: GLUSTER_SERVER_ADD_FAILED(4,436), Failed to add host kvm04 into Cluster dds_cluster. Add host failed: rc=107 out=() err=['Probe returned with Transport endpoint is not connected'] 2025-01-09 02:43:52,278-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] START, GlusterVolumesListVDSCommand(HostName = kvm04, GlusterVolumesListVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 6a43f9e6 2025-01-09 02:43:52,384-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] FINISH, GlusterVolumesListVDSCommand, return: {b79a00d7-cc2a-48d8-b881-699991f3d930=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@205714d2}, log id: 6a43f9e6 2025-01-09 02:43:55,341-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'SpmStatusVDSCommand(HostName = kvm02, SpmStatusVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:55,343-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' TaskStatusListReturn:{status='Status [code=654, message=Not SPM: ()]'} ' 2025-01-09 02:43:55,343-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName = kvm02 2025-01-09 02:43:55,343-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm02, VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2025-01-09 02:43:55,350-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'}), log id: 35cf220b 2025-01-09 02:43:55,350-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:57,661-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:57,661-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName = kvm04 2025-01-09 02:43:57,661-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:57,661-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 35cf220b 2025-01-09 02:43:57,661-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:57,671-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'SpmStatusVDSCommand(HostName = kvm04, SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:57,671-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Irs placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover 2025-01-09 02:43:57,675-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'}), log id: 2ff24c19 2025-01-09 02:43:57,676-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:00,065-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm02 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:00,065-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:00,065-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName = kvm02 2025-01-09 02:44:00,065-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:00,065-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 2ff24c19 2025-01-09 02:44:00,065-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:04,367-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'SpmStatusVDSCommand(HostName = kvm02, SpmStatusVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:44:04,367-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Irs placed on server 'cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6' failed. Proceed Failover 2025-01-09 02:44:04,372-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'}), log id: 23ed7788 2025-01-09 02:44:04,372-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:07,292-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm01 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:07,292-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:07,292-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName = kvm01 2025-01-09 02:44:07,292-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:07,292-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 23ed7788 2025-01-09 02:44:07,292-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:07,375-05 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-594) [6e7f60bb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage 2025-01-09 02:44:07,412-05 INFO [org.ovirt.engine.core.bll.network.host.SyncAllHostNetworksCommand] (default task-7) [9913b205-9b2e-4f8a-86e3-048a9cdff2ba] Running command: SyncAllHostNetworksCommand internal: false. Entities affected : ID: acf66bf6-e159-4705-be47-484c99e746dd Type: VDSAction group CONFIGURE_HOST_NETWORK with role type ADMIN 2025-01-09 02:44:07,421-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, GlusterServersListVDSCommand(HostName = kvm04, VdsIdVDSCommandParametersBase:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 340c8f9d 2025-01-09 02:44:07,423-05 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-594) [6e7f60bb] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2025-01-09 02:44:07,423-05 INFO [org.ovirt.engine.core.bll.network.host.PersistentHostSetupNetworksCommand] (default task-7) [7814c7ae] Running command: PersistentHostSetupNetworksCommand internal: true. Entities affected : ID: acf66bf6-e159-4705-be47-484c99e746dd Type: VDS 2025-01-09 02:44:07,448-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (EE-ManagedThreadFactory-engine-Thread-593) [] Finished reconstruct for pool '39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue 2025-01-09 02:44:07,460-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [7814c7ae] EVENT_ID: PERSIST_HOST_SETUP_NETWORK_ON_HOST(10,902), (1/1): Applying network's changes on host kvm01. (User: admin@ovirt@internalkeycloak-authz) 2025-01-09 02:44:07,461-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Before acquiring lock-timeout 'EngineLock:{exclusiveLocks='[HOST_NETWORKacf66bf6-e159-4705-be47-484c99e746dd=HOST_NETWORK]', sharedLocks=''}' 2025-01-09 02:44:07,461-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Lock-timeout acquired to object 'EngineLock:{exclusiveLocks='[HOST_NETWORKacf66bf6-e159-4705-be47-484c99e746dd=HOST_NETWORK]', sharedLocks=''}' 2025-01-09 02:44:07,497-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Running command: HostSetupNetworksCommand internal: true. Entities affected : ID: acf66bf6-e159-4705-be47-484c99e746dd Type: VDSAction group CONFIGURE_HOST_NETWORK with role type ADMIN 2025-01-09 02:44:07,497-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Before acquiring lock in order to prevent monitoring for host 'kvm01' from data-center 'DDS_Data_Center' 2025-01-09 02:44:07,497-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Lock acquired, from now a monitoring of host will be skipped for host 'kvm01' from data-center 'DDS_Data_Center' 2025-01-09 02:44:07,498-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-7) [7814c7ae] START, HostSetupNetworksVDSCommand(HostName = kvm01, HostSetupNetworksVdsCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vds='Host[kvm01,acf66bf6-e159-4705-be47-484c99e746dd]', rollbackOnFailure='true', commitOnSuccess='true', connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='false', bonding='false', networkName='vlan3', vdsmName='vlan3', nicName='eno2', vlan='3', vmNetwork='true', stp='false', properties='[]', ipv4BootProtocol='STATIC_IP', ipv4Address='172.50.26.252', ipv4Netmask='255.255.255.0', ipv4Gateway='172.50.26.1', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}, HostNetwork:{defaultRoute='true', bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt', nicName='eno1', vlan='null', vmNetwork='true', stp='false', properties='[]', ipv4BootProtocol='STAT IC_IP', ipv4Address='192.168.254.252', ipv4Netmask='255.255.255.0', ipv4Gateway='192.168.254.1', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='[NameServer:{address='192.168.254.222'}, NameServer:{address='192.168.254.232'}]'}]', removedNetworks='[]', bonds='[]', removedBonds='[]', clusterSwitchType='LEGACY', managementNetworkChanged='true'}), log id: 56a27c18 2025-01-09 02:44:07,499-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-7) [7814c7ae] FINISH, HostSetupNetworksVDSCommand, return: , log id: 56a27c18 2025-01-09 02:44:07,631-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, GlusterServersListVDSCommand, return: [192.168.254.108/24:CONNECTED, ovirt.digitaldatatechs.com:CONNECTED, kvm02.digitaldatatechs.com:CONNECTED, kvm01.digitaldatatechs.com:CONNECTED], log id: 340c8f9d 2025-01-09 02:44:07,642-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, AddGlusterServerVDSCommand(HostName = kvm04, AddGlusterServerVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 7c25cce3 2025-01-09 02:44:10,879-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, AddGlusterServerVDSCommand, return: , log id: 7c25cce3 2025-01-09 02:44:10,923-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] EVENT_ID: GLUSTER_SERVER_ADD_FAILED(4,436), Failed to add host kvm04 into Cluster dds_cluster. Add host failed: rc=107 out=() err=['Probe returned with Transport endpoint is not connected'] 2025-01-09 02:44:10,923-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, GlusterVolumesListVDSCommand(HostName = kvm04, GlusterVolumesListVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: effd1e8 2025-01-09 02:44:11,033-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, GlusterVolumesListVDSCommand, return: {b79a00d7-cc2a-48d8-b881-699991f3d930=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@205714d2}, log id: effd1e8 2025-01-09 02:44:17,314-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' TaskStatusListReturn:{status='Status [code=654, message=Not SPM: ()]'} ' 2025-01-09 02:44:17,315-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName = kvm01 2025-01-09 02:44:17,315-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm01, VdsIdVDSCommandParametersBase:{hostId='acf66bf6-e159-4705-be47-484c99e746dd'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2025-01-09 02:44:17,331-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] START, ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'}), log id: bcbf59e 2025-01-09 02:44:17,331-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:18,272-05 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [7814c7ae] EVENT_ID: VDS_NETWORKS_OUT_OF_SYNC(1,110), Host kvm01's following network(s) are not synchronized with their Logical Network configuration: ovirtmgmt,vlan3. 2025-01-09 02:44:18,283-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Host setup networks finished. Lock released. Monitoring can run now for host 'kvm01' from data-center 'DDS_Data_Center' 2025-01-09 02:44:18,285-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Lock freed to object 'EngineLock:{exclusiveLocks='[HOST_NETWORKacf66bf6-e159-4705-be47-484c99e746dd=HOST_NETWORK]', sharedLocks=''}' 2025-01-09 02:44:18,305-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [7814c7ae] EVENT_ID: PERSIST_SETUP_NETWORK_ON_HOST_FINISHED(10,903), (1/1): Successfully applied changes on host kvm01. (User: admin@ovirt@internalkeycloak-authz) 2025-01-09 02:44:18,316-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [7814c7ae] EVENT_ID: HOST_SYNC_ALL_NETWORKS_FINISHED(10,901), Managed to sync all host kvm01 networks. 2025-01-09 02:44:19,668-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm04 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:19,668-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:19,668-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName = kvm04 2025-01-09 02:44:19,668-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:19,668-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: bcbf59e 2025-01-09 02:44:19,668-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:19,679-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'SpmStatusVDSCommand(HostName = kvm04, SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:44:19,679-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Irs placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover 2025-01-09 02:44:19,684-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] START, ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'}), log id: 251dbd22 2025-01-09 02:44:19,684-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:22,181-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:22,181-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName = kvm01 2025-01-09 02:44:22,181-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:22,181-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 251dbd22 2025-01-09 02:44:22,181-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:22,189-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Irs placed on server 'acf66bf6-e159-4705-be47-484c99e746dd' failed. Proceed Failover 2025-01-09 02:44:22,198-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] START, ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'}), log id: 1f60ebb1 2025-01-09 02:44:22,198-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:24,581-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:24,581-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName = kvm02 2025-01-09 02:44:24,581-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:24,582-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 1f60ebb1 2025-01-09 02:44:24,582-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:24,658-05 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-610) [2af75e62] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage 2025-01-09 02:44:24,691-05 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-610) [2af75e62] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2025-01-09 02:44:24,715-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (EE-ManagedThreadFactory-engine-Thread-609) [] Finished reconstruct for pool '39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue While attempting to start vm will be sent after but I think the net allocation issue with hosts is the center of the vm issues,,

2 1

Restart oVirt-Engine
by Jeremey Wise 09 Jan '25

09 Jan '25

How ,without reboot of hosting system, do I restart the oVirt engine? # I tried below but do not seem to effect the virtual machine [root@thor iso]# systemctl restart ov ovirt-ha-agent.service ovirt-imageio.service ovn-controller.service ovs-delete-transient-ports.service ovirt-ha-broker.service ovirt-vmconsole-host-sshd.service ovsdb-server.service ovs-vswitchd.service [root@thor iso]# # You cannot restart the VM " HostedEngine " as it responses: Error while executing action: HostedEngine: - Cannot restart VM. This VM is not managed by the engine. Reason is I had to do some work on a node. Reboot it.. it is back up.. network is all fine.. Cockpit working fine... and gluster fine.. But oVirt-Engine refuses to accept the node is up. -- p <jeremey.wise(a)gmail.com>enguinpages

4 4

Host deploy failure: Configure OVN for oVirt
by stephan.badenhorst＠fnb.co.za 05 Jan '25

05 Jan '25

Good day, I am running into a problem during host deploy via the oVirt engine GUI since upgrading to ovirt-engine-4.5.5-1.el8. The "Configure OVN for oVirt" task seem to fail when trying to run the vdsm-tool ovn-config command. Host deploy used to work fine when the engine was on version 4.5.4. Anyone that can guide me on the right path to get past this issue? Does not seem to be a new problem - https://lists.ovirt.org/archives/list/users@ovirt.org/thread/IDLGSBQFX35EHH… Log extract: 2024-02-01 14:48:19 SAST - TASK [ovirt-provider-ovn-driver : Configure OVN for oVirt] ********************* . . . "stdout" : "fatal: [mob-r1-l-ovirt-aa-1-23.x.fnb.co.za]: FAILED! => {\"changed\": true, \"cmd\": [\"vdsm-tool\", \"ovn-config\", \"192.168.2.100\", \"host23.mydomain.com\"], \"delta\": \"0:00:00.538143\", \"end \": \"2024-02-01 14:48:20.596823\", \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2024-02-01 14:48:20.058680\", \"stderr\": \"Traceback (most recent call last):\\n File \\\"/usr/lib/python3.6/site-packages/vdsm/t ool/ovn_config.py\\\", line 117, in get_network\\n return networks[net_name]\\nKeyError: 'host23.mydomain.com'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most rec ent call last):\\n File \\\"/usr/bin/vdsm-tool\\\", line 195, in main\\n return tool_command[cmd][\\\"command\\\"](*args)\\n File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\\n ip_address = get_ip_addr(get_network(network_caps(), net_name))\\n File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 119, in get_network\\n raise NetworkNotFoundError(net_name)\\nvdsm.tool.ovn _config.NetworkNotFoundError: host23.mydomain.com\", \"stderr_lines\": [\"Traceback (most recent call last):\", \" File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 117, in get_network \", \" return networks[net_name]\", \"KeyError: 'host23.mydomain.com'\", \"\", \"During handling of the above exception, another exception occurred:\", \"\", \"Traceback (most recent call last):\", \" File \ \\"/usr/bin/vdsm-tool\\\", line 195, in main\", \" return tool_command[cmd][\\\"command\\\"](*args)\", \" File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\", \" ip_address = get_ip_addr(get_network(network_caps(), net_name))\", \" File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 119, in get_network\", \" raise NetworkNotFoundError(net_name)\", \"vdsm.tool.ovn_config. NetworkNotFoundError: host23.mydomain.com\"], \"stdout\": \"\", \"stdout_lines\": []}", Thanks in advance!! Stephan

5 5

Re: Host deploy failure: Configure OVN for oVirt
by Andrea Cavegn 03 Jan '25

03 Jan '25

We use ovirt-engine 4.5.5-1.el8 and for us the following did the trick: 1. First you have to decide, if you need the ovn/ovs functionality. This is a cluster level decision! * If needed, you need to have a functional (You can test it) external network provider configured (Administration -> Providers -> External Network Provider) ?? As far as I know, a default "ovirt-provider-ovn" is already present in each installation * Then in the cluster option you choose the "Default Network Provider". If you don't need OVS/OVN, then select "No Default Provider", else select the one you need * If you change here anything, ovirt want you to reinstall all your hosts in this cluster! 2. Then, replace the ansible file in your ovirt-engine: * /usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-provider-ovn-driver/tasks/configure.yml * The new version can be found here: https://github.com/oVirt/ovirt-engine/blob/master/packaging/ansible-runner-… as far as I can tell, only the last line (when statement) changes Probably, only ppl who don't need ovn/ovs had (Setting: "No Default Provider") had this problem. 1. Reinstall the hosts As always, know what you do, before you anything.

1 0

Enroll Certificate
by Денис Панкратьев 03 Jan '25

03 Jan '25

Hello! My ovirt host (4.5.5) certificate is about to expire (in 46 days, Jan 20, 2025). d.pankratiev@zabbix ~]$ echo | openssl s_client -servername ovirt1.gtc.local -connect XXX.XXX.XXX.XXX.XXX:9090 2>/dev/null | openssl x509 -noout -dates 2>/dev/null notBefore=Dec 22 05:27:04 2023 GMT notAfter=Jan 20 05:27:04 2025 GMT 1. I have migrated all VMs to another host 2. I put the host in maintenance mode (Managеment -> Maintenance) 3. Do enroll certificate (Installation -> Enroll Certificate) 4. Everything went successfully, without errors (Enrolling certificate for host ovirt1.gtc.local was completed successfully). But the certificate date did not change, still expire (after 46 days, Jan 20 2025). d.pankratiev@zabbix ~]$ echo | openssl s_client -servername ovirt1.gtc.local -connect XXX.XXX.XXX.XXX.XXX:9090 2>/dev/null | openssl x509 -noout -dates 2>/dev/null notBefore=Dec 22 05:27:04 2023 GMT notAfter=Jan 20 05:27:04 2025 GMT What did I do wrong? Is there anything else I need to do?

5 5

oVirt and TPM
by Fabrice Bacchella 30 Dec '24

30 Dec '24

Did some one use TPM in oVirt ? I tried to active it but failed to use. The VM is configured to use an EFI secure boot. The VM (a RHEL 9.5 OS) see it, but I can’t communicate with it: $ dmesg | grep -i tpm [ 0.000000] efi: SMBIOS=0x7ddd5000 TPMFinalLog=0x7dfe7000 ACPI=0x7df7e000 ACPI 2.0=0x7df7e014 MEMATTR=0x7cbd9018 MOKvar=0x7dd82000 RNG=0x7df71018 TPMEventLog=0x7c544018 [ 0.018626] ACPI: TPM2 0x000000007DF76000 00004C (v04 BOCHS BXPC 00000001 BXPC 00000001) [ 0.018657] ACPI: Reserving TPM2 table memory at [mem 0x7df76000-0x7df7604b] [ 5.312318] systemd[1]: TPM2 PCR Machine ID Measurement was skipped because of an unmet condition check (ConditionPathExists=/sys/firmware/efi/efivars/StubPcrKernelImage-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f). pcsc_scan fails to read it: $ sudo pcsc_scan -s -v PC/SC device scanner V 1.6.2 (c) 2001-2022, Ludovic Rousseau <ludovic.rousseau(a)free.fr> SCardEstablishContext: RPC transport error.

2 2

Curl error (7): Couldn't connect to server for https://mirrorlist.ovirt.org/mirrorlist-ovirt-4.5-el8
by eyup ustaoglu 29 Dec '24

29 Dec '24

Hello. I keep getting this error during the ovirt PREPARE VM phase. Is there a site where I can get the current repositories under /etc/yum.repos.d? Also, have ovirt updates stopped? Why are repo addresses that have been unreachable for a long time still in the ovirt node?

1 0

Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for http://mirrorlist.centos.org/?release=8-stream&arch=x86_64&repo=virt-ovirt-45 [Could not resolve host: mirrorlist.centos.org]
by onr onr 26 Dec '24

26 Dec '24

I am in the Prepare VM step in ovirt installation. and I keep getting the above error. Why is the unused repo address defined even though I downloaded the latest ovirt-node? Where can I find repo addresses for el8 that can be accessed for installation? Is ovirt a discontinued system?

1 0