January 2025 - Users - oVirt List Archives

ovirt-engine-deploy error
by hiroki.tamai.fu＠nttdocomo.com 19 May '25

19 May '25

Hi, everyone. This is my first time using ovirt and I encountered this error in ovirt-engine-deploy. ERROR LOG =========================================================================== [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Create ISO disk] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fix local VM directory permission] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Create local VM] [ ERROR ] fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["virt-install", "-n", "HostedEngineLocal", "--os-variant", "rhel8.0", "--virt-type", "kvm", "--memory", "16384", "--vcpus", "4", "--network", "network=default,mac=00:16:3e:7e:2b:e0,model=virtio", "--disk", "/var/tmp/localvm4swegum3/images/799d4506-f930-41ba-9c0d-1fe7af6af1bc/5d3a11a5-88c9-45d6-a3a6-e4819ce36702", "--import", "--disk", "path=/var/tmp/localvm4swegum3/seed.iso,device=cdrom", "--autoconsole", "text", "--rng", "/dev/random", "--graphics", "none", "--sound", "none", "--controller", "usb,model=none", "--memballoon", "none", "--boot", "hd,bootmenu.enable=on,bios.useserial=on", "--clock", "kvmclock_present=yes", "--serial=pty,log.file=/var/log/libvirt/qemu/HostedEngineLocal-console.log"], "delta": "0:00:01.611656", "end": "2024-12-04 08:55:40.844092", "msg": "non-zero return code", "rc": 1, "start": "2024-12-04 08:55:39.232436", "stderr": "ERROR Failed to connect socket to '/var/run/libvirt/virtnetworkd-soc k': No such file or directory\nDomain installation does not appear to have been successful.\nIf it was, you can restart your domain by running:\n virsh --connect qemu:///system start HostedEngineLocal\notherwise, please restart your installation.", "stderr_lines": ["ERROR Failed to connect socket to '/var/run/libvirt/virtnetworkd-sock': No such file or directory", "Domain installation does not appear to have been successful.", "If it was, you can restart your domain by running:", " virsh --connect qemu:///system start HostedEngineLocal", "otherwise, please restart your installation."], "stdout": "\nStarting install...", "stdout_lines": ["", "Starting install..."]} [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks] =========================================================================== I restored the virtnetworkd-sock file with reference to this ticket, but the deployment still fails with the same error. https://github.com/crc-org/crc/issues/2839 If anyone has encountered a similar issue, could you please let me know how you resolved it? Deploying with the Self-Hosted Engine and using ovirt4.5 el9. Thank you.

2 1

oVirh Host Update Problem
by Vladislav Solovei 01 Apr '25

01 Apr '25

Freshly installed using nightly build, freshly installed OS :) OS: AlmaLinux 9.5 Can't update the system (and can't reinstall the Host) Repository copr:copr.fedorainfracloud.org:ovirt:ovirt-master-snapshot is listed more than once in the configuration Last metadata expiration check: 0:22:12 ago on Fri 24 Jan 2025 11:41:49 EET. Error: Problem: package rdo-ovn-host-2:22.12-2.el9s.noarch from ovirt-master-centos-stream-openstack-yoga-testing requires rdo-ovn = 2:22.12-2.el9s, but none of the providers can be installed - cannot install the best update candidate for package ovn22.09-host-22.09.0-31.el9s.x86_64 - package rdo-ovn-2:22.12-2.el9s.noarch from ovirt-master-centos-stream-openstack-yoga-testing is filtered out by exclude filtering (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) [ovirt-master-centos-stream-openstack-yoga-testing] name=CentOS Stream 9 - OpenStack Yoga Repository - testing baseurl=https://buildlogs.centos.org/9-stream/cloud/$basearch/openstack-yog… gpgcheck=0 enabled=1 exclude= openstack-ansible-core python3-rdo-openvswitch rdo-network-scripts-openvswitch rdo-openvswitch rdo-ovn rdo-ovn-central How can this problem be resolved? Should the rdo-ovn package, which is a dependency of rdo-ovn-host-2:22.12-2, be filtered out?

2 1

Couldn't resolve host name for http://mirrorlist.centos.org/?release=8-stream&arch=x86_64&repo=storage-ceph-pacific
by biswajit_list＠tetrain.com 02 Mar '25

02 Mar '25

[root@esign-dr ~]# cat /etc/os-release NAME="CentOS Stream" VERSION="9" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="9.1.2206.0" VARIANT="oVirt Node 4.5.5" VARIANT_ID="ovirt-node" PRETTY_NAME="oVirt Node 4.5.5" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:9" HOME_URL="https://www.ovirt.org/" BUG_REPORT_URL="https://bugzilla.redhat.com/" PLATFORM_ID="platform:el9" While Deploying Hosted-engine Deploy I keep getting the Following error which is pointing to Centos 8 Stream repo ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:110 fatal: [localhost -> 192.168.222.254]: FAILED! => {"changed": false, "msg": "Failed to download metadata for repo 'ovirt-4.4-centos-ceph-pacific': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for http://mirrorlist.centos.org/?release=8-stream&arch=x86_64&repo=storage-cep… [Could not resolve host: mirrorlist.centos.org]", "rc": 1, "results": []} Can Some one Help TIA Biswajit

4 4

Error when deploy Ovirt4.4 Hosted Engine
by staybox＠gmail.com 17 Feb '25

17 Feb '25

Hello, I get error, need help. [ ERROR ] fatal: [localhost]: FAILED! => {"msg": "The conditional check 'not ipv6_deployment|bool and route_rules_ipv4.stdout | from_json | selectattr('priority', 'equalto', 100) | selectattr('dst', 'equalto', virbr_cidr_ipv4 | ipaddr('address') ) | list | length == 0' failed. The error was: error while evaluating conditional (not ipv6_deployment|bool and route_rules_ipv4.stdout | from_json | selectattr('priority', 'equalto', 100) | selectattr('dst', 'equalto', virbr_cidr_ipv4 | ipaddr('address') ) | list | length == 0): 'dict object' has no attribute 'dst'\n\nThe error appears to be in '/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_engine_setup/tasks/bootstrap_local_vm/01_prepare_routing_rules.yml': line 81, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n changed_when: true\n - name: Add IPv4 inbound route rules\n ^ here\n"}

4 4

How to list all snapshots?
by jorgevisentini＠gmail.com 14 Feb '25

14 Feb '25

Hello everyone! First, I would like to thank everyone involved in this wonderful project. I leave here my sincere thanks! Does anyone know if it is possible to list all snapshots automatically? It can be by ansible, python, shell... any way that helps to list them all without having to enter Domain by Domain. Thank you all!

4 3

Unable to launch new VM on few hosts due "Max free Memory for scheduling new VMs:" has less value
by dushyantk.sun＠gmail.com 12 Feb '25

12 Feb '25

Few hosts shows unable to launch vms due memory is not available to schedule vm. However vm has enough memory when check free command. If any one come across such issue, please let me know how we can fix it in oracle olvm

1 1

VM hangs when expanding direct LUN
by John Mortensen 11 Feb '25

11 Feb '25

Hi, We have multible oVirt clusters (based on CentOS 9 Stream) with several nodes (based on the 4.5 el9 node image) in each and a great number of virtual machines (mainly RHEL 8). The problem we encounter is random, it happens sometimes and other times not. Backend storage consists of SAN attached IBM Storwize V7000. Problem is when we expand a LUN on the storage system and wishes to expand it in oVirt for use in the VM, sometimes this process hangs the VM completely, it shows as running in oVirt GUI, it cannot be paused/shutdown from the GUI - only way to shut it down is to login to the node running the VM and from virsh run a destroy command. Anyone else encountered this problem? /John

3 8

Re: Help With CPU Type
by Jorge Visentini 31 Jan '25

31 Jan '25

Hi. If you do this right, you will need downtime because the VM's processor model will change, even though the VM's processor is only showing the family. Another thing that can happen is that you change the family and the host becomes non-operational in the cluster, so you would have to test that at least. But you can roll back to the original cluster family. I have done this in the past and as far as I remember, the only thing that causes a warning on the VM is that it is running on a different cluster, that's all. Please correct me if I'm wrong. Cheers! Em qui., 23 de jan. de 2025 às 02:13, Daniel Martins via Users < users(a)ovirt.org> escreveu: > Hi There, > > > > Please may you assist me with the below? > > > > I need to remove an oVirt host within a cluster that is using CPU Type > Intel Westmere IBRS SSBD Family. Once the host is removed i need to change > the CPU type to Intel Broadwell which the remaining hosts Use. > > > > So my question is will there need to be down time on the Virtual machines > to change this? > > > > Regards, > > > > > > > > > > > > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/IEJW5M6KUHWW5… > -- Att, Jorge Visentini +55 55 98432-9868

1 0

Edit/Change MAC Address Pools
by milad.norouzi1370＠gmail.com 31 Jan '25

31 Jan '25

According to this link: https://www.ovirt.org/documentation/administration_guide/index.html#sect-MA…, we can edit the MAC address pool. Question: If I edit the MAC address pool, what effect will it have on the existing VMs' MAC addresses? Is there any mention in the documentation or elsewhere that indicates this change could be harmful to current configurations?

2 1

oVirt does not get keycloak groups
by Christian Fischer 31 Jan '25

31 Jan '25

Hi everyone. I have setup a oVirt Installation with Keycloak for a PoC-Evaluation but i'm actually struggling with Keycloak Groups sync to oVirt. I Added Active Directory Users and also a Group Mapper for LDAP Groups -> in Keycloak i see all Users and also that they belong to Groups but the Group-Membership is not populated to oVirt back. As a result i can't grant Permissions groupwise - only per user. For the initial group-sync i ticked the Option "Drop non-existing groups during sync" which also deleted the ovirt-administrator group. Maybe this is the main Problem here. I created this group again and unchecked this option but it still does not sync the groups. What do you recommend? Purge the system and do a clean install or is there another easy fix out there? Thanks, have a nice weekend. Greetings Christian

1 0

Questions and Suggestions for oVirt
by Daniel Gurgel 30 Jan '25

30 Jan '25

With the new leadership of the project, most users who admire and support oVirt would like to know the roadmap and future plans. From my perspective, I understand that Red Hat discontinued its support purely for financial reasons. Therefore, this is probably one of the key areas where oVirt needs to strengthen. Wouldn't it be worth seeking support and collaboration with Veeam and Oracle, given its "OLVM"? Especially considering that, after the VMware/Broadcom licensing changes, Oracle has been aggressively offering OLVM to customers as a migration alternative. A positive development was Veeam Backup & Replication recently adding support for OLVM. This has increased interest in the solution, but there are still limitations, such as migration/backup/restore between different hypervisors (interestingly, this feature is available for Proxmox/KVM, which uses similar technology). These are some of the main questions from customers: 1. When will it be updated to Enterprise Linux 9 or 10? 2. What support will be available for new technologies? 3. How about a change in the Engine/Manager, allowing VMs to be managed even when the Engine is unavailable! (Sorry, but this is one of oVirt's biggest bottlenecks, and inevitably, comparisons with VMware/VCenter arise.) 4. Support for VM migration between Datacenters should also be carefully considered. **Note**: The oVirt website needs to be more user-friendly for general users. I also suggest a fundraising campaign—I would be happy to contribute, and I believe the entire user base would as well!

1 0

Re: [EXT]Re: Re: [SOLVED] Re: Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?
by Cosmin Cioranu 29 Jan '25

29 Jan '25

I have checked the solution and it works, apart from the console feature of the virtual machine. In my case, once a got control over the host, migrated all the vms and then Maintenance and then Enroll Certificate. So: an updated version ``` set -x ## Make the script echo everything out, so if it fails you know where set -e ## Make the script STOP on any error set my_date="$(date +"%Y%m%d%H%M%S")" # Backup the existing CA files /bin/cp -p /etc/pki/ovirt-engine/private/ca.pem /etc/pki/ovirt-engine/private/ca.pem.$my_date /bin/cp -p /etc/pki/ovirt-engine/ca.pem{,.$my_date} /bin/mv /etc/pki/ovirt-engine/certs/ca.der{,.$my_date} # Sign the key openssl x509 -signkey /etc/pki/ovirt-engine/private/ca.pem -in /etc/pki/ovirt-engine/ca.pem -out /etc/pki/ovirt-engine/ca.pem.new -days 3650 -sha256 openssl x509 -in /etc/pki/ovirt-engine/ca.pem.new -text > /etc/pki/ovirt-engine/ca.pem.new.full # Put the files into place /bin/mv -f /etc/pki/ovirt-engine/ca.pem.new.full /etc/pki/ovirt-engine/ca.pem /bin/cp -p /etc/pki/ovirt-engine/ca.pem.new /etc/pki/ovirt-engine/certs/ca.der ``` Now you need to copy the new CA file over to the host: Source: ENGINE `/etc/pki/ovirt-engine/ca.pem` Dest: HOST `/tmp/new-ca.pem` ########### On the oVirt Host ########### # Create a CSR using the information from the existing certificate and the existing key: openssl x509 -x509toreq -in /etc/pki/libvirt/clientcert.pem -out /tmp/HOST.csr -signkey /etc/pki/libvirt/private/clientkey.pem Now you need to copy the new CA file over to the host: Source: HOST `/tmp/HOST.csr` Dest: ENGINE `/etc/pki/ovirt-engine/requests/full.hostname.com.req` ########### On the ENGINE HOST ########### # Now sign it: /usr/share/ovirt-engine/bin/pki-enroll-request.sh --name=full.hostname.com # NB -- adding --san results in an error: --san=host.na.me (So no Subject Alternate Names) Now you need to copy the new Certificate file over to the host: Source: ENGINE /etc/pki/ovirt-engine/certs/full.hostname.com.cer Dest: HOST /tmp/new-cert.pem ########### On the oVirt Host ########### (see bellow the written script) #!/bin/bash #set -x set -e mydate="$(date +"%Y%m%d%H%M%S")" echo "[.] reference ${mydate}" for x in /tmp/new-ca.pem /tmp/new-cert.pem ; do echo -n "[.] checking $x" if [ ! -e "$x" ]; then echo "[not found]" exit else echo "[found]" fi done for x in /etc/pki/libvirt/clientcert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-migrate/server-cert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem /etc/pki/vdsm/libvirt-vnc/server-cert.pem; do echo -n "copying $x to ${x}.${mydate}" /bin/mv -n $x ${x}.${mydate} echo "[done]"; echo -n "[.] copying /tmp/new-cert.pem to $x" /bin/cp /tmp/new-cert.pem ${x} echo "[done]" echo -n "[.] setting permissions to $x 644 root:kvm" chmod 644 ${x} chown root:kvm ${x} echo "[done]" done for x in /etc/pki/vdsm/libvirt-migrate/ca-cert.pem /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-vnc/ca-cert.pem /etc/pki/vdsm/libvirt-spice/ca-cert.pem /etc/pki/CA/cacert.pem; do echo -n "copying $x to ${x}.${mydate}" /bin/mv -n $x ${x}.${mydate} echo "[done]" echo -n "[.] copying /tmp/new-ca.pem to $x" /bin/cp /tmp/new-ca.pem ${x} echo "[done]"; echo -n "[.] setting permissions to $x 644 root:kvm" chmod 644 ${x} chown root:kvm ${x} echo "[done]" done for x in /etc/pki/vdsm/keys/vdsmkey.pem; do echo -n "copying $x to ${x}.${mydate}" /bin/mv -n $x ${x}.${mydate} echo "[done]" echo -n "[.] copying /etc/pki/libvirt/private/clientkey.pem to $x" /bin/cp /etc/pki/libvirt/private/clientkey.pem ${x} echo "[done]"; echo -n "[.] setting permissions to $x 644 root:kvm" chmod 644 ${x} chown root:kvm ${x} echo "[done]" done echo "[.] Checks " openssl x509 -in /etc/pki/libvirt/clientcert.pem -noout -dates openssl x509 -in /etc/pki/vdsm/certs/cacert.pem -noout -dates openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -dates openssl x509 -in /etc/pki/vdsm/libvirt-migrate/ca-cert.pem -noout -dates openssl x509 -in /etc/pki/vdsm/libvirt-migrate/client-cert.pem -noout -dates openssl x509 -in /etc/pki/vdsm/libvirt-migrate/server-cert.pem -noout -dates openssl x509 -in /etc/pki/vdsm/libvirt-spice/ca-cert.pem -noout -dates openssl x509 -in /etc/pki/vdsm/libvirt-spice/server-cert.pem -noout -dates ```` Note: I whould backup the entire pki folder on both node and engine.

1 0

snapshot solution: Existing snapshots that were taken after this one will be erased.
by dhanaraj.ramesh＠yahoo.com 29 Jan '25

29 Jan '25

Hi Team, when I want to commit the older snapshots I'm getting warning stating " Existing snapshots that were taken after this one will be erased.". is there any way we can retain the latest snapshots as is in the chain? I knew cloning and template export options are there to secure that latest snapshot data but these are will consume additional space in storage and take time.

4 3

Re: [External] : Re: Unable to migrate vm with 4 (500g) disks from vmware to ovirt.
by dushyantk.sun＠gmail.com 24 Jan '25

24 Jan '25

Hi Marcos and Simon, Provided document was helpful. Thank you.

1 0

Re: [External] : Re: Unable to migrate vm with 4 (500g) disks from vmware to ovirt.
by dushyantk.sun＠gmail.com 23 Jan '25

23 Jan '25

Currently linux vm which i am trying to migrate has 4 (500GB) disk. All disks on same datastore. It is iscsi datastore. Do we have any configuration where we can enabled verbose mode while migration is going on to see what is happening as currently unable to identify failure. Also is there any limitation if vms with multiple disk is not possible to migrate?

2 2

Unable to migrate vm with 4 (500g) disks from vmware to ovirt.
by dushyantk.sun＠gmail.com 23 Jan '25

23 Jan '25

I have vm with multiple disk attached to it. While migrating vm from vmware to ovirt, it started but gets failed after 5-6 hours. However vms with single disk migrated successfully. Following ovirt UI to migrate vm. Please let me know if any one come across such scenario

4 6

oVirt Translations & Future
by Jean-Louis Dupond 23 Jan '25

23 Jan '25

Hi, We've setup some alternative for the (dead) Zanata translation project for oVirt. You can find the Weblate setup on the following location: https://translate.ovirt.tech/projects/ovirt-engine/ So if somebody wants to add translations, feel free! :) Will try to add translations for the other projects the coming days. Next to that, we are working hard to get oVirt better! If you want to help, feel free to open PR's or send me an mail. A lot can be done, so you won't get bored. Some things that we are looking into: - Java updates - Wildfly update - AlmaLinux 10 / CentOS Stream 10 support Feel free to contact me if you have further questions. Jean-Louis

6 7

Re: [External] : OLVM/ovirt 4.5 support for 5th Gen Intel Xeon Gold (Emerald Rapids)
by Simon Coter 23 Jan '25

23 Jan '25

Hi Hassan, we internally fixed it by reverting our patch and use the oVirt community one: <https://github.com/oVirt/ovirt-engine/pull/978> [978.png] core: Introduce new Intel, AMD and IBM CPU's by JasperB-TeamBlue · Pull Request #978 · oVirt/ovirt-engine<https://github.com/oVirt/ovirt-engine/pull/978> github.com<https://github.com/oVirt/ovirt-engine/pull/978> This will be released as part of the next Errata for OLVM. Simon On Jan 23, 2025, at 9:44 AM, Hassan Shabbir <hshabbir315(a)gmail.com> wrote: Hello Simon, Thank you for the response, what about 5th Gen Emerald Rapids? correct me if i'm wrong: support for 4th Gen Sapphire Rapids is already in place in OLVM. Regards, On Thu, Jan 23, 2025, 1:32 PM Simon Coter <simon.coter(a)oracle.com<mailto:simon.coter@oracle.com>> wrote: We introduced a patch to address the Sapphire Rapids family support on OLVM 4.5.5 We recently synced our patch with the one had been pushed by the oVirt community and should be out quite soon. Simon > On Jan 19, 2025, at 9:28 PM, hshabbir315(a)gmail.com<mailto:hshabbir315@gmail.com> wrote: > > Hi Guys, > > Can someone please confirm if 5th Gen Intel Gold (Emerald Rapids) CPUs are compatible with OLVM/ovirt 4.5.5.x? We have a OLVM/ovirt 4.5.5.x test lab setup cluster running on older hardware (Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz - Haswell Family) with compatibility version 4.7. While going through the engine db backup we observed that there are no entries in the vdc_options table for Emerald Rapids family of Intel CPUs. Does this mean that these CPUs are not supported by ovirt 4.5.5.x? or these CPUs are supported but the extra features/instruction sets offered by these CPUs over Sapphire Rapids family of Intel CPUs will not be available to the VMs? > > Regards, > _______________________________________________ > Users mailing list -- users(a)ovirt.org<mailto:users@ovirt.org> > To unsubscribe send an email to users-leave(a)ovirt.org<mailto:users-leave@ovirt.org> > Privacy Statement: https://urldefense.com/v3/__https://www.ovirt.org/privacy-policy.html__;!!A… > oVirt Code of Conduct: https://urldefense.com/v3/__https://www.ovirt.org/community/about/community… > List Archives: https://urldefense.com/v3/__https://lists.ovirt.org/archives/list/users@ovi…

1 0

OLVM/ovirt 4.5 support for 5th Gen Intel Xeon Gold (Emerald Rapids)
by hshabbir315＠gmail.com 23 Jan '25

23 Jan '25

Hi Guys, Can someone please confirm if 5th Gen Intel Gold (Emerald Rapids) CPUs are compatible with OLVM/ovirt 4.5.5.x? We have a OLVM/ovirt 4.5.5.x test lab setup cluster running on older hardware (Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz - Haswell Family) with compatibility version 4.7. While going through the engine db backup we observed that there are no entries in the vdc_options table for Emerald Rapids family of Intel CPUs. Does this mean that these CPUs are not supported by ovirt 4.5.5.x? or these CPUs are supported but the extra features/instruction sets offered by these CPUs over Sapphire Rapids family of Intel CPUs will not be available to the VMs? Regards,

2 1

Re: [External] : The KVM Host Status Stucking in Connecting
by natchawi28＠gmail.com 22 Jan '25

22 Jan '25

Hi, What about the version differences between the Engine and the KVM host? Does it matter? Engine: Oracle Linux 8.10 Kernel 5.15.0-302.167.6.el8uek.x86_64 KVM Host: Oracle Linux 8.6 Kernel 5.4.17-2136.311.6.1.el8uek.x86_64 I checked the Cluster Compatibility Version, and it appears to be compatible. Thank you,

2 1

ovirtmgmt claims no defroute after RHEL upgrade from 8.10 to 9.5
by Austin Coppock 22 Jan '25

22 Jan '25

First off apologies, I am unable to post log files as my oVirt servers have no access to the Internet. I have several oVirt clusters, all were running oVirt 4.5 on RHEL 8.10. I am in the process of upgrading them to RHEL 9.5. with a few minor updates of oVirt packages. The steps I used to do this are: • Backup up the standalone engine DB • Rebuild the engine to RHEL 9.5, reinstall oVirt, restore the DB • Log back in to the engine, place first hypervisor into maintenance and rebuild to RHEL 9.5. • Remove old hypervisor via the engine GUI and add back in to the cluster • Repeat for each hypervisor until all are upgraded. This has worked perfectly, everything seems to be normal, but I then noticed that if I select the “Network Interfaces” tab on a hypervisor it shows my bonded interface as “Out-of-sync”, clicking the “Sync All Networks” does nothing. If I click on “Setup Host Networks” and hover the mouse over the ovirtmgmt interface I see the message “Default Route. Host: false DC: true” If I run the command “vdsm-client Host getNetworkCapabilities” on the hypervisor, it shows that the {bridges: ovirtmgmt: ipv4defaultroute false} and {networks: ovirtmgmt: ipv4defaultroute false } on my clusters that are still on RHEL 8.10 the above command shows ipv4defaultroute true. Running the command “ip route show” shows that my default route IS on device ovirtmgmt I am not experiencing any network issues, but getting rid of this out-of-sync message would be nice. Is this a bug or a bad status issue? Or do I have some obscure setting not correct? For both RHEL 8.10 and RHEL 9.5 builds we have used Ansible to perform the tasks, (with minor tweaks). So, the same procedure has been used for both.

1 1

The KVM Host Status Stucking in Connecting
by natchawi28＠gmail.com 21 Jan '25

21 Jan '25

Please help me, I found that my Cluster is in Non-responsive status. When I look at the Host menu page, I see that the status is Connecting and it has been stuck for a long time and I can't do anything. Has anyone experienced this and how do I fix it? https://img5.pic.in.th/file/secure-sv1/Screenshot-2025-01-21-151647.png

2 1

additional language
by ebugaev＠itkey.com 20 Jan '25

20 Jan '25

Are there instructions on how to add an additional language to ovirt-engine? What needs to be done so that an additional language appears in the language selection list on the main page of the engine?

1 0

Renewal of VDSM certificate in one node cluster
by ikbal.charen＠kepler-technologies.com 20 Jan '25

20 Jan '25

Hello, I have a single-node cluster running Hosted Engine, version 4.4.10. While I'm able to renew the engine certificate, I'm unable to renew the VDSM certificate because I cannot put the host into maintenance mode (since it is running the Hosted Engine VM). What steps can I take to resolve this? Best regards,

1 0

Host reboot take an awful lot of time
by duparchy＠esrf.fr 20 Jan '25

20 Jan '25

KVM host is upgraded then rebooted. It seems OK (ssh available, no particular activity) but it takes an awful lot of time for the manager to acknowledge it and let us activating the host. This makes the upgrade process of a whole infrastructure (several hosts) very very long. Is it just us ? (oVirt 4.5 "Oracle" flavor)

3 2

Help With CPU Type
by Daniel Martins 20 Jan '25

20 Jan '25

Hi There, Please may you assist me with the below? I need to remove an oVirt host within a cluster that is using CPU Type Intel Westmere IBRS SSBD Family. Once the host is removed i need to change the CPU type to Intel Broadwell which the remaining hosts Use. So my question is will there need to be down time on the Virtual machines to change this? Regards, [cid:image001.png@01DB6B29.AA668C00]

1 0

Re: [External] : Really weird ! Image IO / browser upload only working on 1 cluster (out of many) - anyone got any advice ? (network access, certs seem fine.)
by morgan cox 17 Jan '25

17 Jan '25

Hi Marcos. Thanks for taking the time to respond... In reference to your questions : Are you using a single Engine deployment to manage all DCs/Clusters : Yes Which browser are you using to upload the image : I have tried 2 Firefox and Chrome (note it works on one cluster - the one that hosts1/2 are connected to) Can you try the upload process from a different browser or desktop/laptop?: Tired a different browser, cannot try different laptop - I have network access to the engine . Also - everything else seems ok - in NoVNC, Serial , etc just image io Any suggestions where to look into this ? Thanks

3 2

Intel CPUs supported by OLVM/ovirt 4.5
by Syed Hassan Shabbir 17 Jan '25

17 Jan '25

Hi, Guys, can someone plz guide me regarding ovirt support for the following CPUs: 1- Intel® Xeon® Gold 5520+ Processor 52.5M Cache, 2.20 GHz (Emerald Rapids) 2- Intel® Xeon® Gold 6526Y Processor 37.5M Cache, 2.80 GHz (Emerald Rapids) 3- Intel® Xeon® Gold 6426Y Processor 37.5M Cache, 2.50 GHz (Sapphire Rapids) 4- Intel® Xeon® Gold 5420+ Processor 52.5M Cache, 2.00 GHz (Sapphire Rapids) 5- Intel® Xeon® Silver 4510 Processor 30M Cache, 2.40 GHz (Sapphire Rapids) I want to deploy OLVM on these servers but couldn't find any entries in the vdc_options table for 5th Gen 'Emerald Rapids'. I am testing with ovirt version 4.5. Regards, Hassan

1 0

Welcome Jean-Louis Dupond to the oVirt Board
by Sandro Bonazzola 17 Jan '25

17 Jan '25

On behalf of the oVirt Board I’m thrilled to announce an exciting new addition: Jean-Louis Dupond. Jean-Louis brings a wealth of experience, expertise, and passion for open-source virtualization, making him an exceptional fit for the oVirt community. Read the full announce on the oVirt blog at https://blogs.ovirt.org/2025/01/welcome-jean-louis-dupond-to-the-ovirt-boar… -- Sandro Bonazzola

4 3

Managing Users with Keycloak WebUI
by Vladislav Solovei 16 Jan '25

16 Jan '25

Hello! I have a question about managing users through the Keycloak WebUI. With oVirt 4.5. Can I do this? Is it the right approach? Or should I avoid doing it? For example, if I want to create a user with 2FA (OTP)

1 0

Really weird ! Image IO / browser upload only working on 1 cluster (out of many) - anyone got any advice ? (network access, certs seem fine.)
by morgan cox 16 Jan '25

16 Jan '25

One for the mystery van .. I am running OLVM/ Ovirt 4.5 (engine) - hosts presently mised 4.4.x and 4.5.x - however version doesn't seen to be the issue We have muliple DC's/Clusters each cluster has 2 hosts and then have access to different fc SAN volumes - each san vol is a sperate storaqe domain I,e host 1 + host 2 --> San 1 host 3 + host 4 --> San 2 , etc SAN connection for the hosts/clusters generally is working fine and I can also upload to any SAN volume using the upload script - /usr/share/doc/python3-ovirt-engine-sdk4/examples/upload_disk.py However imageio via browser ... We have a major issue. I am not able to upload (via browser) any files to any storage domain apart from 1... (say San 1) Yes, I have imported the CA to browser, and All storage domains pass the 'test connection' -> "Connection to ovirt-imageio was successful." However I can see what the issue is (but not why there is this issue..) In the upload image section of all domains the available host to use (to piggyback connection) as the same - host 1 + host 2 But those hosts only have access to one SAN not all of them i.e for San 2 I should be able to choose host 3 + 4 ... (but only host 1/2 are showning in all domains..) If I use the upload script it uses correct host and successfully upload - but not via browser.. I have checked and /etc/ovirt-imageio/conf.d/60-vdsm.conf is the same for ALL hosts. Certs seem ok all a I can telnet to all hosts from engine on port 54322 Can anyone give me any ideas how to troubleshoot this ? Thanks

2 1

EOL glusterFS
by Olivier 16 Jan '25

16 Jan '25

Hello, GlusterFS seems to be EOL. At least no more dev from Red Hat, the github doesn't record any move from more than one year. Is there a plan to use another FS on oVirt instead of glusterFS ?

9 9

zanata.ovirt.org
by ebugaev＠itkey.com 15 Jan '25

15 Jan '25

hello, I would like to translate into Russian help me get an account https://zanata.ovirt.org/ to translate the interface, thank you)

3 3

Changing Disk Alocation Policy
by suporte＠logicworks.pt 13 Jan '25

13 Jan '25

Hi, It's possible to change a thin-provision disk to a preallocated disk? Storages are ISCSI Thanks -- Jose Ferradeira http://www.logicworks.pt

4 4

Re: [External] : Changing Disk Alocation Policy
by change_jeeringly679＠dralias.com 13 Jan '25

13 Jan '25

Hey, Jose is right, you can do this via the api. The server needs to be shutdown. I usually use the below curl command on the engine itself (login with ssh). You need 3 pieces of information: - You need the ce.pem of the engine (which you can download from the engine login page) and make it available. In the command below I assume it is in the local directory. - You need an administrator user in engine, in the below example I used admin. Replace "yourpassword" behind the : with your own. - You need the disk-id as it is know in oVirt. You need to replace "yourdiskid", with your own. You can find the id in Engine. Select Storage, then Disks, then find the disk you want to convert and click on it. The number behind ID is the one you want and has the following format: db7b649f-76b6-4fd1-9b90-aa6becc2a283 The command: curl --cacert './ca.pem' --user 'admin@internal:yourpassword!' --request POST --header 'Version: 4' --header 'Content-Type: application/xml' --header 'Accept: application/xml' --data ' <action> <disk> <sparse>true</sparse> <format>cow</format> <backup_mode>incremental</backup_mode> </disk> </action> ' https://engine.blueit.dk/ovirt-engine/api/disks/yourdiskid/convert The command will return immediately. In engine you will see it is working on the disk image. You get the ballon messages in the right bottom corner and you can see it working in the taskview on the webinterface. Once it is done, you can start the VM again, and the disk is now thin provisioned and incremental backups are allowed. Good luck

1 0

CentOS Stream 9 Ovirt 4.5
by eevans＠digitaldatatechs.com 13 Jan '25

13 Jan '25

I have a small lab set up with 3 ovirt Centos 9 nodes and one Centos 9 Ovirt "controller node" that manages. 2 problems: It will no sync networks on hosts. It will no launch a vm unless you go to system and choose an instance size. It will boot and the network card has to be hot added. I'm not sure what I'm missing, I can give log entries showing several bad lines in diggerent modules.' Engine log while updating hosts: tail -n 100 -f engine.log ' 2025-01-09 02:43:15,722-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName = kvm02 2025-01-09 02:43:15,722-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm02, VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2025-01-09 02:43:15,729-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'}), log id: 5e145d4b 2025-01-09 02:43:15,729-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:18,057-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:18,057-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName = kvm04 2025-01-09 02:43:18,057-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:18,057-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 5e145d4b 2025-01-09 02:43:18,057-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:22,344-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'SpmStatusVDSCommand(HostName = kvm04, SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:22,344-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Irs placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover 2025-01-09 02:43:22,349-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'}), log id: 28df8c85 2025-01-09 02:43:22,349-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:25,267-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm01 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:25,267-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:25,267-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName = kvm01 2025-01-09 02:43:25,267-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:25,267-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 28df8c85 2025-01-09 02:43:25,267-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:25,275-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Irs placed on server 'acf66bf6-e159-4705-be47-484c99e746dd' failed. Proceed Failover 2025-01-09 02:43:25,279-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'}), log id: 20ec709f 2025-01-09 02:43:25,279-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:27,675-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm02 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:27,675-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:27,675-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] HostName = kvm02 2025-01-09 02:43:27,676-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='765'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:27,676-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 20ec709f 2025-01-09 02:43:27,676-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:27,741-05 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-572) [6727848e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage 2025-01-09 02:43:27,765-05 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-572) [6727848e] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2025-01-09 02:43:27,789-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (EE-ManagedThreadFactory-engine-Thread-571) [] Finished reconstruct for pool '39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue 2025-01-09 02:43:30,154-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] START, GlusterServersListVDSCommand(HostName = kvm04, VdsIdVDSCommandParametersBase:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 770c0b37 2025-01-09 02:43:30,362-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] FINISH, GlusterServersListVDSCommand, return: [192.168.254.108/24:CONNECTED, ovirt.digitaldatatechs.com:CONNECTED, kvm02.digitaldatatechs.com:CONNECTED, kvm01.digitaldatatechs.com:CONNECTED], log id: 770c0b37 2025-01-09 02:43:30,381-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] START, AddGlusterServerVDSCommand(HostName = kvm04, AddGlusterServerVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 3b2ce207 2025-01-09 02:43:33,630-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] FINISH, AddGlusterServerVDSCommand, return: , log id: 3b2ce207 2025-01-09 02:43:33,640-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] EVENT_ID: GLUSTER_SERVER_ADD_FAILED(4,436), Failed to add host kvm04 into Cluster dds_cluster. Add host failed: rc=107 out=() err=['Probe returned with Transport endpoint is not connected'] 2025-01-09 02:43:33,641-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] START, GlusterVolumesListVDSCommand(HostName = kvm04, GlusterVolumesListVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 6a99b1ab 2025-01-09 02:43:33,748-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] FINISH, GlusterVolumesListVDSCommand, return: {b79a00d7-cc2a-48d8-b881-699991f3d930=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@205714d2}, log id: 6a99b1ab 2025-01-09 02:43:37,689-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'SpmStatusVDSCommand(HostName = kvm02, SpmStatusVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:37,691-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' TaskStatusListReturn:{status='Status [code=654, message=Not SPM: ()]'} ' 2025-01-09 02:43:37,691-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName = kvm02 2025-01-09 02:43:37,691-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm02, VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2025-01-09 02:43:37,698-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'}), log id: 201774c9 2025-01-09 02:43:37,698-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:38,381-05 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-10) [] There is no host with more than 10 running guests, no balancing is needed 2025-01-09 02:43:38,400-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-59) [] START, GlusterTasksListVDSCommand(HostName = kvm02, VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'}), log id: 702ab66f 2025-01-09 02:43:38,515-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-59) [] FINISH, GlusterTasksListVDSCommand, return: [], log id: 702ab66f 2025-01-09 02:43:40,598-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:40,598-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName = kvm01 2025-01-09 02:43:40,598-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:40,598-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 201774c9 2025-01-09 02:43:40,598-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:40,606-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Irs placed on server 'acf66bf6-e159-4705-be47-484c99e746dd' failed. Proceed Failover 2025-01-09 02:43:40,611-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'}), log id: 164d36cb 2025-01-09 02:43:40,611-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:42,950-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm04 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:42,950-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:42,950-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName = kvm04 2025-01-09 02:43:42,950-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:42,950-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 164d36cb 2025-01-09 02:43:42,950-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:42,960-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'SpmStatusVDSCommand(HostName = kvm04, SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:42,960-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Irs placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover 2025-01-09 02:43:42,964-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'}), log id: 863c329 2025-01-09 02:43:42,965-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:45,328-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:45,328-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] HostName = kvm02 2025-01-09 02:43:45,328-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='766'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:45,328-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 863c329 2025-01-09 02:43:45,328-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:45,388-05 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-580) [2e23e34] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage 2025-01-09 02:43:45,412-05 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-580) [2e23e34] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2025-01-09 02:43:45,436-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (EE-ManagedThreadFactory-engine-Thread-579) [] Finished reconstruct for pool '39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue 2025-01-09 02:43:48,763-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] START, GlusterServersListVDSCommand(HostName = kvm04, VdsIdVDSCommandParametersBase:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: db1640e 2025-01-09 02:43:48,970-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] FINISH, GlusterServersListVDSCommand, return: [192.168.254.108/24:CONNECTED, ovirt.digitaldatatechs.com:CONNECTED, kvm02.digitaldatatechs.com:CONNECTED, kvm01.digitaldatatechs.com:CONNECTED], log id: db1640e 2025-01-09 02:43:48,983-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] START, AddGlusterServerVDSCommand(HostName = kvm04, AddGlusterServerVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 717f84ae 2025-01-09 02:43:52,255-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] FINISH, AddGlusterServerVDSCommand, return: , log id: 717f84ae 2025-01-09 02:43:52,277-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] EVENT_ID: GLUSTER_SERVER_ADD_FAILED(4,436), Failed to add host kvm04 into Cluster dds_cluster. Add host failed: rc=107 out=() err=['Probe returned with Transport endpoint is not connected'] 2025-01-09 02:43:52,278-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] START, GlusterVolumesListVDSCommand(HostName = kvm04, GlusterVolumesListVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 6a43f9e6 2025-01-09 02:43:52,384-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-96) [] FINISH, GlusterVolumesListVDSCommand, return: {b79a00d7-cc2a-48d8-b881-699991f3d930=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@205714d2}, log id: 6a43f9e6 2025-01-09 02:43:55,341-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'SpmStatusVDSCommand(HostName = kvm02, SpmStatusVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:55,343-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' TaskStatusListReturn:{status='Status [code=654, message=Not SPM: ()]'} ' 2025-01-09 02:43:55,343-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName = kvm02 2025-01-09 02:43:55,343-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm02, VdsIdVDSCommandParametersBase:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2025-01-09 02:43:55,350-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'}), log id: 35cf220b 2025-01-09 02:43:55,350-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:43:57,661-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:43:57,661-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName = kvm04 2025-01-09 02:43:57,661-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:57,661-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 35cf220b 2025-01-09 02:43:57,661-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:43:57,671-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'SpmStatusVDSCommand(HostName = kvm04, SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:43:57,671-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Irs placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover 2025-01-09 02:43:57,675-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'}), log id: 2ff24c19 2025-01-09 02:43:57,676-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:00,065-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm02 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:00,065-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:00,065-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName = kvm02 2025-01-09 02:44:00,065-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:00,065-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 2ff24c19 2025-01-09 02:44:00,065-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:04,367-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'SpmStatusVDSCommand(HostName = kvm02, SpmStatusVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:44:04,367-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Irs placed on server 'cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6' failed. Proceed Failover 2025-01-09 02:44:04,372-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'}), log id: 23ed7788 2025-01-09 02:44:04,372-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:07,292-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm01 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:07,292-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:07,292-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] HostName = kvm01 2025-01-09 02:44:07,292-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='767'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:07,292-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 23ed7788 2025-01-09 02:44:07,292-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-40) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:07,375-05 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-594) [6e7f60bb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage 2025-01-09 02:44:07,412-05 INFO [org.ovirt.engine.core.bll.network.host.SyncAllHostNetworksCommand] (default task-7) [9913b205-9b2e-4f8a-86e3-048a9cdff2ba] Running command: SyncAllHostNetworksCommand internal: false. Entities affected : ID: acf66bf6-e159-4705-be47-484c99e746dd Type: VDSAction group CONFIGURE_HOST_NETWORK with role type ADMIN 2025-01-09 02:44:07,421-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, GlusterServersListVDSCommand(HostName = kvm04, VdsIdVDSCommandParametersBase:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 340c8f9d 2025-01-09 02:44:07,423-05 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-594) [6e7f60bb] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2025-01-09 02:44:07,423-05 INFO [org.ovirt.engine.core.bll.network.host.PersistentHostSetupNetworksCommand] (default task-7) [7814c7ae] Running command: PersistentHostSetupNetworksCommand internal: true. Entities affected : ID: acf66bf6-e159-4705-be47-484c99e746dd Type: VDS 2025-01-09 02:44:07,448-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (EE-ManagedThreadFactory-engine-Thread-593) [] Finished reconstruct for pool '39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue 2025-01-09 02:44:07,460-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [7814c7ae] EVENT_ID: PERSIST_HOST_SETUP_NETWORK_ON_HOST(10,902), (1/1): Applying network's changes on host kvm01. (User: admin@ovirt@internalkeycloak-authz) 2025-01-09 02:44:07,461-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Before acquiring lock-timeout 'EngineLock:{exclusiveLocks='[HOST_NETWORKacf66bf6-e159-4705-be47-484c99e746dd=HOST_NETWORK]', sharedLocks=''}' 2025-01-09 02:44:07,461-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Lock-timeout acquired to object 'EngineLock:{exclusiveLocks='[HOST_NETWORKacf66bf6-e159-4705-be47-484c99e746dd=HOST_NETWORK]', sharedLocks=''}' 2025-01-09 02:44:07,497-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Running command: HostSetupNetworksCommand internal: true. Entities affected : ID: acf66bf6-e159-4705-be47-484c99e746dd Type: VDSAction group CONFIGURE_HOST_NETWORK with role type ADMIN 2025-01-09 02:44:07,497-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Before acquiring lock in order to prevent monitoring for host 'kvm01' from data-center 'DDS_Data_Center' 2025-01-09 02:44:07,497-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Lock acquired, from now a monitoring of host will be skipped for host 'kvm01' from data-center 'DDS_Data_Center' 2025-01-09 02:44:07,498-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-7) [7814c7ae] START, HostSetupNetworksVDSCommand(HostName = kvm01, HostSetupNetworksVdsCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vds='Host[kvm01,acf66bf6-e159-4705-be47-484c99e746dd]', rollbackOnFailure='true', commitOnSuccess='true', connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='false', bonding='false', networkName='vlan3', vdsmName='vlan3', nicName='eno2', vlan='3', vmNetwork='true', stp='false', properties='[]', ipv4BootProtocol='STATIC_IP', ipv4Address='172.50.26.252', ipv4Netmask='255.255.255.0', ipv4Gateway='172.50.26.1', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}, HostNetwork:{defaultRoute='true', bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt', nicName='eno1', vlan='null', vmNetwork='true', stp='false', properties='[]', ipv4BootProtocol='STAT IC_IP', ipv4Address='192.168.254.252', ipv4Netmask='255.255.255.0', ipv4Gateway='192.168.254.1', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='[NameServer:{address='192.168.254.222'}, NameServer:{address='192.168.254.232'}]'}]', removedNetworks='[]', bonds='[]', removedBonds='[]', clusterSwitchType='LEGACY', managementNetworkChanged='true'}), log id: 56a27c18 2025-01-09 02:44:07,499-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-7) [7814c7ae] FINISH, HostSetupNetworksVDSCommand, return: , log id: 56a27c18 2025-01-09 02:44:07,631-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, GlusterServersListVDSCommand, return: [192.168.254.108/24:CONNECTED, ovirt.digitaldatatechs.com:CONNECTED, kvm02.digitaldatatechs.com:CONNECTED, kvm01.digitaldatatechs.com:CONNECTED], log id: 340c8f9d 2025-01-09 02:44:07,642-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, AddGlusterServerVDSCommand(HostName = kvm04, AddGlusterServerVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: 7c25cce3 2025-01-09 02:44:10,879-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.AddGlusterServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, AddGlusterServerVDSCommand, return: , log id: 7c25cce3 2025-01-09 02:44:10,923-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] EVENT_ID: GLUSTER_SERVER_ADD_FAILED(4,436), Failed to add host kvm04 into Cluster dds_cluster. Add host failed: rc=107 out=() err=['Probe returned with Transport endpoint is not connected'] 2025-01-09 02:44:10,923-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, GlusterVolumesListVDSCommand(HostName = kvm04, GlusterVolumesListVDSParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20'}), log id: effd1e8 2025-01-09 02:44:11,033-05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, GlusterVolumesListVDSCommand, return: {b79a00d7-cc2a-48d8-b881-699991f3d930=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@205714d2}, log id: effd1e8 2025-01-09 02:44:17,314-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' TaskStatusListReturn:{status='Status [code=654, message=Not SPM: ()]'} ' 2025-01-09 02:44:17,315-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName = kvm01 2025-01-09 02:44:17,315-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = kvm01, VdsIdVDSCommandParametersBase:{hostId='acf66bf6-e159-4705-be47-484c99e746dd'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2025-01-09 02:44:17,331-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] START, ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'}), log id: bcbf59e 2025-01-09 02:44:17,331-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:18,272-05 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [7814c7ae] EVENT_ID: VDS_NETWORKS_OUT_OF_SYNC(1,110), Host kvm01's following network(s) are not synchronized with their Logical Network configuration: ovirtmgmt,vlan3. 2025-01-09 02:44:18,283-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Host setup networks finished. Lock released. Monitoring can run now for host 'kvm01' from data-center 'DDS_Data_Center' 2025-01-09 02:44:18,285-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-7) [7814c7ae] Lock freed to object 'EngineLock:{exclusiveLocks='[HOST_NETWORKacf66bf6-e159-4705-be47-484c99e746dd=HOST_NETWORK]', sharedLocks=''}' 2025-01-09 02:44:18,305-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [7814c7ae] EVENT_ID: PERSIST_SETUP_NETWORK_ON_HOST_FINISHED(10,903), (1/1): Successfully applied changes on host kvm01. (User: admin@ovirt@internalkeycloak-authz) 2025-01-09 02:44:18,316-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [7814c7ae] EVENT_ID: HOST_SYNC_ALL_NETWORKS_FINISHED(10,901), Managed to sync all host kvm01 networks. 2025-01-09 02:44:19,668-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm04 command ConnectStoragePoolVDS failed: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:19,668-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:19,668-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName = kvm04 2025-01-09 02:44:19,668-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm04, ConnectStoragePoolVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', vdsId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:19,668-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: bcbf59e 2025-01-09 02:44:19,668-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:19,679-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'SpmStatusVDSCommand(HostName = kvm04, SpmStatusVDSCommandParameters:{hostId='1408650c-aa50-452a-b2cc-24d36ae3cc20', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2025-01-09 02:44:19,679-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Irs placed on server '1408650c-aa50-452a-b2cc-24d36ae3cc20' failed. Proceed Failover 2025-01-09 02:44:19,684-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] START, ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'}), log id: 251dbd22 2025-01-09 02:44:19,684-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:22,181-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:22,181-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName = kvm01 2025-01-09 02:44:22,181-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm01, ConnectStoragePoolVDSCommandParameters:{hostId='acf66bf6-e159-4705-be47-484c99e746dd', vdsId='acf66bf6-e159-4705-be47-484c99e746dd', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:22,181-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 251dbd22 2025-01-09 02:44:22,181-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:22,189-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Irs placed on server 'acf66bf6-e159-4705-be47-484c99e746dd' failed. Proceed Failover 2025-01-09 02:44:22,198-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] START, ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'}), log id: 1f60ebb1 2025-01-09 02:44:22,198-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Executing with domain map: {bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94=active, ca1b2f4f-26cf-48cc-b219-7295a763b1ae=active, 028f24af-01a2-4c0d-9374-7d8be6e12469=active} 2025-01-09 02:44:24,581-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94']]' 2025-01-09 02:44:24,581-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] HostName = kvm02 2025-01-09 02:44:24,581-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Command 'ConnectStoragePoolVDSCommand(HostName = kvm02, ConnectStoragePoolVDSCommandParameters:{hostId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', vdsId='cf4a626e-ad6e-4cc4-95f0-4d7c1329f0f6', storagePoolId='39acc250-bfba-11ef-adb1-70e2840574e3', masterVersion='768'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:24,582-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 1f60ebb1 2025-01-09 02:44:24,582-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=39acc250-bfba-11ef-adb1-70e2840574e3, msdUUID=bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94' 2025-01-09 02:44:24,658-05 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-610) [2af75e62] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: bd71c70a-7d9b-4cb4-ae5d-4c75a96a1d94 Type: Storage 2025-01-09 02:44:24,691-05 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (EE-ManagedThreadFactory-engine-Thread-610) [2af75e62] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2025-01-09 02:44:24,715-05 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (EE-ManagedThreadFactory-engine-Thread-609) [] Finished reconstruct for pool '39acc250-bfba-11ef-adb1-70e2840574e3'. Clearing event queue While attempting to start vm will be sent after but I think the net allocation issue with hosts is the center of the vm issues,,

2 1

Restart oVirt-Engine
by Jeremey Wise 09 Jan '25

09 Jan '25

How ,without reboot of hosting system, do I restart the oVirt engine? # I tried below but do not seem to effect the virtual machine [root@thor iso]# systemctl restart ov ovirt-ha-agent.service ovirt-imageio.service ovn-controller.service ovs-delete-transient-ports.service ovirt-ha-broker.service ovirt-vmconsole-host-sshd.service ovsdb-server.service ovs-vswitchd.service [root@thor iso]# # You cannot restart the VM " HostedEngine " as it responses: Error while executing action: HostedEngine: - Cannot restart VM. This VM is not managed by the engine. Reason is I had to do some work on a node. Reboot it.. it is back up.. network is all fine.. Cockpit working fine... and gluster fine.. But oVirt-Engine refuses to accept the node is up. -- p <jeremey.wise(a)gmail.com>enguinpages

4 4

Host deploy failure: Configure OVN for oVirt
by stephan.badenhorst＠fnb.co.za 05 Jan '25

05 Jan '25

Good day, I am running into a problem during host deploy via the oVirt engine GUI since upgrading to ovirt-engine-4.5.5-1.el8. The "Configure OVN for oVirt" task seem to fail when trying to run the vdsm-tool ovn-config command. Host deploy used to work fine when the engine was on version 4.5.4. Anyone that can guide me on the right path to get past this issue? Does not seem to be a new problem - https://lists.ovirt.org/archives/list/users@ovirt.org/thread/IDLGSBQFX35EHH… Log extract: 2024-02-01 14:48:19 SAST - TASK [ovirt-provider-ovn-driver : Configure OVN for oVirt] ********************* . . . "stdout" : "fatal: [mob-r1-l-ovirt-aa-1-23.x.fnb.co.za]: FAILED! => {\"changed\": true, \"cmd\": [\"vdsm-tool\", \"ovn-config\", \"192.168.2.100\", \"host23.mydomain.com\"], \"delta\": \"0:00:00.538143\", \"end \": \"2024-02-01 14:48:20.596823\", \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2024-02-01 14:48:20.058680\", \"stderr\": \"Traceback (most recent call last):\\n File \\\"/usr/lib/python3.6/site-packages/vdsm/t ool/ovn_config.py\\\", line 117, in get_network\\n return networks[net_name]\\nKeyError: 'host23.mydomain.com'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most rec ent call last):\\n File \\\"/usr/bin/vdsm-tool\\\", line 195, in main\\n return tool_command[cmd][\\\"command\\\"](*args)\\n File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\\n ip_address = get_ip_addr(get_network(network_caps(), net_name))\\n File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 119, in get_network\\n raise NetworkNotFoundError(net_name)\\nvdsm.tool.ovn _config.NetworkNotFoundError: host23.mydomain.com\", \"stderr_lines\": [\"Traceback (most recent call last):\", \" File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 117, in get_network \", \" return networks[net_name]\", \"KeyError: 'host23.mydomain.com'\", \"\", \"During handling of the above exception, another exception occurred:\", \"\", \"Traceback (most recent call last):\", \" File \ \\"/usr/bin/vdsm-tool\\\", line 195, in main\", \" return tool_command[cmd][\\\"command\\\"](*args)\", \" File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\", \" ip_address = get_ip_addr(get_network(network_caps(), net_name))\", \" File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 119, in get_network\", \" raise NetworkNotFoundError(net_name)\", \"vdsm.tool.ovn_config. NetworkNotFoundError: host23.mydomain.com\"], \"stdout\": \"\", \"stdout_lines\": []}", Thanks in advance!! Stephan

5 5

Re: Host deploy failure: Configure OVN for oVirt
by Andrea Cavegn 03 Jan '25

03 Jan '25

We use ovirt-engine 4.5.5-1.el8 and for us the following did the trick: 1. First you have to decide, if you need the ovn/ovs functionality. This is a cluster level decision! * If needed, you need to have a functional (You can test it) external network provider configured (Administration -> Providers -> External Network Provider) ?? As far as I know, a default "ovirt-provider-ovn" is already present in each installation * Then in the cluster option you choose the "Default Network Provider". If you don't need OVS/OVN, then select "No Default Provider", else select the one you need * If you change here anything, ovirt want you to reinstall all your hosts in this cluster! 2. Then, replace the ansible file in your ovirt-engine: * /usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-provider-ovn-driver/tasks/configure.yml * The new version can be found here: https://github.com/oVirt/ovirt-engine/blob/master/packaging/ansible-runner-… as far as I can tell, only the last line (when statement) changes Probably, only ppl who don't need ovn/ovs had (Setting: "No Default Provider") had this problem. 1. Reinstall the hosts As always, know what you do, before you anything.

1 0

Enroll Certificate
by Денис Панкратьев 03 Jan '25

03 Jan '25

Hello! My ovirt host (4.5.5) certificate is about to expire (in 46 days, Jan 20, 2025). d.pankratiev@zabbix ~]$ echo | openssl s_client -servername ovirt1.gtc.local -connect XXX.XXX.XXX.XXX.XXX:9090 2>/dev/null | openssl x509 -noout -dates 2>/dev/null notBefore=Dec 22 05:27:04 2023 GMT notAfter=Jan 20 05:27:04 2025 GMT 1. I have migrated all VMs to another host 2. I put the host in maintenance mode (Managеment -> Maintenance) 3. Do enroll certificate (Installation -> Enroll Certificate) 4. Everything went successfully, without errors (Enrolling certificate for host ovirt1.gtc.local was completed successfully). But the certificate date did not change, still expire (after 46 days, Jan 20 2025). d.pankratiev@zabbix ~]$ echo | openssl s_client -servername ovirt1.gtc.local -connect XXX.XXX.XXX.XXX.XXX:9090 2>/dev/null | openssl x509 -noout -dates 2>/dev/null notBefore=Dec 22 05:27:04 2023 GMT notAfter=Jan 20 05:27:04 2025 GMT What did I do wrong? Is there anything else I need to do?

5 5