February 2024 - Users - oVirt List Archives

Ovirt 4.5 HA over NFS fails when a single host goes down
by youssef.khristo＠elsewedyintel.com 12 Mar '24

12 Mar '24

Greetings, we have recently installed ovirt as a hosted-engine with high availability on six nodes over NFS storage (no Gluster), with power management through an on-board IPMI device, and the setup was successful. All the nodes (from Supermicro) are identical in every aspect, so no hardware differences exist and no modifications to the servers' hardware were performed. The hosted-engine was deployed on a second host, where two of the six hosts only were required to host the HE VM. The network interface on each node is bonded between two physical fiber optics NICs in LACP mode with a VLAN on top, serving as the sole networking interface for the server/node, no separate VM or storage networks were needed, as the host OS, hosted-engine vm, and storage are required to be on the same network and VLAN. We started by testing the high-availability of the hosted-engine VM (as it was deployed on two of the six nodes) by rebooting or powering off one of the hosts, and the VM would migrate successfully to the second HE node. The main goal of our experiments is to test the robustness of the setup, as it is required for the cluster to remain functional even when up to two hosts are brought down (whether due to a network or power issue), however, when rebooting or powering off one of the hosts, the HE VM goes down and takes the entire cluster with it, where we can't even access the web portal. Once the host is rebooted, the HE VM and the cluster becomes functional again. Sometimes the HE VM stays down for a set amount of time (5 to 6 minutes) and then goes back up, and sometimes it goes down until the problematic host is back up. This behavior happens to other VMs as well not the the HE. We suspected an issue with the NFS storage, however, during ovirt operation it is being mounted properly over /rhev/data-center/mnt/<nfs:directory>, while the expected behavior is for the cluster to stay operational and any other VMs to be migrated to other hosts. During one of the tests, we tried to mount the NFS storage on a different directory and there was no problem, we were even able to perform commands such as ls without any issues, as well as writing a text file at the directory's root, and be able to modify it normally. We suspected a couple of things the first being that the HE is unable to fence the problematic host (the one we took down), however, power management is setup properly. The other thing we suspected is the cluster hosts (after taking down one of them) are unable to acquire storage lease, which is weird since the host in question is down and non-operational, hence no locks should be in place. The reason behind this suspicion is the following two errors that we receive frequently when one host or more goes down from the engine\ovirt-engine\engine.log file: 1- "EVENT_ID: VM_DOWN_ERROR(119), VM HostedEngine is down with error. Exit message: resource busy: Failed to acquire lock: Lease is held by another host." 2- "[<id>] Command 'GetVmLeaseInfoVDSCommand( VmLeaseVDSParameters:{expectedEngineErrors='[NoSuchVmLeaseOnDomain]', storagePoolId='<pool-id>', ignoreFailoverLimit='false', leaseId='<lease-id>', storageDomainId='<domain-id>'})' execution failed: IRSGenericException: IRSErrorException: No such lease: 'lease=<lease-id>'" This is a third warning from the /var/log/vdsm/vdsm.log file 1- "WARN (check/loop) [storage.check] Checker '/rhev/data-center/mnt/<nfs-domain:/directory>/<id>/dom_md/metadata' is blocked for 310.00 seconds (check:265)" All the tests are done without setting nodes into maintenance mode as we are simulating an emergency situation. No HE configuration were modified via the config-engine command, the default values are used. Is this a normal behavior? Are we missing something? Do we need to tweak a certain configuration using the config-engine command to get a better behavior (e.g., shorter down period)? Best regards

5 6

Repo for mirrorlist.ovirt.org not working
by nicolas＠devels.es 01 Mar '24

01 Mar '24

Hi, We recently noticed that https://mirrorlist.ovirt.org/mirrorlist-ovirt-4.5-el8 is returning a 503 error: $ LANG=C wget https://mirrorlist.ovirt.org/mirrorlist-ovirt-4.5-el8 --2024-02-14 12:01:10-- https://mirrorlist.ovirt.org/mirrorlist-ovirt-4.5-el8 Resolving mirrorlist.ovirt.org (mirrorlist.ovirt.org) 8.43.85.224, 2620:52:3:1:5054:ff:fef7:14a9 Connecting to mirrorlist.ovirt.org (mirrorlist.ovirt.org)|8.43.85.224|:443... connected. HTTP request sent, awaiting response... 503 Service Unavailable 2024-02-14 12:02:10 ERROR 503: Service Unavailable. Is this some scheduled downtime or is it an error? Thanks.

7 6

Advice on upgrading oVirt 4.2 CentOS 7.x cluster to oVirt 4.5 Rocky Linux 8.9
by ovirt＠the-dawg.net 29 Feb '24

29 Feb '24

I have multiple aging clusters that I now need to upgrade both the host OS and the oVirt release. I have seen the notes about upgrading 4.2 to 4.3 and then being able to upgrade to 4.5, but what's not so clear is how to upgrade the host nodes' OS versions. After thinking about it, I wonder if this method is sane. Any advice or guidance is greatly appreciated. 1) spin up a new cluster running oVirt 4.5.x on the latest Rocky Linux release. 2) shutdown VMs on the original cluster 3) detach the domain storage 4) snapshot the domain storage (for rollback) 5) import the domain on the new cluster. 6) reload the OS on the original cluster to the new OS release (no local storage) 7) add those hosts to the new cluster

2 1

Snapshots and mess with Storage
by thorsten.bolten＠googlemail.com 29 Feb '24

29 Feb '24

Hi, I experienced a weird Failure. I hope I explain it understandable and not too hard to read We had issues with our backups, that they can't finalize Here is a Thread someone having the same issue. https://forums.veeam.com/ovirt-kvm-f62/veeam-rhv-12-1-command-removeimageti… so I made some investigations and recognized, it seems to be an Issue with Snapshot 1. I created a Snapshot https://paste.fo/c0b8e77a3400 what works 2. Then I tried to deleted the snapshot, what didn't work https://paste.fo/013e5632e0d6 So I logged in to a node and checked several things First strange thing was, I can see pvs vgs etc, I am used, that ovirt hides them in the System (on one of those nodes it's still the case) but anyways [root@ovnb05 ~]# pvs WARNING: Couldn't find device with uuid uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb. WARNING: VG 515bebca-972b-42ac-abff-d76af0071613 is missing PV uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb (last written to /dev/mapper/360002ac000000000000000880002a7fd). PV VG Fmt Attr PSize PFree /dev/mapper/360002ac000000000000000870002a7fd 515bebca-972b-42ac-abff-d76af0071613 lvm2 a-- <12.00t 10.77t /dev/sda3 rl lvm2 a-- 892.16g 0 [unknown] 515bebca-972b-42ac-abff-d76af0071613 lvm2 a-m <12.00t <12.00t So I did [root@ovnb05 ~]# pvdisplay WARNING: Couldn't find device with uuid uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb. WARNING: VG 515bebca-972b-42ac-abff-d76af0071613 is missing PV uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb (last written to /dev/mapper/360002ac000000000000000880002a7fd). --- Physical volume --- PV Name /dev/mapper/360002ac000000000000000870002a7fd VG Name 515bebca-972b-42ac-abff-d76af0071613 PV Size 12.00 TiB / not usable 384.00 MiB Allocatable yes PE Size 128.00 MiB Total PE 98301 Free PE 88247 Allocated PE 10054 PV UUID D6Typd-s7lA-PyoI-PvrL-rd9N-mS64-VeV1j3 --- Physical volume --- PV Name [unknown] VG Name 515bebca-972b-42ac-abff-d76af0071613 PV Size 12.00 TiB / not usable 384.00 MiB Allocatable yes PE Size 128.00 MiB Total PE 98301 Free PE 98301 Allocated PE 0 PV UUID uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb --- Physical volume --- PV Name /dev/sda3 VG Name rl PV Size 892.16 GiB / not usable 2.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 228393 Free PE 0 Allocated PE 228393 PV UUID kRe0EF-736L-QwZA-iGtM-67wp-8cr4-DKKcX2 I have never seen an unknown before, so I guess something happened to that LUN But the LUN is available and in /var/log/messages I can't find any issue with the LUN or the path etc. [root@ovnb05 data-center]# blkid |grep uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb /dev/sdi: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member" /dev/sdg: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member" /dev/sde: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member" /dev/sdn: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member" /dev/sdc: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member" /dev/sdl: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member" /dev/sdr: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member" /dev/sdp: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member" So I did some nasty stuff, like removing it and trying to add it again, through that process it told me that there is no meta Data on the Device So at the End I am a little bit lost now. Maybe the storage was failing for some reason or maybe veeam messed something up, but what leaves me afraid is, that the Ovirt-Engine didn't show me any Problem with the Cluster or the Storage. I mean it's obvious, that there was/is a huge failure with the storage and Ovirt did not recognize neither did it alarm me. At the moment I detached the LUN from the Storage Domain and created a seperate one (cause I recognized that having several LUN's in a Storage Domain seems to be an bad Idea) and one PVS is now hidden, the other one not [root@ovnb05 data-center]# pvs PV VG Fmt Attr PSize PFree /dev/mapper/360002ac000000000000000870002a7fd 515bebca-972b-42ac-abff-d76af0071613 lvm2 a-- <12.00t 11.81t /dev/sda3 rl lvm2 a-- 892.16g 0 [root@ovnb05 data-center]# cat /etc/lvm/devices/system.devices # LVM uses devices listed in this file. # Created by LVM command vgs pid 6123 at Fri Feb 16 18:16:35 2024 VERSION=1.1.7 IDTYPE=mpath_uuid IDNAME=mpath-360002ac000000000000000870002a7fd DEVNAME=/dev/mapper/360002ac000000000000000870002a7fd PVID=D6Typds7lAPyoIPvrLrd9NmS64VeV1j3 IDTYPE=sys_wwid IDNAME=naa.61c721d06b5fb2002c49fbecd02f9d90 DEVNAME=/dev/sda3 PVID=kRe0EF736LQwZAiGtM67wp8cr4DKKcX2 PART=3 [root@ovnb05 data-center]# multipath -ll 360000970000197800382533030303031 dm-5 EMC,SYMMETRIX size=5.6M features='1 queue_if_no_path' hwhandler='0' wp=ro `-+- policy='service-time 0' prio=1 status=active `- 9:0:25:0 sdj 8:144 active ready running 360002ac000000000000000870002a7fd dm-3 3PARdata,VV size=12T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- 9:0:8:1 sdh 8:112 active ready running |- 9:0:0:1 sdb 8:16 active ready running |- 9:0:1:1 sdd 8:48 active ready running |- 9:0:2:1 sdf 8:80 active ready running |- 10:0:7:1 sdq 65:0 active ready running |- 10:0:0:1 sdk 8:160 active ready running |- 10:0:1:1 sdm 8:192 active ready running `- 10:0:2:1 sdo 8:224 active ready running 360002ac000000000000000880002a7fd dm-4 3PARdata,VV size=12T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- 9:0:8:2 sdi 8:128 active ready running |- 9:0:0:2 sdc 8:32 active ready running |- 9:0:1:2 sde 8:64 active ready running |- 9:0:2:2 sdg 8:96 active ready running |- 10:0:7:2 sdr 65:16 active ready running |- 10:0:0:2 sdl 8:176 active ready running |- 10:0:1:2 sdn 8:208 active ready running `- 10:0:2:2 sdp 8:240 active ready running 360002ac0000000000000008f0002a7fd dm-26 3PARdata,VV size=12T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- 10:0:1:3 sdt 65:48 active ready running |- 10:0:0:3 sds 65:32 active ready running |- 10:0:7:3 sdv 65:80 active ready running |- 10:0:2:3 sdu 65:64 active ready running |- 9:0:1:3 sdx 65:112 active ready running |- 9:0:2:3 sdy 65:128 active ready running |- 9:0:0:3 sdw 65:96 active ready running `- 9:0:8:3 sdz 65:144 active ready running If I forgot something or you need more input just let me know, maybe you can light me up what's wrong thx in advance for any help/suggestions

1 0

I need install new host with O.S. Ubuntu 22 could you share to me which steps that I can follow
by oscar.loza＠toshibagcs.com 28 Feb '24

28 Feb '24

hi goo day: I need install new host with O.S. Ubuntu 22, could you share to me which steps that I can follow? thanks

2 1

ovirt-engine-appliance bad gpg key
by Nathanaël Blanchet 28 Feb '24

28 Feb '24

Hello, I tried to reinstall HE from a backup but I can't download ovirt- engine-appliance because of a GPG failure: 2024-02-28 17:26:32,642+0000 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 fatal: [localhost]: FAILED! => {"attempts": 10, "changed": false, "msg": "Failed to validate GPG signature for ovirt-engine-appliance-4.5-20231201120252.1.el8.x86_64: Public key for ovirt-engine-appliance-4.5- 20231201120252.1.el8.x86_64.rpm is not installed Thank you to help me how to do because there is non -nogpgcheck option as a workaround. -- Nathanaël Blanchet Administrateur Systèmes et Réseaux Service Informatique et REseau (SIRE) Département des systèmes d'information 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet(a)abes.fr

1 0

virtio-scsi. SSD vs HDD
by duparchy＠esrf.fr 28 Feb '24

28 Feb '24

HI, Is there a way to set a disk as being SSD (which really is). Or, better, set an entire Storage Domain being SSD ? This would prevent the schedule optimization to be automatically ON in Windows. and maybe enable other underlying optimization (TRIM ?)

1 0

VM actual size is showing more than the virtual size
by rootpentester20＠gmail.com 28 Feb '24

28 Feb '24

Hi team, I am using Ovirt 4.4 and ISCSI as my storage domain. Some of my VMs have actual sizes showing more than virtual size and when we look inside the Vm the used space is much less than the actual size. What could be an issue here and how can we resolve the same? Any help will be highly appreciated.

2 1

hosted engine deployment fails with certificate error
by jerome lardeux 27 Feb '24

27 Feb '24

Hello I,m trying to deploy a hosted engine in my lab, but i keep getting the same error Every time, while trying to install evirt packages, i got this: "ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 fatal: [localhost -> 192.168.222.91]: FAILED! => {"changed": false, "msg": "Failed to download metadata for repo 'ovirt-45-upstream': Cannot prepare internal mirrorlist: Curl error (60): Peer certificate cannot be authenticated with given CA certificates for https://mirrorlist.ovirt.org/mirrorlist-ovirt-4.5-el8 [SSL certificate problem: self signed certificate in certificate chain]", "rc": 1, "results": []}" it is an ovirt node 4.5 thank you for your help

1 0

Locked VHDD - Help To Unlock/Delete
by Matthew J Black 27 Feb '24

27 Feb '24

Hi All, I've gone and made a (bad) mistake. When creating a new VM I accidentally selected the wrong Storage Domain for the VM's VHDD; a Storage Domain with less space available than the size of the new (thick provisioned) VHDD. So now I want to delete that VHDD (and possibly the accompanying VM) and start over. However, the VHDD is "Locked" and I can't figure out a way to delete it (or Unlock it, or delete the VM, or Unattach the VHDD, etc, etc, etc). I can't find anything online so hopefully someone here can help me out. For the record I have full Admin access to *all* physical and virtual resources and I don't mind doing things via CLI (which I suspect is what's going to be required). I'm also familia with SQL (MariaDB, MS-SQL) but not Postgres, so if this is going to required manipulating the oVirt DB then so-be-it, I can do that as well. Thanks in advance Cheers Dulux-Oz

1 1