- Users - oVirt List Archives

how to debug sanlock issue
by Jiří Sléžka 16 Jan '22

16 Jan '22

Hello, I have 3 node HCI cluster with glusterfs. oVirt 4.4.9.5-1. In last 2 weeks I experience 2 outages where HE and all/some vms were restarted. While digging in logs I can see that sanlock cannot renew leases and it leads to killing vms as is very good described in [1]. It looks to me like some hw issue with one of the hosts but cannot find which one. for example today's outage restarted vms on hosts 1 and 2 but not 3. Sanlock logs there are these lines in /var/log/messages on host 2 (ovirt-hci02) Jan 13 08:27:25 ovirt-hci02 sanlock[1263]: 2022-01-13 08:27:25 1416706 [341378]: s7 delta_renew read timeout 10 sec offset 0 /rhev/data-center/mnt/glusterSD/10.0.4.11:_vms/6de5ae6d-c7cc-4292-bdbf-10495a38837b/dom_md/ids Jan 13 08:28:59 ovirt-hci02 sanlock[1263]: 2022-01-13 08:28:59 1416800 [341257]: write_sectors delta_leader offset 1024 rv -202 /rhev/data-center/mnt/glusterSD/10.0.4.11:_engine/816a3d0b-2e10-4900-b3cb-4a9b5cd0dd5d/dom_md/ids Jan 13 08:29:27 ovirt-hci02 sanlock[1263]: 2022-01-13 08:29:27 1416828 [4189968]: write_sectors delta_leader offset 1024 rv -202 /rhev/data-center/mnt/glusterSD/10.0.4.11:_engine/816a3d0b-2e10-4900-b3cb-4a9b5cd0dd5d/dom_md/ids but not on hosts 1 and 3. Could it indicate that there could be storage related problem on host 1? could you please suggest further/better debugging approach? Thanx a lot, Jiri [1] https://www.ovirt.org/develop/developer-guide/vdsm/sanlock.html

2 2

External engine CentOS 8.4 and 4.4.8 to Rocky Linux 8.5 and 4.4.9
by Gianluca Cecchi 15 Jan '22

15 Jan '22

Just wanted to share what in subject and feedback/comments welcome. Starting env is an external engine on CentOS 8.4, updated up to late September, with 4.4.8 async 2 (ovirt-engine-4.4.8.6-1.el8.noarch.rpm) and kernel 4.18.0-305.19.1.el8_4 [g.cecchi@ovmgr1 ~]$ cat /etc/centos-release CentOS Linux release 8.4.2105 I migrated to Rocky Linux 8.5 and then run engine-setup to take it to current 4.4.9 level. The system is a VM on a vSphere environment. Currently there are 3 hosts with CentOS 8.4, that I'm going then to migrate to Rocky Linux too. Downloaded the current migration script from https://raw.githubusercontent.com/rocky-linux/rocky-tools/main/migrate2rock… chmod +x migrate2rocky.sh My engine needs a proxy to exit on the internet and the script doesn't check my /etc/yum.conf configuration so I set export https_proxy=http://my_proxy:my_proxy_port export http_proxy=http://my_proxy:my_proxy_port (not sure if this necessary...) otherwise when you run the script you get curl: (7) Failed to connect to dl.rockylinux.org port 443: Connection timed out Error downloading the Rocky Linux signing key. then as root ./migrate2rocky.sh -r notable things observed: . some packages from ovirt repos are gathered too... I don't know if correct in this stage... Running dnf update before we attempt the migration. Last metadata expiration check: 0:00:47 ago on Thu Jan 13 11:56:21 2022. Dependencies resolved. ======================================================================== . . . otopi-common noarch 1.9.6-1.el8 ovirt-4.4 94 k ovirt-ansible-collection noarch 1.6.5-1.el8 ovirt-4.4 289 k ovirt-engine-dwh-grafana-integration-setup noarch 4.4.9.1-1.el8 ovirt-4.4 88 k ovirt-engine-dwh-setup noarch 4.4.9.1-1.el8 ovirt-4.4 95 k ovirt-engine-extension-aaa-ldap noarch 1.4.5-1.el8 ovirt-4.4 130 k ovirt-engine-metrics noarch 1.4.4-1.el8 ovirt-4.4 92 k ovirt-engine-setup noarch 4.4.9.5-1.el8 ovirt-4.4 18 k ovirt-engine-setup-base noarch 4.4.9.5-1.el8 ovirt-4.4 116 k ovirt-engine-setup-plugin-cinderlib noarch 4.4.9.5-1.el8 ovirt-4.4 39 k ovirt-engine-setup-plugin-imageio noarch 4.4.9.5-1.el8 ovirt-4.4 27 k ovirt-engine-setup-plugin-ovirt-engine noarch 4.4.9.5-1.el8 ovirt-4.4 203 k ovirt-engine-setup-plugin-ovirt-engine-common noarch 4.4.9.5-1.el8 ovirt-4.4 122 k ovirt-engine-setup-plugin-vmconsole-proxy-helper noarch 4.4.9.5-1.el8 ovirt-4.4 38 k ovirt-engine-setup-plugin-websocket-proxy noarch 4.4.9.5-1.el8 ovirt-4.4 39 k ovirt-engine-vmconsole-proxy-helper noarch 4.4.9.5-1.el8 ovirt-4.4 26 k ovirt-engine-websocket-proxy noarch 4.4.9.5-1.el8 ovirt-4.4 33 k ovirt-imageio-client x86_64 2.3.0-1.el8 ovirt-4.4 21 k ovirt-imageio-common x86_64 2.3.0-1.el8 ovirt-4.4 158 k ovirt-imageio-daemon x86_64 2.3.0-1.el8 ovirt-4.4 15 k ovirt-release44 noarch 4.4.9.3-1.el8 ovirt-4.4 20 k ovirt-web-ui noarch 1.7.2-1.el8 ovirt-4.4 11 M . . . qemu-img x86_64 15:6.0.0-33.el8 ovirt-4.4-advanced-virtualization 1.9 M . . . Transaction Summary ========================================================================================= Install 5 Packages Upgrade 393 Packages Remove 3 Packages Total download size: 689 M . . . Generating grub configuration file ... File descriptor 3 (/dev/pts/1) leaked on vgs invocation. Parent PID 4056644: /usr/sbin/grub2-probe File descriptor 4 (/dev/pts/1) leaked on vgs invocation. Parent PID 4056644: /usr/sbin/grub2-probe File descriptor 5 (/var/log/migrate2rocky.log) leaked on vgs invocation. Parent PID 4056644: /usr/sbin/grub2-probe File descriptor 3 (/dev/pts/1) leaked on vgs invocation. Parent PID 4056644: /usr/sbin/grub2-probe . . . Adding boot menu entry for EFI firmware configuration done BootCurrent: 0003 BootOrder: 0004,0003,0000,0001,0002 Boot0000* EFI Virtual disk (0.0) Boot0001* EFI VMware Virtual SATA CDROM Drive (0.0) Boot0002* EFI Network Boot0003* CentOS Linux Boot0004* Rocky Linux Done, please reboot your system. A log of this installation can be found at /var/log/migrate2rocky.log Due to the oVirt related packages update, I then executed the upgrade, running the "engine-setup" command, without rebooting the server at this stage In the output I got these things with many packages to be installed; I don't know if it is something depending on 4.4.8 -> 4.4.9 or on Rocky migration: . . . Checking for product updates... Setup needs to install or update the following packages: [install] adobe-mappings-cmap-20171205-3.el8.noarch will be installed [install] adobe-mappings-cmap-deprecated-20171205-3.el8.noarch will be installed [install] adobe-mappings-pdf-20180407-1.el8.noarch will be installed [install] atk-2.28.1-1.el8.x86_64 will be installed [install] blosc-1.17.0-1.el8.x86_64 will be installed [install] ceph-common-16.2.7-1.el8.x86_64 will be installed [install] dejavu-sans-fonts-2.35-7.el8.noarch will be installed [install] fribidi-1.0.4-8.el8.x86_64 will be installed [install] gd-2.2.5-7.el8.x86_64 will be installed [install] gdk-pixbuf2-modules-2.36.12-5.el8.x86_64 will be installed [install] google-droid-sans-fonts-20120715-13.el8.noarch will be installed [install] gperftools-libs-2.9.1-1.el8.x86_64 will be installed [install] graphviz-2.40.1-43.el8.x86_64 will be installed [install] gtk-update-icon-cache-3.22.30-8.el8.x86_64 will be installed [install] gtk2-2.24.32-5.el8.x86_64 will be installed [install] hdf5-1.10.5-5.el8.x86_64 will be installed [install] hicolor-icon-theme-0.17-2.el8.noarch will be installed [install] jasper-libs-2.0.14-5.el8.x86_64 will be installed [install] jbig2dec-libs-0.16-1.el8.x86_64 will be installed [install] jbigkit-libs-2.1-14.el8.x86_64 will be installed [install] leveldb-1.20-1.el8.x86_64 will be installed [install] libICE-1.0.9-15.el8.x86_64 will be installed [install] libSM-1.2.3-1.el8.x86_64 will be installed [install] libXaw-1.0.13-10.el8.x86_64 will be installed [install] libXcomposite-0.4.4-14.el8.x86_64 will be installed [install] libXcursor-1.1.15-3.el8.x86_64 will be installed [install] libXdamage-1.1.4-14.el8.x86_64 will be installed [install] libXfixes-5.0.3-7.el8.x86_64 will be installed [install] libXft-2.3.3-1.el8.x86_64 will be installed [install] libXi-1.7.10-1.el8.x86_64 will be installed [install] libXinerama-1.1.4-1.el8.x86_64 will be installed [install] libXmu-1.1.3-1.el8.x86_64 will be installed [install] libXpm-3.5.12-8.el8.x86_64 will be installed [install] libXrandr-1.5.2-1.el8.x86_64 will be installed [install] libXt-1.1.5-12.el8.x86_64 will be installed [install] libXxf86misc-1.0.4-1.el8.x86_64 will be installed [install] libXxf86vm-1.1.4-9.el8.x86_64 will be installed [install] libaec-1.0.2-3.el8.x86_64 will be installed [install] libbabeltrace-1.5.4-3.el8.x86_64 will be installed [install] libcephfs2-16.2.7-1.el8.x86_64 will be installed . . . [install] urw-base35-nimbus-roman-fonts-20170801-10.el8.noarch will be installed [install] urw-base35-nimbus-sans-fonts-20170801-10.el8.noarch will be installed [install] urw-base35-p052-fonts-20170801-10.el8.noarch will be installed [install] urw-base35-standard-symbols-ps-fonts-20170801-10.el8.noarch will be installed [install] urw-base35-z003-fonts-20170801-10.el8.noarch will be installed [install] xorg-x11-font-utils-7.5-41.el8.x86_64 will be installed [install] xorg-x11-fonts-ISO8859-1-100dpi-7.5-19.el8.noarch will be installed [install] xorg-x11-server-utils-7.7-27.el8.x86_64 will be installed Replying "No" will abort Setup. You can pass the option "--offline" to prevent installing or updating packages. Do you wish to update them now? (Yes, No) [Yes]: . . . I went ahead pressing Return button DNF Downloaded Ceph packages for x86_64 [ ERROR ] DNF Importing GPG key 0xE451E5B5: Userid : "CentOS Storage SIG ( http://wiki.centos.org/SpecialInterestGroup/S torage) <security(a)centos.org>" Fingerprint: 7412 9C0B 173B 071A 3775 951A D4A2 E50B E451 E5B5 From : https://www.centos.org/keys/RPM-GPG-KEY-CentOS-SIG-Storage Please confirm Confirm use of GPG Key userid=CentOS Storage SIG ( http://wiki.ce ntos.org/SpecialInterestGroup/Storage) <security(a)centos.org> hexkeyid=E451E5B5 [yes/no]: yes [ INFO ] DNF Installed: urw-base35-fonts-common-20170801-10.el8.noarch . . . --== END OF SUMMARY ==-- [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20220113133933-7l9rq4.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20220113134624-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ INFO ] Execution of setup completed successfully so it ended ok at this stage, before reboot # yum update Last metadata expiration check: 1:38:48 ago on Thu 13 Jan 2022 12:07:45 PM CET. Dependencies resolved. Nothing to do. Complete! reboot But after reboot [root@ovmgr1 g.cecchi]# dnf update error: rpmdb: BDB0113 Thread/process 2771/140020593752960 failed: BDB1507 Thread died in Berkeley DB library error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Packages index using db5 - (-30973) error: cannot open Packages database in /var/lib/rpm Error: Error: rpmdb open failed [root@ovmgr1 g.cecchi]# ls -lrt /var/lib/rpm total 241400 -rw-r--r--. 1 root root 8192 Jan 13 12:07 Enhancename -rw-r--r--. 1 root root 8192 Jan 13 12:14 Filetriggername -rw-r--r--. 1 root root 8192 Jan 13 12:14 Supplementname -rw-r--r--. 1 root root 8192 Jan 13 13:44 Triggername -rw-r--r--. 1 root root 8192 Jan 13 13:44 Suggestname -rw-r--r--. 1 root root 8192 Jan 13 13:44 Recommendname -rw-r--r--. 1 root root 8192 Jan 13 13:44 Transfiletriggername -rw-r--r--. 1 root root 32768 Jan 13 13:44 Obsoletename -rw-r--r--. 1 root root 16384 Jan 13 13:44 Conflictname -rw-r--r--. 1 root root 232259584 Jan 13 13:44 Packages -rw-r--r--. 1 root root 69632 Jan 13 13:44 Name -rw-r--r--. 1 root root 6549504 Jan 13 13:44 Basenames -rw-r--r--. 1 root root 32768 Jan 13 13:44 Group -rw-r--r--. 1 root root 466944 Jan 13 13:44 Requirename -rw-r--r--. 1 root root 3350528 Jan 13 13:44 Providename -rw-r--r--. 1 root root 2478080 Jan 13 13:44 Dirnames -rw-r--r--. 1 root root 98304 Jan 13 13:45 Sigmd5 -rw-r--r--. 1 root root 36864 Jan 13 13:45 Installtid -rw-r--r--. 1 root root 167936 Jan 13 13:45 Sha1header -rw-r--r--. 1 root root 1318912 Jan 13 13:50 __db.003 -rw-r--r--. 1 root root 90112 Jan 13 13:50 __db.002 -rw-r--r--. 1 root root 286720 Jan 13 13:50 __db.001 [root@ovmgr1 g.cecchi]# date I followed what discussed here to rebuild the rpm database: https://cloudlinux.zendesk.com/hc/en-us/articles/115004075294-Fix-rpmdb-Thr… [root@ovmgr1 g.cecchi]# mkdir /var/lib/rpm/backup [root@ovmgr1 g.cecchi]# cp -a /var/lib/rpm/__db* /var/lib/rpm/backup/ [root@ovmgr1 g.cecchi]# rm -f /var/lib/rpm/__db.[0-9][0-9]* [root@ovmgr1 g.cecchi]# rpm --quiet -qa [root@ovmgr1 g.cecchi]# rpm --rebuilddb error: could not delete old database at /var/lib/rpmold.2997 [root@ovmgr1 g.cecchi]# yum clean all 163 files removed [root@ovmgr1 g.cecchi]# [root@ovmgr1 g.cecchi]# ls -lrt /var/lib/rpm total 155112 -rw-r--r--. 1 root root 8192 Jan 13 13:53 Enhancename -rw-r--r--. 1 root root 8192 Jan 13 13:53 Filetriggername -rw-r--r--. 1 root root 8192 Jan 13 13:53 Supplementname -rw-r--r--. 1 root root 8192 Jan 13 13:53 Suggestname -rw-r--r--. 1 root root 8192 Jan 13 13:53 Transfiletriggername -rw-r--r--. 1 root root 8192 Jan 13 13:53 Recommendname -rw-r--r--. 1 root root 8192 Jan 13 13:53 Triggername -rw-r--r--. 1 root root 73728 Jan 13 13:53 Sigmd5 -rw-r--r--. 1 root root 114688 Jan 13 13:53 Sha1header -rw-r--r--. 1 root root 356352 Jan 13 13:53 Requirename -rw-r--r--. 1 root root 3325952 Jan 13 13:53 Providename -rw-r--r--. 1 root root 146223104 Jan 13 13:53 Packages -rw-r--r--. 1 root root 32768 Jan 13 13:53 Obsoletename -rw-r--r--. 1 root root 69632 Jan 13 13:53 Name -rw-r--r--. 1 root root 20480 Jan 13 13:53 Installtid -rw-r--r--. 1 root root 28672 Jan 13 13:53 Group -rw-r--r--. 1 root root 2342912 Jan 13 13:53 Dirnames -rw-r--r--. 1 root root 16384 Jan 13 13:53 Conflictname -rw-r--r--. 1 root root 5820416 Jan 13 13:53 Basenames -rw-r--r--. 1 root root 90112 Jan 13 13:53 __db.002 -rw-r--r--. 1 root root 286720 Jan 13 13:53 __db.001 -rw-r--r--. 1 root root 263800 Jan 13 13:53 __db.003 [root@ovmgr1 g.cecchi]# [root@ovmgr1 g.cecchi]# dnf update Ceph packages for x86_64 1.0 MB/s | 412 kB 00:00 Rocky Linux 8 - AppStream 8.9 MB/s | 8.7 MB 00:00 Rocky Linux 8 - BaseOS 4.3 MB/s | 4.6 MB 00:01 Rocky Linux 8 - Extras 6.2 kB/s | 10 kB 00:01 Rocky Linux 8 - PowerTools 3.5 MB/s | 2.5 MB 00:00 Latest oVirt 4.4 Release 388 kB/s | 2.6 MB 00:06 Extra Packages for Enterprise Linux 8 - x86_64 6.9 MB/s | 11 MB 00:01 CentOS-8 - Gluster 8 133 kB/s | 77 kB 00:00 virtio-win builds roughly matching what will be shipped 92 kB/s | 143 kB 00:01 Copr repo for EL8_collection owned by sbonazzo 226 kB/s | 246 kB 00:01 Copr repo for gluster-ansible owned by sac 10 kB/s | 7.3 kB 00:00 Advanced Virtualization packages for x86_64 1.2 MB/s | 255 kB 00:00 CentOS-8 - oVirt 4.4 5.3 MB/s | 2.3 MB 00:00 CentOS-8 - OpsTools - collectd 1.3 MB/s | 149 kB 00:00 CentOS-8 - NFV OpenvSwitch 533 kB/s | 67 kB 00:00 OpenStack Victoria Repository 6.2 MB/s | 3.7 MB 00:00 Dependencies resolved. Nothing to do. Complete! [root@ovmgr1 g.cecchi]# Is the above repos' list supposed to be ok? Also, currently I have these 3 files [root@ovmgr1 g.cecchi]# cat /etc/rocky-release Rocky Linux release 8.5 (Green Obsidian) [root@ovmgr1 g.cecchi]# cat /etc/centos-release Rocky Linux release 8.5 (Green Obsidian) [root@ovmgr1 g.cecchi]# cat /etc/redhat-release Rocky Linux release 8.5 (Green Obsidian) [root@ovmgr1 g.cecchi]# I don't know if I have to remove any of those.... On manager now, connecting to the web admin UI: oVirt Open Virtualization Manager Software Version:4.4.9.5-1.el8 The hosts are there and so the VMs. More tests and checks to be done before updating hosts. In /etc/alternatives directory some java links were missing as detailed in Rocky migration page https://github.com/rocky-linux/rocky-tools/tree/main/migrate2rocky followed their instructions [root@ovmgr1 g.cecchi]# rpm -qa --scripts java-{1.8.0,11}-openjdk-{headless,devel} | sed -n '/postinstall/, /exit/{ /postinstall/! { /exit/ ! p} }' | sh [root@ovmgr1 g.cecchi]# Gianluca

2 1

Re: How-to get oVirt host certificated date
by Konstantin Shalygin 14 Jan '22

14 Jan '22

> It's possible to get, may be from Postgres, the host certificate date? > Engine run this check sometimes, but trigger this check seems impossible Anybody? @Sandro please help engine make check once per day and print to logs How can we run a manual check or see info in PostgreSQL database? This is required because the days until the end of the certificate's life expire, waiting for the next day in order to understand the result of deploying a new certificate is a strange situation Thanks, k

3 5

VFs Mellanox
by Jorge Visentini 12 Jan '22

12 Jan '22

Hi all. Is it possible to use the Mellanox VFs with bridge from VMs? My switch is configured as a trunk accepting VLAN 1000. I configured Mellanox ConnectX-3 to create 2 VFs. 1 VF I used for iSCSI on VLAN 1000 (it worked) and another VF I set up a bridge on VLAN 1000 to use on the VMs. What is happening is that the VM's mac hits the switch on VLAN 1000, but has no traffic. Below are some images of my setup... *Switch:* [image: image.png] *oVirt Host:* [image: image.png] [image: image.png] [image: image.png] [image: image.png] [image: image.png] [image: image.png] *oVirt Guest VM:* [image: image.png] Can anyone give a little help? All the best! -- Att, Jorge Visentini +55 55 98432-9868

1 0

After restored hosted engine as a new vms, all the existing & live hosts and storage domain are showing offline
by dhanaraj.ramesh＠yahoo.com 12 Jan '22

12 Jan '22

Hi Team due to some reasons one of our slef hosted engine went down, hence we provisioned new centos vm and restored the hosted engine backup file. After restoring we could see all the Datacenter, cluster, storage, hosts and vm entities in restored ovirt engine but they all are offline. please help us how to we connect all the running hosts and storage domain, vms back to restored hosted engine. do we need to get a down time for all the vms to restore the hosts? how do I activate all the existing storage domains as is.

4 6

Installin ovirt and ovirt host on orcale 8.5 linux issues
by Nazeem Durgahee 11 Jan '22

11 Jan '22

Hi, Having issue when installing Ovirt on Oracle linux 8.1,8.2,8,3 whereby it is unable to set the interpreter for python. It is looking for python 2 instead, Kindly find attached logs. Need your help on that. Regards NAZEEM DURGAHEE Senior System Engineer Advanced Infrastructure Services Harel Mallac Technologies Ltd. 1st Floor, Block A, Phoenix Central, Pont Fer, Phoenix 73524, Mauritius t: (230) 207 3300 | ext: 1154 f: (230) 207 3333 | m: (230) e: nazeem.durgahee(a)harelmallac.com w: harelmallactechnologies.com | BRN: C07007231 Disclaimer: This e-mail and any attachment thereof are intended only for the person or entity to whom it is addressed in the email and contain confidential and/or privileged information. If you are not the intended recipient of this e-mail, please notify the sender immediately and delete the material from all platforms. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Before opening any attachments, please check them for viruses and defects, since we do not accept any liability for loss and/or damage which may be caused as a result. Data Privacy: Any personal data that you share with us by email will be processed in accordance with our privacy policy found on www.harelmallac.com

3 4

How-to get oVirt host certificated date
by Konstantin Shalygin 11 Jan '22

11 Jan '22

Hi, It's possible to get, may be from Postgres, the host certificate date? Engine run this check sometimes, but trigger this check seems impossible Thanks

1 0

After restored hosted engine as a new vms, all the existing & live hosts and storage domain are showing offline
by dhanaraj.ramesh＠yahoo.com 11 Jan '22

11 Jan '22

Hi Team due to some reasons one of our slef hosted engine went down, hence we provisioned new centos vm and restored the hosted engine backup file. After restoring we could see all the Datacenter, cluster, storage, hosts and vm entities in restored ovirt engine but they all are offline. please help us how to we connect all the running hosts and storage domain, vms back to restored hosted engine. do we need to get a down time for all the vms to restore the hosts? how do I activate all the existing storage domains as is.

2 2

Gluster Hook differences between fresh and old clusters
by Strahil Nikolov 11 Jan '22

11 Jan '22

Hi All, recently I have migrated from 4.3.10 to 4.4.9 and it seems something odd is happening. Symptoms: - A lot of warnings for Gluster hook discrepancies - Trying to refresh the hooks via the sync button fails (engine error: https://justpaste.it/827zo ) - Existing "Default" cluster tracks more hooks than a fresh new cluster New cluster hooks: http://i.imgur.com/FEL2Z1D.png Migrated cluster: https://i.imgur.com/L8dWYZY.png What can I do to resolve the issue ? I've tried to resync the hooks, move away /var/lib/glusterd/hooks/1/ and reinstall gluster packages, try to resolve via the "Resolve Conflicts" in the UI and nothing helped so far. Best Regards, Strahil Nikolov

2 4

Error while removing snapshot: Unable to get volume info
by francesco＠shellrent.com 10 Jan '22

10 Jan '22

Hi all, I'm trying to remove a snapshot from a HA VM in a setup with glusterfs (2 nodes C8 stream oVirt 4.4 + 1 arbiter C8). The error that appears in the vdsm log of the host is: 2022-01-10 09:33:03,003+0100 ERROR (jsonrpc/4) [api] FINISH merge error=Merge failed: {'top': '441354e7-c234-4079-b494-53fa99cdce6f', 'base': 'fdf38f20-3416-4d75-a159-2a341b1ed637', 'job': '50206e3a-8018-4ea8-b191-e4bc859ae0c7', 'reason': 'Unable to get volume info for domain 574a3cd1-5617-4742-8de9-4732be4f27e0 volume 441354e7-c234-4079-b494-53fa99cdce6f'} (api:131) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/virt/livemerge.py", line 285, in merge drive.domainID, drive.poolID, drive.imageID, job.top) File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 5988, in getVolumeInfo (domainID, volumeID)) vdsm.virt.errors.StorageUnavailableError: Unable to get volume info for domain 574a3cd1-5617-4742-8de9-4732be4f27e0 volume 441354e7-c234-4079-b494-53fa99cdce6f During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 124, in method ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/API.py", line 776, in merge drive, baseVolUUID, topVolUUID, bandwidth, jobUUID) File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 5833, in merge driveSpec, baseVolUUID, topVolUUID, bandwidth, jobUUID) File "/usr/lib/python3.6/site-packages/vdsm/virt/livemerge.py", line 288, in merge str(e), top=top, base=job.base, job=job_id) The volume list in the host differs from the engine one: HOST: vdsm-tool dump-volume-chains 574a3cd1-5617-4742-8de9-4732be4f27e0 | grep -A10 0b995271-e7f3-41b3-aff7-b5ad7942c10d image: 0b995271-e7f3-41b3-aff7-b5ad7942c10d - fdf38f20-3416-4d75-a159-2a341b1ed637 status: OK, voltype: INTERNAL, format: COW, legality: LEGAL, type: SPARSE, capacity: 53687091200, truesize: 44255387648 - 10df3adb-38f4-41d1-be84-b8b5b86e92cc status: OK, voltype: LEAF, format: COW, legality: LEGAL, type: SPARSE, capacity: 53687091200, truesize: 7335407616 ls -1 0b995271-e7f3-41b3-aff7-b5ad7942c10d 10df3adb-38f4-41d1-be84-b8b5b86e92cc 10df3adb-38f4-41d1-be84-b8b5b86e92cc.lease 10df3adb-38f4-41d1-be84-b8b5b86e92cc.meta fdf38f20-3416-4d75-a159-2a341b1ed637 fdf38f20-3416-4d75-a159-2a341b1ed637.lease fdf38f20-3416-4d75-a159-2a341b1ed637.meta ENGINE: engine=# select * from images where image_group_id='0b995271-e7f3-41b3-aff7-b5ad7942c10d'; -[ RECORD 1 ]---------+------------------------------------- image_guid | 10df3adb-38f4-41d1-be84-b8b5b86e92cc creation_date | 2022-01-07 11:23:43+01 size | 53687091200 it_guid | 00000000-0000-0000-0000-000000000000 parentid | 441354e7-c234-4079-b494-53fa99cdce6f imagestatus | 1 lastmodified | 2022-01-07 11:23:39.951+01 vm_snapshot_id | bd2291a4-8018-4874-a400-8d044a95347d volume_type | 2 volume_format | 4 image_group_id | 0b995271-e7f3-41b3-aff7-b5ad7942c10d _create_date | 2022-01-07 11:23:41.448463+01 _update_date | 2022-01-07 11:24:10.414777+01 active | t volume_classification | 0 qcow_compat | 2 -[ RECORD 2 ]---------+------------------------------------- image_guid | 441354e7-c234-4079-b494-53fa99cdce6f creation_date | 2021-12-15 07:16:31.647+01 size | 53687091200 it_guid | 00000000-0000-0000-0000-000000000000 parentid | fdf38f20-3416-4d75-a159-2a341b1ed637 imagestatus | 1 lastmodified | 2022-01-07 11:23:41.448+01 vm_snapshot_id | 2d610958-59e3-4685-b209-139b4266012f volume_type | 2 volume_format | 4 image_group_id | 0b995271-e7f3-41b3-aff7-b5ad7942c10d _create_date | 2021-12-15 07:16:32.37005+01 _update_date | 2022-01-07 11:23:41.448463+01 active | f volume_classification | 1 qcow_compat | 0 -[ RECORD 3 ]---------+------------------------------------- image_guid | fdf38f20-3416-4d75-a159-2a341b1ed637 creation_date | 2020-08-12 17:16:07+02 size | 53687091200 it_guid | 00000000-0000-0000-0000-000000000000 parentid | 00000000-0000-0000-0000-000000000000 imagestatus | 4 lastmodified | 2021-12-15 07:16:32.369+01 vm_snapshot_id | 603811ba-3cdd-4388-a971-05e300ced0c3 volume_type | 2 volume_format | 4 image_group_id | 0b995271-e7f3-41b3-aff7-b5ad7942c10d _create_date | 2020-08-12 17:16:07.506823+02 _update_date | 2021-12-15 07:16:32.37005+01 active | f volume_classification | 1 qcow_compat | 2 However in the engine gui I see only two snapshots ID: 1- 10df3adb-38f4-41d1-be84-b8b5b86e92cc (status ok) 2- 441354e7-c234-4079-b494-53fa99cdce6f (Disk status illegal) So the situation is: - on the host I see two volumes, status OK - on the engine GUI I see two volumes, one OK and the other one, disk in illegal status - on the engine DB I see three volumes, that are the combo of the two previous situations I would avoid restarting the VM, any advice for fixing this messy situation? I can attach engine log/vdsm.log Thank you for your time and help, Francesco

3 2