[ovirt-users] External engine CentOS 8.4 and 4.4.8 to Rocky Linux 8.5 and 4.4.9

Just wanted to share what in subject and feedback/comments welcome. Starting env is an external engine on CentOS 8.4, updated up to late September, with 4.4.8 async 2 (ovirt-engine-4.4.8.6-1.el8.noarch.rpm) and kernel 4.18.0-305.19.1.el8_4 [g.cecchi@ovmgr1 ~]$ cat /etc/centos-release CentOS Linux release 8.4.2105 I migrated to Rocky Linux 8.5 and then run engine-setup to take it to current 4.4.9 level. The system is a VM on a vSphere environment. Currently there are 3 hosts with CentOS 8.4, that I'm going then to migrate to Rocky Linux too. Downloaded the current migration script from https://raw.githubusercontent.com/rocky-linux/rocky-tools/main/migrate2ro... chmod +x migrate2rocky.sh My engine needs a proxy to exit on the internet and the script doesn't check my /etc/yum.conf configuration so I set export https_proxy=http://my_proxy:my_proxy_port export http_proxy=http://my_proxy:my_proxy_port (not sure if this necessary...) otherwise when you run the script you get curl: (7) Failed to connect to dl.rockylinux.org port 443: Connection timed out Error downloading the Rocky Linux signing key. then as root ./migrate2rocky.sh -r notable things observed: . some packages from ovirt repos are gathered too... I don't know if correct in this stage... Running dnf update before we attempt the migration. Last metadata expiration check: 0:00:47 ago on Thu Jan 13 11:56:21 2022. Dependencies resolved. ======================================================================== . . . otopi-common noarch 1.9.6-1.el8 ovirt-4.4 94 k ovirt-ansible-collection noarch 1.6.5-1.el8 ovirt-4.4 289 k ovirt-engine-dwh-grafana-integration-setup noarch 4.4.9.1-1.el8 ovirt-4.4 88 k ovirt-engine-dwh-setup noarch 4.4.9.1-1.el8 ovirt-4.4 95 k ovirt-engine-extension-aaa-ldap noarch 1.4.5-1.el8 ovirt-4.4 130 k ovirt-engine-metrics noarch 1.4.4-1.el8 ovirt-4.4 92 k ovirt-engine-setup noarch 4.4.9.5-1.el8 ovirt-4.4 18 k ovirt-engine-setup-base noarch 4.4.9.5-1.el8 ovirt-4.4 116 k ovirt-engine-setup-plugin-cinderlib noarch 4.4.9.5-1.el8 ovirt-4.4 39 k ovirt-engine-setup-plugin-imageio noarch 4.4.9.5-1.el8 ovirt-4.4 27 k ovirt-engine-setup-plugin-ovirt-engine noarch 4.4.9.5-1.el8 ovirt-4.4 203 k ovirt-engine-setup-plugin-ovirt-engine-common noarch 4.4.9.5-1.el8 ovirt-4.4 122 k ovirt-engine-setup-plugin-vmconsole-proxy-helper noarch 4.4.9.5-1.el8 ovirt-4.4 38 k ovirt-engine-setup-plugin-websocket-proxy noarch 4.4.9.5-1.el8 ovirt-4.4 39 k ovirt-engine-vmconsole-proxy-helper noarch 4.4.9.5-1.el8 ovirt-4.4 26 k ovirt-engine-websocket-proxy noarch 4.4.9.5-1.el8 ovirt-4.4 33 k ovirt-imageio-client x86_64 2.3.0-1.el8 ovirt-4.4 21 k ovirt-imageio-common x86_64 2.3.0-1.el8 ovirt-4.4 158 k ovirt-imageio-daemon x86_64 2.3.0-1.el8 ovirt-4.4 15 k ovirt-release44 noarch 4.4.9.3-1.el8 ovirt-4.4 20 k ovirt-web-ui noarch 1.7.2-1.el8 ovirt-4.4 11 M . . . qemu-img x86_64 15:6.0.0-33.el8 ovirt-4.4-advanced-virtualization 1.9 M . . . Transaction Summary ========================================================================================= Install 5 Packages Upgrade 393 Packages Remove 3 Packages Total download size: 689 M . . . Generating grub configuration file ... File descriptor 3 (/dev/pts/1) leaked on vgs invocation. Parent PID 4056644: /usr/sbin/grub2-probe File descriptor 4 (/dev/pts/1) leaked on vgs invocation. Parent PID 4056644: /usr/sbin/grub2-probe File descriptor 5 (/var/log/migrate2rocky.log) leaked on vgs invocation. Parent PID 4056644: /usr/sbin/grub2-probe File descriptor 3 (/dev/pts/1) leaked on vgs invocation. Parent PID 4056644: /usr/sbin/grub2-probe . . . Adding boot menu entry for EFI firmware configuration done BootCurrent: 0003 BootOrder: 0004,0003,0000,0001,0002 Boot0000* EFI Virtual disk (0.0) Boot0001* EFI VMware Virtual SATA CDROM Drive (0.0) Boot0002* EFI Network Boot0003* CentOS Linux Boot0004* Rocky Linux Done, please reboot your system. A log of this installation can be found at /var/log/migrate2rocky.log Due to the oVirt related packages update, I then executed the upgrade, running the "engine-setup" command, without rebooting the server at this stage In the output I got these things with many packages to be installed; I don't know if it is something depending on 4.4.8 -> 4.4.9 or on Rocky migration: . . . Checking for product updates... Setup needs to install or update the following packages: [install] adobe-mappings-cmap-20171205-3.el8.noarch will be installed [install] adobe-mappings-cmap-deprecated-20171205-3.el8.noarch will be installed [install] adobe-mappings-pdf-20180407-1.el8.noarch will be installed [install] atk-2.28.1-1.el8.x86_64 will be installed [install] blosc-1.17.0-1.el8.x86_64 will be installed [install] ceph-common-16.2.7-1.el8.x86_64 will be installed [install] dejavu-sans-fonts-2.35-7.el8.noarch will be installed [install] fribidi-1.0.4-8.el8.x86_64 will be installed [install] gd-2.2.5-7.el8.x86_64 will be installed [install] gdk-pixbuf2-modules-2.36.12-5.el8.x86_64 will be installed [install] google-droid-sans-fonts-20120715-13.el8.noarch will be installed [install] gperftools-libs-2.9.1-1.el8.x86_64 will be installed [install] graphviz-2.40.1-43.el8.x86_64 will be installed [install] gtk-update-icon-cache-3.22.30-8.el8.x86_64 will be installed [install] gtk2-2.24.32-5.el8.x86_64 will be installed [install] hdf5-1.10.5-5.el8.x86_64 will be installed [install] hicolor-icon-theme-0.17-2.el8.noarch will be installed [install] jasper-libs-2.0.14-5.el8.x86_64 will be installed [install] jbig2dec-libs-0.16-1.el8.x86_64 will be installed [install] jbigkit-libs-2.1-14.el8.x86_64 will be installed [install] leveldb-1.20-1.el8.x86_64 will be installed [install] libICE-1.0.9-15.el8.x86_64 will be installed [install] libSM-1.2.3-1.el8.x86_64 will be installed [install] libXaw-1.0.13-10.el8.x86_64 will be installed [install] libXcomposite-0.4.4-14.el8.x86_64 will be installed [install] libXcursor-1.1.15-3.el8.x86_64 will be installed [install] libXdamage-1.1.4-14.el8.x86_64 will be installed [install] libXfixes-5.0.3-7.el8.x86_64 will be installed [install] libXft-2.3.3-1.el8.x86_64 will be installed [install] libXi-1.7.10-1.el8.x86_64 will be installed [install] libXinerama-1.1.4-1.el8.x86_64 will be installed [install] libXmu-1.1.3-1.el8.x86_64 will be installed [install] libXpm-3.5.12-8.el8.x86_64 will be installed [install] libXrandr-1.5.2-1.el8.x86_64 will be installed [install] libXt-1.1.5-12.el8.x86_64 will be installed [install] libXxf86misc-1.0.4-1.el8.x86_64 will be installed [install] libXxf86vm-1.1.4-9.el8.x86_64 will be installed [install] libaec-1.0.2-3.el8.x86_64 will be installed [install] libbabeltrace-1.5.4-3.el8.x86_64 will be installed [install] libcephfs2-16.2.7-1.el8.x86_64 will be installed . . . [install] urw-base35-nimbus-roman-fonts-20170801-10.el8.noarch will be installed [install] urw-base35-nimbus-sans-fonts-20170801-10.el8.noarch will be installed [install] urw-base35-p052-fonts-20170801-10.el8.noarch will be installed [install] urw-base35-standard-symbols-ps-fonts-20170801-10.el8.noarch will be installed [install] urw-base35-z003-fonts-20170801-10.el8.noarch will be installed [install] xorg-x11-font-utils-7.5-41.el8.x86_64 will be installed [install] xorg-x11-fonts-ISO8859-1-100dpi-7.5-19.el8.noarch will be installed [install] xorg-x11-server-utils-7.7-27.el8.x86_64 will be installed Replying "No" will abort Setup. You can pass the option "--offline" to prevent installing or updating packages. Do you wish to update them now? (Yes, No) [Yes]: . . . I went ahead pressing Return button DNF Downloaded Ceph packages for x86_64 [ ERROR ] DNF Importing GPG key 0xE451E5B5: Userid : "CentOS Storage SIG ( http://wiki.centos.org/SpecialInterestGroup/S torage) <security(a)centos.org&gt;&quot; Fingerprint: 7412 9C0B 173B 071A 3775 951A D4A2 E50B E451 E5B5 From : https://www.centos.org/keys/RPM-GPG-KEY-CentOS-SIG-Storage Please confirm Confirm use of GPG Key userid=CentOS Storage SIG ( http://wiki.ce ntos.org/SpecialInterestGroup/Storage) <security(a)centos.org&gt; hexkeyid=E451E5B5 [yes/no]: yes [ INFO ] DNF Installed: urw-base35-fonts-common-20170801-10.el8.noarch . . . --== END OF SUMMARY ==-- [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20220113133933-7l9rq4.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20220113134624-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ INFO ] Execution of setup completed successfully so it ended ok at this stage, before reboot # yum update Last metadata expiration check: 1:38:48 ago on Thu 13 Jan 2022 12:07:45 PM CET. Dependencies resolved. Nothing to do. Complete! reboot But after reboot [root@ovmgr1 g.cecchi]# dnf update error: rpmdb: BDB0113 Thread/process 2771/140020593752960 failed: BDB1507 Thread died in Berkeley DB library error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Packages index using db5 - (-30973) error: cannot open Packages database in /var/lib/rpm Error: Error: rpmdb open failed [root@ovmgr1 g.cecchi]# ls -lrt /var/lib/rpm total 241400 -rw-r--r--. 1 root root 8192 Jan 13 12:07 Enhancename -rw-r--r--. 1 root root 8192 Jan 13 12:14 Filetriggername -rw-r--r--. 1 root root 8192 Jan 13 12:14 Supplementname -rw-r--r--. 1 root root 8192 Jan 13 13:44 Triggername -rw-r--r--. 1 root root 8192 Jan 13 13:44 Suggestname -rw-r--r--. 1 root root 8192 Jan 13 13:44 Recommendname -rw-r--r--. 1 root root 8192 Jan 13 13:44 Transfiletriggername -rw-r--r--. 1 root root 32768 Jan 13 13:44 Obsoletename -rw-r--r--. 1 root root 16384 Jan 13 13:44 Conflictname -rw-r--r--. 1 root root 232259584 Jan 13 13:44 Packages -rw-r--r--. 1 root root 69632 Jan 13 13:44 Name -rw-r--r--. 1 root root 6549504 Jan 13 13:44 Basenames -rw-r--r--. 1 root root 32768 Jan 13 13:44 Group -rw-r--r--. 1 root root 466944 Jan 13 13:44 Requirename -rw-r--r--. 1 root root 3350528 Jan 13 13:44 Providename -rw-r--r--. 1 root root 2478080 Jan 13 13:44 Dirnames -rw-r--r--. 1 root root 98304 Jan 13 13:45 Sigmd5 -rw-r--r--. 1 root root 36864 Jan 13 13:45 Installtid -rw-r--r--. 1 root root 167936 Jan 13 13:45 Sha1header -rw-r--r--. 1 root root 1318912 Jan 13 13:50 __db.003 -rw-r--r--. 1 root root 90112 Jan 13 13:50 __db.002 -rw-r--r--. 1 root root 286720 Jan 13 13:50 __db.001 [root@ovmgr1 g.cecchi]# date I followed what discussed here to rebuild the rpm database: https://cloudlinux.zendesk.com/hc/en-us/articles/115004075294-Fix-rpmdb-T... [root@ovmgr1 g.cecchi]# mkdir /var/lib/rpm/backup [root@ovmgr1 g.cecchi]# cp -a /var/lib/rpm/__db* /var/lib/rpm/backup/ [root@ovmgr1 g.cecchi]# rm -f /var/lib/rpm/__db.[0-9][0-9]* [root@ovmgr1 g.cecchi]# rpm --quiet -qa [root@ovmgr1 g.cecchi]# rpm --rebuilddb error: could not delete old database at /var/lib/rpmold.2997 [root@ovmgr1 g.cecchi]# yum clean all 163 files removed [root@ovmgr1 g.cecchi]# [root@ovmgr1 g.cecchi]# ls -lrt /var/lib/rpm total 155112 -rw-r--r--. 1 root root 8192 Jan 13 13:53 Enhancename -rw-r--r--. 1 root root 8192 Jan 13 13:53 Filetriggername -rw-r--r--. 1 root root 8192 Jan 13 13:53 Supplementname -rw-r--r--. 1 root root 8192 Jan 13 13:53 Suggestname -rw-r--r--. 1 root root 8192 Jan 13 13:53 Transfiletriggername -rw-r--r--. 1 root root 8192 Jan 13 13:53 Recommendname -rw-r--r--. 1 root root 8192 Jan 13 13:53 Triggername -rw-r--r--. 1 root root 73728 Jan 13 13:53 Sigmd5 -rw-r--r--. 1 root root 114688 Jan 13 13:53 Sha1header -rw-r--r--. 1 root root 356352 Jan 13 13:53 Requirename -rw-r--r--. 1 root root 3325952 Jan 13 13:53 Providename -rw-r--r--. 1 root root 146223104 Jan 13 13:53 Packages -rw-r--r--. 1 root root 32768 Jan 13 13:53 Obsoletename -rw-r--r--. 1 root root 69632 Jan 13 13:53 Name -rw-r--r--. 1 root root 20480 Jan 13 13:53 Installtid -rw-r--r--. 1 root root 28672 Jan 13 13:53 Group -rw-r--r--. 1 root root 2342912 Jan 13 13:53 Dirnames -rw-r--r--. 1 root root 16384 Jan 13 13:53 Conflictname -rw-r--r--. 1 root root 5820416 Jan 13 13:53 Basenames -rw-r--r--. 1 root root 90112 Jan 13 13:53 __db.002 -rw-r--r--. 1 root root 286720 Jan 13 13:53 __db.001 -rw-r--r--. 1 root root 263800 Jan 13 13:53 __db.003 [root@ovmgr1 g.cecchi]# [root@ovmgr1 g.cecchi]# dnf update Ceph packages for x86_64 1.0 MB/s | 412 kB 00:00 Rocky Linux 8 - AppStream 8.9 MB/s | 8.7 MB 00:00 Rocky Linux 8 - BaseOS 4.3 MB/s | 4.6 MB 00:01 Rocky Linux 8 - Extras 6.2 kB/s | 10 kB 00:01 Rocky Linux 8 - PowerTools 3.5 MB/s | 2.5 MB 00:00 Latest oVirt 4.4 Release 388 kB/s | 2.6 MB 00:06 Extra Packages for Enterprise Linux 8 - x86_64 6.9 MB/s | 11 MB 00:01 CentOS-8 - Gluster 8 133 kB/s | 77 kB 00:00 virtio-win builds roughly matching what will be shipped 92 kB/s | 143 kB 00:01 Copr repo for EL8_collection owned by sbonazzo 226 kB/s | 246 kB 00:01 Copr repo for gluster-ansible owned by sac 10 kB/s | 7.3 kB 00:00 Advanced Virtualization packages for x86_64 1.2 MB/s | 255 kB 00:00 CentOS-8 - oVirt 4.4 5.3 MB/s | 2.3 MB 00:00 CentOS-8 - OpsTools - collectd 1.3 MB/s | 149 kB 00:00 CentOS-8 - NFV OpenvSwitch 533 kB/s | 67 kB 00:00 OpenStack Victoria Repository 6.2 MB/s | 3.7 MB 00:00 Dependencies resolved. Nothing to do. Complete! [root@ovmgr1 g.cecchi]# Is the above repos' list supposed to be ok? Also, currently I have these 3 files [root@ovmgr1 g.cecchi]# cat /etc/rocky-release Rocky Linux release 8.5 (Green Obsidian) [root@ovmgr1 g.cecchi]# cat /etc/centos-release Rocky Linux release 8.5 (Green Obsidian) [root@ovmgr1 g.cecchi]# cat /etc/redhat-release Rocky Linux release 8.5 (Green Obsidian) [root@ovmgr1 g.cecchi]# I don't know if I have to remove any of those.... On manager now, connecting to the web admin UI: oVirt Open Virtualization Manager Software Version:4.4.9.5-1.el8 The hosts are there and so the VMs. More tests and checks to be done before updating hosts. In /etc/alternatives directory some java links were missing as detailed in Rocky migration page https://github.com/rocky-linux/rocky-tools/tree/main/migrate2rocky followed their instructions [root@ovmgr1 g.cecchi]# rpm -qa --scripts java-{1.8.0,11}-openjdk-{headless,devel} | sed -n '/postinstall/, /exit/{ /postinstall/! { /exit/ ! p} }' | sh [root@ovmgr1 g.cecchi]# Gianluca