Usually rebuilding the rpm database consists of:
rm /var/lib/rpm/__db*
- backup of /var/lib/rpm (just in case)
cd /var/lib/rpm
/usr/lib/rpm/rpmdb_verify Packages -> expected to fail
mv Packages Packages.bad
/usr/lib/rpm/rpmdb_dump Packages.bad | /usr/lib/rpm/rpmdb_load Packages

/usr/lib/rpm/rpmdb_verify Packages -> should succeed
rpm -v --rebuilddb

I doubt the procedure has changed on EL8.
In worst case scenario, you can copy the db from identical host.

Best Regards,
Strahil Nikolov


On Fri, Jan 14, 2022 at 10:08, Gianluca Cecchi
<gianluca.cecchi@gmail.com> wrote:
Just wanted to share what in subject and feedback/comments welcome.
Starting env is an external engine on CentOS 8.4, updated up to late September, with 4.4.8 async 2 (ovirt-engine-4.4.8.6-1.el8.noarch.rpm) and kernel 4.18.0-305.19.1.el8_4

[g.cecchi@ovmgr1 ~]$ cat /etc/centos-release
CentOS Linux release 8.4.2105

I migrated to Rocky Linux 8.5 and then run engine-setup to take it to current 4.4.9 level.
The system is a VM on a vSphere environment.
Currently there are 3 hosts with CentOS 8.4, that I'm going then to migrate to Rocky Linux too.

Downloaded the current migration script from

https://raw.githubusercontent.com/rocky-linux/rocky-tools/main/migrate2rocky/migrate2rocky.sh
chmod +x migrate2rocky.sh

My engine needs a proxy to exit on the internet and the script doesn't check my /etc/yum.conf configuration

so I set
export https_proxy=http://my_proxy:my_proxy_port
export http_proxy=http://my_proxy:my_proxy_port (not sure if this necessary...)

otherwise when you run the script you get
curl: (7) Failed to connect to dl.rockylinux.org port 443: Connection timed out
Error downloading the Rocky Linux signing key.

then as root

./migrate2rocky.sh -r

notable things observed:

. some packages from ovirt repos are gathered too... I don't know if correct in this stage...

Running dnf update before we attempt the migration.
Last metadata expiration check: 0:00:47 ago on Thu Jan 13 11:56:21 2022.
Dependencies resolved.
========================================================================
. . .
 otopi-common                                       noarch  1.9.6-1.el8                              ovirt-4.4                           94 k
 ovirt-ansible-collection                           noarch  1.6.5-1.el8                              ovirt-4.4                          289 k
 ovirt-engine-dwh-grafana-integration-setup         noarch  4.4.9.1-1.el8                            ovirt-4.4                           88 k
 ovirt-engine-dwh-setup                             noarch  4.4.9.1-1.el8                            ovirt-4.4                           95 k
 ovirt-engine-extension-aaa-ldap                    noarch  1.4.5-1.el8                              ovirt-4.4                          130 k
 ovirt-engine-metrics                               noarch  1.4.4-1.el8                              ovirt-4.4                           92 k
 ovirt-engine-setup                                 noarch  4.4.9.5-1.el8                            ovirt-4.4                           18 k
 ovirt-engine-setup-base                            noarch  4.4.9.5-1.el8                            ovirt-4.4                          116 k
 ovirt-engine-setup-plugin-cinderlib                noarch  4.4.9.5-1.el8                            ovirt-4.4                           39 k
 ovirt-engine-setup-plugin-imageio                  noarch  4.4.9.5-1.el8                            ovirt-4.4                           27 k
 ovirt-engine-setup-plugin-ovirt-engine             noarch  4.4.9.5-1.el8                            ovirt-4.4                          203 k
 ovirt-engine-setup-plugin-ovirt-engine-common      noarch  4.4.9.5-1.el8                            ovirt-4.4                          122 k
 ovirt-engine-setup-plugin-vmconsole-proxy-helper   noarch  4.4.9.5-1.el8                            ovirt-4.4                           38 k
 ovirt-engine-setup-plugin-websocket-proxy          noarch  4.4.9.5-1.el8                            ovirt-4.4                           39 k
 ovirt-engine-vmconsole-proxy-helper                noarch  4.4.9.5-1.el8                            ovirt-4.4                           26 k
 ovirt-engine-websocket-proxy                       noarch  4.4.9.5-1.el8                            ovirt-4.4                           33 k
 ovirt-imageio-client                               x86_64  2.3.0-1.el8                              ovirt-4.4                           21 k
 ovirt-imageio-common                               x86_64  2.3.0-1.el8                              ovirt-4.4                          158 k
 ovirt-imageio-daemon                               x86_64  2.3.0-1.el8                              ovirt-4.4                           15 k
 ovirt-release44                                    noarch  4.4.9.3-1.el8                            ovirt-4.4                           20 k
 ovirt-web-ui                                       noarch  1.7.2-1.el8                              ovirt-4.4                           11 M
. . .
 qemu-img                                           x86_64  15:6.0.0-33.el8                          ovirt-4.4-advanced-virtualization  1.9 M
. . .
Transaction Summary
=========================================================================================

Install    5 Packages
Upgrade  393 Packages
Remove     3 Packages

Total download size: 689 M
. . .
Generating grub configuration file ...
File descriptor 3 (/dev/pts/1) leaked on vgs invocation. Parent PID 4056644: /usr/sbin/grub2-probe
File descriptor 4 (/dev/pts/1) leaked on vgs invocation. Parent PID 4056644: /usr/sbin/grub2-probe
File descriptor 5 (/var/log/migrate2rocky.log) leaked on vgs invocation. Parent PID 4056644: /usr/sbin/grub2-probe
File descriptor 3 (/dev/pts/1) leaked on vgs invocation. Parent PID 4056644: /usr/sbin/grub2-probe
. . .
Adding boot menu entry for EFI firmware configuration
done
BootCurrent: 0003
BootOrder: 0004,0003,0000,0001,0002
Boot0000* EFI Virtual disk (0.0)
Boot0001* EFI VMware Virtual SATA CDROM Drive (0.0)
Boot0002* EFI Network
Boot0003* CentOS Linux
Boot0004* Rocky Linux


Done, please reboot your system.
A log of this installation can be found at /var/log/migrate2rocky.log

Due to the oVirt related packages update, I then executed the upgrade, running the "engine-setup" command, without rebooting the server at this stage

In the output I got these things with many packages to be installed; I don't know if it is something depending on 4.4.8 -> 4.4.9 or on Rocky migration:

. . .
Checking for product updates...
          Setup needs to install or update the following packages:
          [install] adobe-mappings-cmap-20171205-3.el8.noarch will be installed
          [install] adobe-mappings-cmap-deprecated-20171205-3.el8.noarch will be installed
          [install] adobe-mappings-pdf-20180407-1.el8.noarch will be installed
          [install] atk-2.28.1-1.el8.x86_64 will be installed
          [install] blosc-1.17.0-1.el8.x86_64 will be installed
          [install] ceph-common-16.2.7-1.el8.x86_64 will be installed
          [install] dejavu-sans-fonts-2.35-7.el8.noarch will be installed
          [install] fribidi-1.0.4-8.el8.x86_64 will be installed
          [install] gd-2.2.5-7.el8.x86_64 will be installed
          [install] gdk-pixbuf2-modules-2.36.12-5.el8.x86_64 will be installed
          [install] google-droid-sans-fonts-20120715-13.el8.noarch will be installed
          [install] gperftools-libs-2.9.1-1.el8.x86_64 will be installed
          [install] graphviz-2.40.1-43.el8.x86_64 will be installed
          [install] gtk-update-icon-cache-3.22.30-8.el8.x86_64 will be installed
          [install] gtk2-2.24.32-5.el8.x86_64 will be installed
          [install] hdf5-1.10.5-5.el8.x86_64 will be installed
          [install] hicolor-icon-theme-0.17-2.el8.noarch will be installed
          [install] jasper-libs-2.0.14-5.el8.x86_64 will be installed
          [install] jbig2dec-libs-0.16-1.el8.x86_64 will be installed
          [install] jbigkit-libs-2.1-14.el8.x86_64 will be installed
          [install] leveldb-1.20-1.el8.x86_64 will be installed
          [install] libICE-1.0.9-15.el8.x86_64 will be installed
          [install] libSM-1.2.3-1.el8.x86_64 will be installed
          [install] libXaw-1.0.13-10.el8.x86_64 will be installed
          [install] libXcomposite-0.4.4-14.el8.x86_64 will be installed
          [install] libXcursor-1.1.15-3.el8.x86_64 will be installed
          [install] libXdamage-1.1.4-14.el8.x86_64 will be installed
          [install] libXfixes-5.0.3-7.el8.x86_64 will be installed
          [install] libXft-2.3.3-1.el8.x86_64 will be installed
          [install] libXi-1.7.10-1.el8.x86_64 will be installed
          [install] libXinerama-1.1.4-1.el8.x86_64 will be installed
          [install] libXmu-1.1.3-1.el8.x86_64 will be installed
          [install] libXpm-3.5.12-8.el8.x86_64 will be installed
          [install] libXrandr-1.5.2-1.el8.x86_64 will be installed
          [install] libXt-1.1.5-12.el8.x86_64 will be installed
          [install] libXxf86misc-1.0.4-1.el8.x86_64 will be installed
          [install] libXxf86vm-1.1.4-9.el8.x86_64 will be installed
          [install] libaec-1.0.2-3.el8.x86_64 will be installed
          [install] libbabeltrace-1.5.4-3.el8.x86_64 will be installed
          [install] libcephfs2-16.2.7-1.el8.x86_64 will be installed
. . .
          [install] urw-base35-nimbus-roman-fonts-20170801-10.el8.noarch will be installed
          [install] urw-base35-nimbus-sans-fonts-20170801-10.el8.noarch will be installed
          [install] urw-base35-p052-fonts-20170801-10.el8.noarch will be installed
          [install] urw-base35-standard-symbols-ps-fonts-20170801-10.el8.noarch will be installed
          [install] urw-base35-z003-fonts-20170801-10.el8.noarch will be installed
          [install] xorg-x11-font-utils-7.5-41.el8.x86_64 will be installed
          [install] xorg-x11-fonts-ISO8859-1-100dpi-7.5-19.el8.noarch will be installed
          [install] xorg-x11-server-utils-7.7-27.el8.x86_64 will be installed
          Replying "No" will abort Setup. You can pass the option "--offline" to prevent installing or updating packages.
          Do you wish to update them now? (Yes, No) [Yes]:
. . . I went ahead pressing Return button

DNF Downloaded Ceph packages for x86_64
          [ ERROR ] DNF Importing GPG key 0xE451E5B5:
          Userid     : "CentOS Storage SIG (http://wiki.centos.org/SpecialInterestGroup/S
torage) <security@centos.org>"
          Fingerprint: 7412 9C0B 173B 071A 3775 951A D4A2 E50B E451 E5B5
          From       : https://www.centos.org/keys/RPM-GPG-KEY-CentOS-SIG-Storage
         
          Please confirm Confirm use of GPG Key userid=CentOS Storage SIG (http://wiki.ce
ntos.org/SpecialInterestGroup/Storage) <security@centos.org> hexkeyid=E451E5B5 [yes/no]: yes
          [ INFO  ] DNF Installed: urw-base35-fonts-common-20170801-10.el8.noarch
. . .
          --== END OF SUMMARY ==--
         
          [ INFO  ] Stage: Clean up
          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20220113133933-7l9rq4.log
          [ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20220113134624-setup.conf'
          [ INFO  ] Stage: Pre-termination
          [ INFO  ] Stage: Termination
          [ INFO  ] Execution of setup completed successfully

so it ended ok

at this stage, before reboot

# yum update
Last metadata expiration check: 1:38:48 ago on Thu 13 Jan 2022 12:07:45 PM CET.
Dependencies resolved.
Nothing to do.
Complete!

reboot

But after reboot

[root@ovmgr1 g.cecchi]# dnf update
error: rpmdb: BDB0113 Thread/process 2771/140020593752960 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm
Error: Error: rpmdb open failed


[root@ovmgr1 g.cecchi]# ls -lrt /var/lib/rpm
total 241400
-rw-r--r--. 1 root root      8192 Jan 13 12:07 Enhancename
-rw-r--r--. 1 root root      8192 Jan 13 12:14 Filetriggername
-rw-r--r--. 1 root root      8192 Jan 13 12:14 Supplementname
-rw-r--r--. 1 root root      8192 Jan 13 13:44 Triggername
-rw-r--r--. 1 root root      8192 Jan 13 13:44 Suggestname
-rw-r--r--. 1 root root      8192 Jan 13 13:44 Recommendname
-rw-r--r--. 1 root root      8192 Jan 13 13:44 Transfiletriggername
-rw-r--r--. 1 root root     32768 Jan 13 13:44 Obsoletename
-rw-r--r--. 1 root root     16384 Jan 13 13:44 Conflictname
-rw-r--r--. 1 root root 232259584 Jan 13 13:44 Packages
-rw-r--r--. 1 root root     69632 Jan 13 13:44 Name
-rw-r--r--. 1 root root   6549504 Jan 13 13:44 Basenames
-rw-r--r--. 1 root root     32768 Jan 13 13:44 Group
-rw-r--r--. 1 root root    466944 Jan 13 13:44 Requirename
-rw-r--r--. 1 root root   3350528 Jan 13 13:44 Providename
-rw-r--r--. 1 root root   2478080 Jan 13 13:44 Dirnames
-rw-r--r--. 1 root root     98304 Jan 13 13:45 Sigmd5
-rw-r--r--. 1 root root     36864 Jan 13 13:45 Installtid
-rw-r--r--. 1 root root    167936 Jan 13 13:45 Sha1header
-rw-r--r--. 1 root root   1318912 Jan 13 13:50 __db.003
-rw-r--r--. 1 root root     90112 Jan 13 13:50 __db.002
-rw-r--r--. 1 root root    286720 Jan 13 13:50 __db.001
[root@ovmgr1 g.cecchi]# date

I followed what discussed here to rebuild the rpm database:
https://cloudlinux.zendesk.com/hc/en-us/articles/115004075294-Fix-rpmdb-Thread-died-in-Berkeley-DB-library

[root@ovmgr1 g.cecchi]# mkdir /var/lib/rpm/backup
[root@ovmgr1 g.cecchi]# cp -a /var/lib/rpm/__db* /var/lib/rpm/backup/
[root@ovmgr1 g.cecchi]# rm -f /var/lib/rpm/__db.[0-9][0-9]*
[root@ovmgr1 g.cecchi]# rpm --quiet -qa
[root@ovmgr1 g.cecchi]# rpm --rebuilddb
error: could not delete old database at /var/lib/rpmold.2997

[root@ovmgr1 g.cecchi]# yum clean all
163 files removed
[root@ovmgr1 g.cecchi]#


[root@ovmgr1 g.cecchi]# ls -lrt /var/lib/rpm
total 155112
-rw-r--r--. 1 root root      8192 Jan 13 13:53 Enhancename
-rw-r--r--. 1 root root      8192 Jan 13 13:53 Filetriggername
-rw-r--r--. 1 root root      8192 Jan 13 13:53 Supplementname
-rw-r--r--. 1 root root      8192 Jan 13 13:53 Suggestname
-rw-r--r--. 1 root root      8192 Jan 13 13:53 Transfiletriggername
-rw-r--r--. 1 root root      8192 Jan 13 13:53 Recommendname
-rw-r--r--. 1 root root      8192 Jan 13 13:53 Triggername
-rw-r--r--. 1 root root     73728 Jan 13 13:53 Sigmd5
-rw-r--r--. 1 root root    114688 Jan 13 13:53 Sha1header
-rw-r--r--. 1 root root    356352 Jan 13 13:53 Requirename
-rw-r--r--. 1 root root   3325952 Jan 13 13:53 Providename
-rw-r--r--. 1 root root 146223104 Jan 13 13:53 Packages
-rw-r--r--. 1 root root     32768 Jan 13 13:53 Obsoletename
-rw-r--r--. 1 root root     69632 Jan 13 13:53 Name
-rw-r--r--. 1 root root     20480 Jan 13 13:53 Installtid
-rw-r--r--. 1 root root     28672 Jan 13 13:53 Group
-rw-r--r--. 1 root root   2342912 Jan 13 13:53 Dirnames
-rw-r--r--. 1 root root     16384 Jan 13 13:53 Conflictname
-rw-r--r--. 1 root root   5820416 Jan 13 13:53 Basenames
-rw-r--r--. 1 root root     90112 Jan 13 13:53 __db.002
-rw-r--r--. 1 root root    286720 Jan 13 13:53 __db.001
-rw-r--r--. 1 root root    263800 Jan 13 13:53 __db.003
[root@ovmgr1 g.cecchi]#


[root@ovmgr1 g.cecchi]# dnf update
Ceph packages for x86_64                                 1.0 MB/s | 412 kB     00:00    
Rocky Linux 8 - AppStream                                8.9 MB/s | 8.7 MB     00:00    
Rocky Linux 8 - BaseOS                                   4.3 MB/s | 4.6 MB     00:01    
Rocky Linux 8 - Extras                                   6.2 kB/s |  10 kB     00:01    
Rocky Linux 8 - PowerTools                               3.5 MB/s | 2.5 MB     00:00    
Latest oVirt 4.4 Release                                 388 kB/s | 2.6 MB     00:06    
Extra Packages for Enterprise Linux 8 - x86_64           6.9 MB/s |  11 MB     00:01    
CentOS-8 - Gluster 8                                     133 kB/s |  77 kB     00:00    
virtio-win builds roughly matching what will be shipped   92 kB/s | 143 kB     00:01    
Copr repo for EL8_collection owned by sbonazzo           226 kB/s | 246 kB     00:01    
Copr repo for gluster-ansible owned by sac                10 kB/s | 7.3 kB     00:00    
Advanced Virtualization packages for x86_64              1.2 MB/s | 255 kB     00:00    
CentOS-8 - oVirt 4.4                                     5.3 MB/s | 2.3 MB     00:00    
CentOS-8 - OpsTools - collectd                           1.3 MB/s | 149 kB     00:00    
CentOS-8 - NFV OpenvSwitch                               533 kB/s |  67 kB     00:00    
OpenStack Victoria Repository                            6.2 MB/s | 3.7 MB     00:00    
Dependencies resolved.
Nothing to do.
Complete!
[root@ovmgr1 g.cecchi]#

Is the above repos' list supposed to be ok?

Also, currently I have these 3 files

[root@ovmgr1 g.cecchi]# cat /etc/rocky-release
Rocky Linux release 8.5 (Green Obsidian)

[root@ovmgr1 g.cecchi]# cat /etc/centos-release
Rocky Linux release 8.5 (Green Obsidian)

[root@ovmgr1 g.cecchi]# cat /etc/redhat-release
Rocky Linux release 8.5 (Green Obsidian)
[root@ovmgr1 g.cecchi]#


I don't know if I have to remove any of those....

On manager now, connecting to the web admin UI:
oVirt Open Virtualization Manager
Software Version:4.4.9.5-1.el8

The hosts are there and so the VMs. More tests and checks to be done before updating hosts.
In /etc/alternatives directory some java links were missing as detailed in Rocky migration page
https://github.com/rocky-linux/rocky-tools/tree/main/migrate2rocky

followed their instructions

[root@ovmgr1 g.cecchi]# rpm -qa --scripts java-{1.8.0,11}-openjdk-{headless,devel} | sed -n '/postinstall/, /exit/{ /postinstall/! { /exit/ ! p} }' | sh
[root@ovmgr1 g.cecchi]#

Gianluca
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/YUDJRC22SQPAPAIURQIVSEMGITDRQOOM/