Good afternoon all, We had a confluence of events hit all at once and need help desperately. Our Ovirt engine system recently crashed and is unrecoverable. Due to a power maintenance event at the data center, 1/3 of our VM's are offline. I have recent backups from the engine created with engine-backup. I installed a clean Centos 9 and followed the directions to install the ovirt-engine . After I restore the backup, the engine-setup fails on the keycloak configuration. *From clean system:* *Install: **(Observe failed scriptlet during install, but rom install still succeeds)* [root@ovirt2 administrator]# dnf install -y ovirt-engine Last metadata expiration check: 2:08:15 ago on Tue 08 Aug 2023 10:11:31 AM CDT. Dependencies resolved. ============================================================================================================================================================= Package Architecture Version Repository Size ============================================================================================================================================================= Installing: ovirt-engine noarch 4.5.4-1.el9 centos-ovirt45 13 M Installing dependencies: SuperLU x86_64 5.3.0-2.el9 epel 182 k (Snip ...) * Running scriptlet: ovirt-vmconsole-1.0.9-1.el9.noarch 60/425Failed to resolve allow statement at /var/lib/selinux/targeted/tmp/modules/400/ovirt_vmconsole/cil:539Failed to resolve AST/usr/sbin/semodule: Failed!* (Snip ...) xmlrpc-common-3.1.3-1.1.el9.noarch xorg-x11-fonts-ISO8859-1-100dpi-7.5-33.el9.noarch zziplib-0.13.71-9.el9.x86_64 Complete! *Engine-restore (no visible issues):* [root@ovirt2 administrator]# engine-backup --mode=restore --log=restore1.log --file=Downloads/engine-2023-08-06.22.00.02.bak --provision-all-databases --restore-permissions Start of engine-backup with mode 'restore' scope: all archive file: Downloads/engine-2023-08-06.22.00.02.bak log file: restore1.log Preparing to restore: - Unpacking file 'Downloads/engine-2023-08-06.22.00.02.bak' Restoring: - Files ------------------------------------------------------------------------------ Please note: Operating system is different from the one used during backup. Current operating system: centos9 Operating system at backup: centos8 Apache httpd configuration will not be restored. You will be asked about it on the next engine-setup run. ------------------------------------------------------------------------------ Provisioning PostgreSQL users/databases: - user 'engine', database 'engine' - user 'ovirt_engine_history', database 'ovirt_engine_history' - user 'ovirt_engine_history_grafana' on database 'ovirt_engine_history' Restoring: - Engine database 'engine' - Cleaning up temporary tables in engine database 'engine' - Updating DbJustRestored VdcOption in engine database - Resetting DwhCurrentlyRunning in dwh_history_timekeeping in engine database - Resetting HA VM status ------------------------------------------------------------------------------ Please note: The engine database was backed up at 2023-08-06 22:00:19.000000000 -0500 . Objects that were added, removed or changed after this date, such as virtual machines, disks, etc., are missing in the engine, and will probably require recovery or recreation. ------------------------------------------------------------------------------ - DWH database 'ovirt_engine_history' - Grafana database '/var/lib/grafana/grafana.db' You should now run engine-setup. Done. [root@ovirt2 administrator]# *Engine-setup :* [root@ovirt2 administrator]# engine-setup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20230808124501-joveku.log Version: otopi-1.10.3 (otopi-1.10.3-1.el9) [ INFO ] The engine DB has been restored from a backup *[ ERROR ] Failed to execute stage 'Environment setup': Cannot connect to Keycloak database 'ovirt_engine_keycloak' using existing credentials: ovirt_engine_keycloak@localhost:5432*[ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20230808124501-joveku.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20230808124504-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination *[ ERROR ] Execution of setup failed[root@ovirt2 administrator]#* *Engine-cleanup results:* (snip) [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-remove-20230808120445-mj4eef.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20230808120508-cleanup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ INFO ] Execution of cleanup completed successfully [root@cen-90-tmpl administrator]# *Engine backup (restore) results:* [root@ovirt2 administrator]# engine-backup --mode=restore --log=restore1.log --file=Downloads/engine-2023-08-06.22.00.02.bak --provision-all-databases --restore-permissions Start of engine-backup with mode 'restore' scope: all archive file: Downloads/engine-2023-08-06.22.00.02.bak log file: restore1.log Preparing to restore: - Unpacking file 'Downloads/engine-2023-08-06.22.00.02.bak' Restoring: - Files ------------------------------------------------------------------------------ Please note: Operating system is different from the one used during backup. Current operating system: centos9 Operating system at backup: centos8 Apache httpd configuration will not be restored. You will be asked about it on the next engine-setup run. ------------------------------------------------------------------------------ Provisioning PostgreSQL users/databases: - user 'engine', database 'engine' *FATAL: Existing database 'engine' or user 'engine' found and temporary ones created - Please clean up everything and try again* Any advice would be appreciated. *David Johnson*