9:20 p.m.
Good afternoon all,
We had a confluence of events hit all at once and need help desperately.
Our Ovirt engine system recently crashed and is unrecoverable. Due to a
power maintenance event at the data center, 1/3 of our VM's are offline.
I have recent backups from the engine created with engine-backup.
I installed a clean Centos 9 and followed the directions to install
the ovirt-engine .
After I restore the backup, the engine-setup fails on the keycloak
configuration.
*From clean system:*
*Install: **(Observe failed scriptlet during install, but rom install still
succeeds)*
[root@ovirt2 administrator]# dnf install -y ovirt-engine
Last metadata expiration check: 2:08:15 ago on Tue 08 Aug 2023 10:11:31 AM
CDT.
Dependencies resolved.
=============================================================================================================================================================
Package Architecture
Version Repository
Size
=============================================================================================================================================================
Installing:
ovirt-engine noarch
4.5.4-1.el9 centos-ovirt45
13 M
Installing dependencies:
SuperLU x86_64
5.3.0-2.el9 epel
182 k
(Snip ...)
* Running scriptlet: ovirt-vmconsole-1.0.9-1.el9.noarch
60/425Failed to resolve allow statement at
/var/lib/selinux/targeted/tmp/modules/400/ovirt_vmconsole/cil:539Failed to
resolve AST/usr/sbin/semodule: Failed!*
(Snip ...)
xmlrpc-common-3.1.3-1.1.el9.noarch
xorg-x11-fonts-ISO8859-1-100dpi-7.5-33.el9.noarch
zziplib-0.13.71-9.el9.x86_64
Complete!
*Engine-restore (no visible issues):*
[root@ovirt2 administrator]# engine-backup --mode=restore
--log=restore1.log --file=Downloads/engine-2023-08-06.22.00.02.bak
--provision-all-databases --restore-permissions
Start of engine-backup with mode 'restore'
scope: all
archive file: Downloads/engine-2023-08-06.22.00.02.bak
log file: restore1.log
Preparing to restore:
- Unpacking file 'Downloads/engine-2023-08-06.22.00.02.bak'
Restoring:
- Files
------------------------------------------------------------------------------
Please note:
Operating system is different from the one used during backup.
Current operating system: centos9
Operating system at backup: centos8
Apache httpd configuration will not be restored.
You will be asked about it on the next engine-setup run.
------------------------------------------------------------------------------
Provisioning PostgreSQL users/databases:
- user 'engine', database 'engine'
- user 'ovirt_engine_history', database 'ovirt_engine_history'
- user 'ovirt_engine_history_grafana' on database 'ovirt_engine_history'
Restoring:
- Engine database 'engine'
- Cleaning up temporary tables in engine database 'engine'
- Updating DbJustRestored VdcOption in engine database
- Resetting DwhCurrentlyRunning in dwh_history_timekeeping in engine
database
- Resetting HA VM status
------------------------------------------------------------------------------
Please note:
The engine database was backed up at 2023-08-06 22:00:19.000000000 -0500 .
Objects that were added, removed or changed after this date, such as virtual
machines, disks, etc., are missing in the engine, and will probably require
recovery or recreation.
------------------------------------------------------------------------------
- DWH database 'ovirt_engine_history'
- Grafana database '/var/lib/grafana/grafana.db'
You should now run engine-setup.
Done.
[root@ovirt2 administrator]#
*Engine-setup :*
[root@ovirt2 administrator]# engine-setup
[ INFO ] Stage: Initializing
[ INFO ] Stage: Environment setup
Configuration files:
/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf,
/etc/ovirt-engine-setup.conf.d/10-packaging.conf,
/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf
Log file:
/var/log/ovirt-engine/setup/ovirt-engine-setup-20230808124501-joveku.log
Version: otopi-1.10.3 (otopi-1.10.3-1.el9)
[ INFO ] The engine DB has been restored from a backup
*[ ERROR ] Failed to execute stage 'Environment setup': Cannot connect to
Keycloak database 'ovirt_engine_keycloak' using existing credentials:
ovirt_engine_keycloak@localhost:5432*[ INFO ] Stage: Clean up
Log file is located at
/var/log/ovirt-engine/setup/ovirt-engine-setup-20230808124501-joveku.log
[ INFO ] Generating answer file
'/var/lib/ovirt-engine/setup/answers/20230808124504-setup.conf'
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
*[ ERROR ] Execution of setup failed[root@ovirt2 administrator]#*
*Engine-cleanup results:*
(snip)
[ INFO ] Stage: Clean up
Log file is located at
/var/log/ovirt-engine/setup/ovirt-engine-remove-20230808120445-mj4eef.log
[ INFO ] Generating answer file
'/var/lib/ovirt-engine/setup/answers/20230808120508-cleanup.conf'
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ INFO ] Execution of cleanup completed successfully
[root@cen-90-tmpl administrator]#
*Engine backup (restore) results:*
[root@ovirt2 administrator]# engine-backup --mode=restore
--log=restore1.log --file=Downloads/engine-2023-08-06.22.00.02.bak
--provision-all-databases --restore-permissions
Start of engine-backup with mode 'restore'
scope: all
archive file: Downloads/engine-2023-08-06.22.00.02.bak
log file: restore1.log
Preparing to restore:
- Unpacking file 'Downloads/engine-2023-08-06.22.00.02.bak'
Restoring:
- Files
------------------------------------------------------------------------------
Please note:
Operating system is different from the one used during backup.
Current operating system: centos9
Operating system at backup: centos8
Apache httpd configuration will not be restored.
You will be asked about it on the next engine-setup run.
------------------------------------------------------------------------------
Provisioning PostgreSQL users/databases:
- user 'engine', database 'engine'
*FATAL: Existing database 'engine' or user 'engine' found and temporary
ones created - Please clean up everything and try again*
Any advice would be appreciated.
*David Johnson*