Thank you so much, Gianluca! Yes, the source and target environments are the same version. I'm not able to find /root/DisableFenceAtStartupInSec.txt anywhere, but maybe that's because at this point I've reverted to the original hosted_engine? Here is the output of the commands you sent: ------ [root@lnxvirt-engine ~]# engine-config -g DisableFenceAtStartupInSec Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false DisableFenceAtStartupInSec: 300 version: general [root@lnxvirt-engine ~]# set -euo pipefail && engine-config -g DisableFenceAtStartupInSec | cut -d' ' -f 2 Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false 300 engine=# select * from vdc_options where option_name='DisableFenceAtStartupInSec'; option_id | option_name | option_value | version | default_value -----------+----------------------------+--------------+---------+--------------- 45 | DisableFenceAtStartupInSec | 300 | general | 300 (1 row) ------ I also tried the following from the host I tried running the restore on. ------ [root@lnxvirt07 ~]# set -euo pipefail && echo "Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false DisableFenceAtStartupInSec: 300 version: general" | cut -d' ' -f 2 up 300 ------ Any additional questions or suggestions would be greatly appreciated. Thanks again, Devin On Oct 14, 2023, at 12:30 PM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: On Sat, Oct 14, 2023 at 5:53 PM Devin A. Bougie <devin.bougie@cornell.edu<mailto:devin.bougie@cornell.edu>> wrote: Hello, We have a functioning oVirt 4.5.4 cluster running on fully-updated EL9.2 hosts. We are trying to migrate the self-hosted engine to a new iSCSI storage domain using the existing hosts, following the documented procedure: - set the cluster into global maintenance mode - backup the engine using "engine-backup --scope=all --mode=backup --file=backup.bck --log=backuplog.log" - shutdown the engine - restore the engine using "hosted-engine --deploy --4 --restore-from-file=backup.bck" This almost works, but fails with the attached log file. Any help or suggestions would be greatly appreciated, including alternate procedures for migrating a self-hosted engine from one domain to another. Many thanks, Devin If I'm right, the starting error seems to be this one: 2023-10-14 11:06:16,529-0400 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 fatal: [local host -> 192.168.1.25]: FAILED! => {"changed": true, "cmd": "set -euo pipefail && engine-config -g DisableFenceAtStartupInSec | c ut -d' ' -f2 > /root/DisableFenceAtStartupInSec.txt", "delta": "0:00:01.495195", "end": "2023-10-14 11:06:16.184479", "msg": "no n-zero return code", "rc": 1, "start": "2023-10-14 11:06:14.689284", "stderr": "Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=f alse", "stderr_lines": ["Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false"], "stdout": "", "stdout_lines": []} As the return code is 1 ("rc": 1,) and determines the failure of the playbook, possibly the old environment doesn't have DisableFenceAtStartupInSec engine config property correctly set and/or the "cut" command fails... Or some other problem with that config parameter. Can you verify what it put into /root/DisableFenceAtStartupInSec.txt? I have only a 4.4.10 env at hand and on it: [root@ovengine01 ~]# engine-config -g DisableFenceAtStartupInSec Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false DisableFenceAtStartupInSec: 300 version: general [root@ovengine01 ~]# [root@ovengine01 ~]# set -euo pipefail && engine-config -g DisableFenceAtStartupInSec | cut -d' ' -f 2 Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false 300 [root@ovengine01 ~]# what is the output of this command on your old env: engine-config -g DisableFenceAtStartupInSec ? Are the source and target environments the same version? If you have access to your old env could you also run this query on engine database: select * from vdc_options where option_name='DisableFenceAtStartupInSec'; eg this way [root@ovengine01 ~]# su - postgres [postgres@ovengine01 ~]$ psql engine psql (12.9) Type "help" for help. engine=# select * from vdc_options where option_name='DisableFenceAtStartupInSec'; option_id | option_name | option_value | version | default_value -----------+----------------------------+--------------+---------+--------------- 40 | DisableFenceAtStartupInSec | 300 | general | 300 (1 row) engine=# Gianluca

Hi Gianluca, Thanks for taking another look. I'm not sure what to make of /var/log/ovirt-hosted-engine-setup/engine-logs-2023-10-14T14:30:39Z/log/ovirt-engine/setup/restore-backup-20231014150412.log, but here it is. Does that explain anything to you, or give an idea of where to look next? Thanks again! Devin ------ 2023-10-14 11:04:12 7680: Start of engine-backup mode restore scope all file /root/engine_backup 2023-10-14 11:04:12 7680: OUTPUT: Start of engine-backup with mode 'restore' 2023-10-14 11:04:12 7680: OUTPUT: scope: all 2023-10-14 11:04:12 7680: OUTPUT: archive file: /root/engine_backup 2023-10-14 11:04:12 7680: OUTPUT: log file: /var/log/ovirt-engine/setup/restore-backup-20231014150412.log 2023-10-14 11:04:12 7680: OUTPUT: Preparing to restore: 2023-10-14 11:04:12 7680: OUTPUT: - Unpacking file '/root/engine_backup' 2023-10-14 11:04:12 7680: Opening tarball /root/engine_backup to /tmp/engine-backup.Onm6LsDR0g 2023-10-14 11:04:13 7680: Verifying hash 2023-10-14 11:04:13 7680: Verifying version 2023-10-14 11:04:13 7680: Reading config 2023-10-14 11:04:13 7680: Scope after checking backup content: 2023-10-14 11:04:13 7680: SCOPE_FILES:1 2023-10-14 11:04:13 7680: SCOPE_ENGINE_DB:1 2023-10-14 11:04:13 7680: SCOPE_DWH_DB:1 2023-10-14 11:04:13 7680: SCOPE_CINDERLIB_DB: 2023-10-14 11:04:13 7680: SCOPE_KEYCLOAK_DB: 2023-10-14 11:04:13 7680: SCOPE_GRAFANA_DB:1 2023-10-14 11:04:13 7680: OUTPUT: Restoring: 2023-10-14 11:04:13 7680: OUTPUT: - Files 2023-10-14 11:04:13 7680: Restoring files tar: var/lib/grafana/plugins/performancecopilot-pcp-app: Cannot open: File exists tar: Exiting with failure status due to previous errors 2023-10-14 11:04:13 7680: FATAL: Failed restoring /etc/ovirt-engine /etc/ovirt-engine-dwh /etc/ovirt-provider-ovn/conf.d /etc/ovirt-provider-ovn/logger.conf /etc/ovirt-vmconsole /etc/pki/ovirt-engine /etc/pki/ovirt-vmconsole /etc/ovirt-engine-setup.conf.d /etc/httpd/conf.d/internalsso-openidc.conf /etc/httpd/conf.d/ovirt-engine-grafana-proxy.conf /etc/httpd/conf.d/ovirt-engine-root-redirect.conf /etc/httpd/conf.d/ssl.conf /etc/httpd/conf.d/z-ovirt-engine-proxy.conf /etc/httpd/conf.d/z-ovirt-engine-keycloak-proxy.conf /etc/httpd/http.keytab /etc/httpd/conf.d/ovirt-sso.conf /etc/yum/pluginconf.d/versionlock.list /etc/dnf/plugins/versionlock.list /etc/firewalld/services/ovirt-https.xml /etc/firewalld/services/ovirt-http.xml /etc/firewalld/services/ovirt-postgres.xml /etc/firewalld/services/ovirt-provider-ovn.xml /etc/firewalld/services/ovn-central-firewall-service.xml /var/lib/openvswitch /etc/grafana /var/lib/ovirt-engine/content /var/lib/ovirt-engine/setup /var/lib/grafana/plugins ------

On Wed, Oct 25, 2023 at 5:50 PM Devin A. Bougie <devin.bougie(a)cornell.edu&gt; wrote: The key thing here is that for some reason it seems it is not able to connect to the database and so when "engine-config" command tries to get the second field of the output (the " | cut -d' ' -f2" part) it gets the "to" string here: Connection to the Database failed and anyway it returns error with failure of the overall playbook It should be investigated if there is a problem with the database itself on the new engine or if for some reason the "engine-config" command is not able to implicitly connect to the database

Does anyone know how to regenerate /etc/ovirt-hosted-engine/hosted-engine.conf? Or where exactly I find each field to create the file manually? After trying to switch from an iscsi storage domain to NFS for the new hosted engine, I finally have the engine back up and running. However, when trying to reinstall a host to move it to the new hosted engine domain, I get "Failed to fetch hosted engine configuration file." /etc/ovirt-hosted-engine/hosted-engine.conf doesn't exist on the host I ran "hosted-engine --deploy" on. And on all other hosts, it still references the old hosted engine VM and storage domain. Thanks, Devin