Thank you so much, Gianluca! Yes, the source and target environments are the same version. I'm not able to find /root/DisableFenceAtStartupInSec.txt anywhere, but maybe that's because at this point I've reverted to the original hosted_engine? Here is the output of the commands you sent: ------ [root@lnxvirt-engine ~]# engine-config -g DisableFenceAtStartupInSec Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false DisableFenceAtStartupInSec: 300 version: general [root@lnxvirt-engine ~]# set -euo pipefail && engine-config -g DisableFenceAtStartupInSec | cut -d' ' -f 2 Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false 300 engine=# select * from vdc_options where option_name='DisableFenceAtStartupInSec'; option_id | option_name | option_value | version | default_value -----------+----------------------------+--------------+---------+--------------- 45 | DisableFenceAtStartupInSec | 300 | general | 300 (1 row) ------ I also tried the following from the host I tried running the restore on. ------ [root@lnxvirt07 ~]# set -euo pipefail && echo "Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false DisableFenceAtStartupInSec: 300 version: general" | cut -d' ' -f 2 up 300 ------ Any additional questions or suggestions would be greatly appreciated. Thanks again, Devin On Oct 14, 2023, at 12:30 PM, Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote: On Sat, Oct 14, 2023 at 5:53 PM Devin A. Bougie <devin.bougie@cornell.edu<mailto:devin.bougie@cornell.edu>> wrote: Hello, We have a functioning oVirt 4.5.4 cluster running on fully-updated EL9.2 hosts. We are trying to migrate the self-hosted engine to a new iSCSI storage domain using the existing hosts, following the documented procedure: - set the cluster into global maintenance mode - backup the engine using "engine-backup --scope=all --mode=backup --file=backup.bck --log=backuplog.log" - shutdown the engine - restore the engine using "hosted-engine --deploy --4 --restore-from-file=backup.bck" This almost works, but fails with the attached log file. Any help or suggestions would be greatly appreciated, including alternate procedures for migrating a self-hosted engine from one domain to another. Many thanks, Devin If I'm right, the starting error seems to be this one: 2023-10-14 11:06:16,529-0400 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 fatal: [local host -> 192.168.1.25]: FAILED! => {"changed": true, "cmd": "set -euo pipefail && engine-config -g DisableFenceAtStartupInSec | c ut -d' ' -f2 > /root/DisableFenceAtStartupInSec.txt", "delta": "0:00:01.495195", "end": "2023-10-14 11:06:16.184479", "msg": "no n-zero return code", "rc": 1, "start": "2023-10-14 11:06:14.689284", "stderr": "Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=f alse", "stderr_lines": ["Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false"], "stdout": "", "stdout_lines": []} As the return code is 1 ("rc": 1,) and determines the failure of the playbook, possibly the old environment doesn't have DisableFenceAtStartupInSec engine config property correctly set and/or the "cut" command fails... Or some other problem with that config parameter. Can you verify what it put into /root/DisableFenceAtStartupInSec.txt? I have only a 4.4.10 env at hand and on it: [root@ovengine01 ~]# engine-config -g DisableFenceAtStartupInSec Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false DisableFenceAtStartupInSec: 300 version: general [root@ovengine01 ~]# [root@ovengine01 ~]# set -euo pipefail && engine-config -g DisableFenceAtStartupInSec | cut -d' ' -f 2 Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false 300 [root@ovengine01 ~]# what is the output of this command on your old env: engine-config -g DisableFenceAtStartupInSec ? Are the source and target environments the same version? If you have access to your old env could you also run this query on engine database: select * from vdc_options where option_name='DisableFenceAtStartupInSec'; eg this way [root@ovengine01 ~]# su - postgres [postgres@ovengine01 ~]$ psql engine psql (12.9) Type "help" for help. engine=# select * from vdc_options where option_name='DisableFenceAtStartupInSec'; option_id | option_name | option_value | version | default_value -----------+----------------------------+--------------+---------+--------------- 40 | DisableFenceAtStartupInSec | 300 | general | 300 (1 row) engine=# Gianluca

Hi Gianluca, Thanks for taking another look. I'm not sure what to make of /var/log/ovirt-hosted-engine-setup/engine-logs-2023-10-14T14:30:39Z/log/ovirt-engine/setup/restore-backup-20231014150412.log, but here it is. Does that explain anything to you, or give an idea of where to look next? Thanks again! Devin ------ 2023-10-14 11:04:12 7680: Start of engine-backup mode restore scope all file /root/engine_backup 2023-10-14 11:04:12 7680: OUTPUT: Start of engine-backup with mode 'restore' 2023-10-14 11:04:12 7680: OUTPUT: scope: all 2023-10-14 11:04:12 7680: OUTPUT: archive file: /root/engine_backup 2023-10-14 11:04:12 7680: OUTPUT: log file: /var/log/ovirt-engine/setup/restore-backup-20231014150412.log 2023-10-14 11:04:12 7680: OUTPUT: Preparing to restore: 2023-10-14 11:04:12 7680: OUTPUT: - Unpacking file '/root/engine_backup' 2023-10-14 11:04:12 7680: Opening tarball /root/engine_backup to /tmp/engine-backup.Onm6LsDR0g 2023-10-14 11:04:13 7680: Verifying hash 2023-10-14 11:04:13 7680: Verifying version 2023-10-14 11:04:13 7680: Reading config 2023-10-14 11:04:13 7680: Scope after checking backup content: 2023-10-14 11:04:13 7680: SCOPE_FILES:1 2023-10-14 11:04:13 7680: SCOPE_ENGINE_DB:1 2023-10-14 11:04:13 7680: SCOPE_DWH_DB:1 2023-10-14 11:04:13 7680: SCOPE_CINDERLIB_DB: 2023-10-14 11:04:13 7680: SCOPE_KEYCLOAK_DB: 2023-10-14 11:04:13 7680: SCOPE_GRAFANA_DB:1 2023-10-14 11:04:13 7680: OUTPUT: Restoring: 2023-10-14 11:04:13 7680: OUTPUT: - Files 2023-10-14 11:04:13 7680: Restoring files tar: var/lib/grafana/plugins/performancecopilot-pcp-app: Cannot open: File exists tar: Exiting with failure status due to previous errors 2023-10-14 11:04:13 7680: FATAL: Failed restoring /etc/ovirt-engine /etc/ovirt-engine-dwh /etc/ovirt-provider-ovn/conf.d /etc/ovirt-provider-ovn/logger.conf /etc/ovirt-vmconsole /etc/pki/ovirt-engine /etc/pki/ovirt-vmconsole /etc/ovirt-engine-setup.conf.d /etc/httpd/conf.d/internalsso-openidc.conf /etc/httpd/conf.d/ovirt-engine-grafana-proxy.conf /etc/httpd/conf.d/ovirt-engine-root-redirect.conf /etc/httpd/conf.d/ssl.conf /etc/httpd/conf.d/z-ovirt-engine-proxy.conf /etc/httpd/conf.d/z-ovirt-engine-keycloak-proxy.conf /etc/httpd/http.keytab /etc/httpd/conf.d/ovirt-sso.conf /etc/yum/pluginconf.d/versionlock.list /etc/dnf/plugins/versionlock.list /etc/firewalld/services/ovirt-https.xml /etc/firewalld/services/ovirt-http.xml /etc/firewalld/services/ovirt-postgres.xml /etc/firewalld/services/ovirt-provider-ovn.xml /etc/firewalld/services/ovn-central-firewall-service.xml /var/lib/openvswitch /etc/grafana /var/lib/ovirt-engine/content /var/lib/ovirt-engine/setup /var/lib/grafana/plugins ------

On Oct 15, 2023, at 5:59 AM, Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Sat, Oct 14, 2023 at 7:05 PM Devin A. Bougie <devin.bougie@cornell.edu> wrote: [snip] Any additional questions or suggestions would be greatly appreciated.

Thanks again, Devin

There is another FATAL line regarding restore itself, before the message I pointed out in my previous message. Can you analyze and/or share the contents of /var/log/ovirt-engine/setup/restore-backup-20231014150412.log?

Gianluca

On Wed, Oct 25, 2023 at 5:50 PM Devin A. Bougie <devin.bougie@cornell.edu> wrote:

I've had a chance to try this restore again, and this time login to the local (new) hosted-engine VM to verify that /root/DisableFenceAtStartupInSec.txt just contains: to

And if I try the "engine-config -g DisableFenceAtStartupInSec" from the new hosted-engine VM, my connection closes. [root@lnxvirt-engine ~]# cat /root/DisableFenceAtStartupInSec.txt to [root@lnxvirt-engine ~]# set -euo pipefail && engine-config -g DisableFenceAtStartupInSec Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false Connection to the Database failed. Please check that the hostname and port number are correct and that the Database service is up and running. Connection to 192.168.222.25 closed.

Any new suggestions or more tests I can run would be greatly appreciated.

Thanks, Devin

The key thing here is that for some reason it seems it is not able to connect to the database and so when "engine-config" command tries to get the second field of the output (the " | cut -d' ' -f2" part) it gets the "to" string here: Connection to the Database failed and anyway it returns error with failure of the overall playbook It should be investigated if there is a problem with the database itself on the new engine or if for some reason the "engine-config" command is not able to implicitly connect to the database

Does anyone know how to regenerate /etc/ovirt-hosted-engine/hosted-engine.conf? Or where exactly I find each field to create the file manually? After trying to switch from an iscsi storage domain to NFS for the new hosted engine, I finally have the engine back up and running. However, when trying to reinstall a host to move it to the new hosted engine domain, I get "Failed to fetch hosted engine configuration file." /etc/ovirt-hosted-engine/hosted-engine.conf doesn't exist on the host I ran "hosted-engine --deploy" on. And on all other hosts, it still references the old hosted engine VM and storage domain. Thanks, Devin

On Oct 25, 2023, at 3:55 PM, Devin A. Bougie <devin.bougie@cornell.edu> wrote:

Thanks again, Gianluca.

I'm currently ssh'd into the new local engine VM, and Postgres is running. However, an engine DB doesn't exist? Should it at this point, and do you have any other suggestions of where I should look?

Devin

------ [root@lnxvirt-engine ~]# su - postgres Last login: Wed Oct 25 15:47:18 EDT 2023 on pts/0 [postgres@lnxvirt-engine ~]$ psql psql (12.12) Type "help" for help.

postgres=# \l List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges -----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres (3 rows)

postgres=# ------

...

On Oct 25, 2023, at 12:40 PM, Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Wed, Oct 25, 2023 at 5:50 PM Devin A. Bougie <devin.bougie@cornell.edu> wrote: I've had a chance to try this restore again, and this time login to the local (new) hosted-engine VM to verify that /root/DisableFenceAtStartupInSec.txt just contains: to

And if I try the "engine-config -g DisableFenceAtStartupInSec" from the new hosted-engine VM, my connection closes. [root@lnxvirt-engine ~]# cat /root/DisableFenceAtStartupInSec.txt to [root@lnxvirt-engine ~]# set -euo pipefail && engine-config -g DisableFenceAtStartupInSec Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false Connection to the Database failed. Please check that the hostname and port number are correct and that the Database service is up and running. Connection to 192.168.222.25 closed.

Any new suggestions or more tests I can run would be greatly appreciated.

Thanks, Devin

The key thing here is that for some reason it seems it is not able to connect to the database and so when "engine-config" command tries to get the second field of the output (the " | cut -d' ' -f2" part) it gets the "to" string here:

Connection to the Database failed and anyway it returns error with failure of the overall playbook It should be investigated if there is a problem with the database itself on the new engine or if for some reason the "engine-config" command is not able to implicitly connect to the database

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/XJ6PEGB2LUCIOX...