Currently, our upgrade to 4.4 fails with error:
FATAL: Existing database 'ovirt_engine_history' or user 'ovirt_engine_history' found and temporary ones created
We have upgraded the running 4.3 installation to the latest version and also use the latest packages for the upgrade on the new CentOS 8.2 installation. The back-up is made following the Hosted Engine upgrade steps in the manual, using: `engine-backup --scope=all --mode=backup --file=backup.bck --log=backuplog.log`
The upgrade is performed after copying the backup.bck file to the new server and using `hosted-engine --deploy --restore-from-file=backup.bck`
After creating the Engine VM, the installation process hangs when the backup is restored. We tried it several times, using a complete or a partial back-up.
Old/current oVirt version: 4.3.10.4-1.el7
New version: 4.4.1.8
ovirt-ansible-hosted-engine-setup: 1.1.6
Did anyone get the same error while upgrading an existing installation?
Thanks!
Error log Ansible on Host:
2020-07-15 12:34:09,361+0200 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup : Run engine-backup]
2020-07-15 12:35:28,778+0200 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:103 {'msg': 'non-zero return code', 'cmd': 'engine-backup --mode=restore --log=/var/log/ovirt-engine/setup/restore-backup-$(date -u +%Y%m%d%H%M%S).log --file=/root/engine_backup --provision-all-databases --restore-permissions', 'stdout': "Start of engine-backup with mode 'restore'\nscope: all\narchive file: /root/engine_backup\nlog file: /var/log/ovirt-engine/setup/restore-backup-20200715103410.log\nPreparing to restore:\n- Unpacking file '/root/engine_backup'\nRestoring:\n- Files\n------------------------------------------------------------------------------\nPlease note:\n\nOperating system is different from the one used during backup.\nCurrent operating system: centos8\nOperating system at backup: centos7\n\nApache httpd configuration will not be restored.\nYou will be asked about it on the next engine-setup run.\n----------------------------------------------------------
--------------------\nProvisioning PostgreSQL users/databases:\n- user 'engine', database 'engine'\n- extra user 'ovirt_engine_history' having grants on database engine, created with a random password\n- user 'ovirt_engine_history', database 'ovirt_engine_history'", 'stderr': "FATAL: Existing database 'ovirt_engine_history' or user 'ovirt_engine_history' found and temporary ones created - Please clean up everything and try again", 'rc': 1, 'start': '2020-07-15 12:34:10.824630', 'end': '2020-07-15 12:35:28.488261', 'delta': '0:01:17.663631', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'engine-backup --mode=restore --log=/var/log/ovirt-engine/setup/restore-backup-$(date -u +%Y%m%d%H%M%S).log --file=/root/engine_backup --provision-all-databases --restore-permissions', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines'
: ["Start of engine-backup with mode 'restore'", 'scope: all', 'archive file: /root/engine_backup', 'log file: /var/log/ovirt-engine/setup/restore-backup-20200715103410.log', 'Preparing to restore:', "- Unpacking file '/root/engine_backup'", 'Restoring:', '- Files', '------------------------------------------------------------------------------', 'Please note:', '', 'Operating system is different from the one used during backup.', 'Current operating system: centos8', 'Operating system at backup: centos7', '', 'Apache httpd configuration will not be restored.', 'You will be asked about it on the next engine-setup run.', '------------------------------------------------------------------------------', 'Provisioning PostgreSQL users/databases:', "- user 'engine', database 'engine'", "- extra user 'ovirt_engine_history' having grants on database engine, created with a random password", "- user 'ovirt_engine_history', database 'ovirt_engine_history'"], 'stderr_lines': ["FATAL: Existing d
atabase 'ovirt_engine_history' or user 'ovirt_engine_history' found and temporary ones created - Please clean up everything and try again"], '_ansible_no_log': False, '_ansible_delegated_vars': {'ansible_host': 'ovirt-management.dc1.triplon', 'ansible_port': None, 'ansible_user': 'root'}}
2020-07-15 12:35:28,879+0200 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:107 fatal: [localhost -> ovirt-management.dc1.triplon]: FAILED! => {"changed": true, "cmd": "engine-backup --mode=restore --log=/var/log/ovirt-engine/setup/restore-backup-$(date -u +%Y%m%d%H%M%S).log --file=/root/engine_backup --provision-all-databases --restore-permissions", "delta": "0:01:17.663631", "end": "2020-07-15 12:35:28.488261", "msg": "non-zero return code", "rc": 1, "start": "2020-07-15 12:34:10.824630", "stderr": "FATAL: Existing database 'ovirt_engine_history' or user 'ovirt_engine_history' found and temporary ones created - Please clean up everything and try again", "stderr_lines": ["FATAL: Existing database 'ovirt_engine_history' or user 'ovirt_engine_history' found and temporary ones created - Please clean up everything and try again"], "stdout": "Start of engine-backup with mode 'restore'\nscope: all\narchive file: /root/engine_backup\nlog file: /var/log/ov
irt-engine/setup/restore-backup-20200715103410.log\nPreparing to restore:\n- Unpacking file '/root/engine_backup'\nRestoring:\n- Files\n------------------------------------------------------------------------------\nPlease note:\n\nOperating system is different from the one used during backup.\nCurrent operating system: centos8\nOperating system at backup: centos7\n\nApache httpd configuration will not be restored.\nYou will be asked about it on the next engine-setup run.\n------------------------------------------------------------------------------\nProvisioning PostgreSQL users/databases:\n- user 'engine', database 'engine'\n- extra user 'ovirt_engine_history' having grants on database engine, created with a random password\n- user 'ovirt_engine_history', database 'ovirt_engine_history'", "stdout_lines": ["Start of engine-backup with mode 'restore'", "scope: all", "archive file: /root/engine_backup", "log file: /var/log/ovirt-engine/setup/restore-backup-20200715103410.log", "Prep
aring to restore:", "- Unpacking file '/root/engine_backup'", "Restoring:", "- Files", "------------------------------------------------------------------------------", "Please note:", "", "Operating system is different from the one used during backup.", "Current operating system: centos8", "Operating system at backup: centos7", "", "Apache httpd configuration will not be restored.", "You will be asked about it on the next engine-setup run.", "------------------------------------------------------------------------------", "Provisioning PostgreSQL users/databases:", "- user 'engine', database 'engine'", "- extra user 'ovirt_engine_history' having grants on database engine, created with a random password", "- user 'ovirt_engine_history', database 'ovirt_engine_history'"]}
restore-backup-xx.log file on the created Hosted Engine (retrieved by Ansible after failing):
2020-07-15 15:57:36 6437: Start of engine-backup mode restore scope all file /root/engine_backup
2020-07-15 15:57:36 6437: OUTPUT: Start of engine-backup with mode 'restore'
2020-07-15 15:57:36 6437: OUTPUT: scope: all
2020-07-15 15:57:36 6437: OUTPUT: archive file: /root/engine_backup
2020-07-15 15:57:36 6437: OUTPUT: log file: /var/log/ovirt-engine/setup/restore-backup-20200715135736.log
2020-07-15 15:57:36 6437: OUTPUT: Preparing to restore:
2020-07-15 15:57:36 6437: OUTPUT: - Unpacking file '/root/engine_backup'
2020-07-15 15:57:36 6437: Opening tarball /root/engine_backup to /tmp/engine-backup.5tvwGDx3qs
2020-07-15 15:57:37 6437: Verifying hash
2020-07-15 15:57:37 6437: Verifying version
2020-07-15 15:57:37 6437: Reading config
2020-07-15 15:57:37 6437: OUTPUT: Restoring:
2020-07-15 15:57:37 6437: OUTPUT: - Files
2020-07-15 15:57:37 6437: Restoring files
2020-07-15 15:57:38 6437: Reloading configuration
2020-07-15 15:57:38 6437: OUTPUT: Provisioning PostgreSQL users/databases:
2020-07-15 15:57:38 6437: provisionDB: user engine host localhost port 5432 database engine secured False secured_host_validation False
2020-07-15 15:57:38 6437: OUTPUT: - user 'engine', database 'engine'
[ INFO ] Stage: Initializing
[ INFO ] Stage: Environment setup
Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf, /tmp/engine-backup.5tvwGDx3qs/pg-provision-answer-file
Log file: /var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20200715155739-3o20a7.log
Version: otopi-1.9.2 (otopi-1.9.2-1.el8)
[ INFO ] Stage: Environment packages setup
[ INFO ] Stage: Programs detection
[ INFO ] Stage: Environment customization
[ INFO ] Stage: Setup validation
[ INFO ] Stage: Transaction setup
[ INFO ] Stage: Misc configuration (early)
[ INFO ] Stage: Package installation
[ INFO ] Stage: Misc configuration
[ INFO ] Initializing PostgreSQL
[ INFO ] Creating PostgreSQL 'engine' database
[ INFO ] Configuring PostgreSQL
[ INFO ] Install selinux module /usr/share/ovirt-engine/selinux/ansible-runner-service.cil
[ INFO ] Stage: Transaction commit
[ INFO ] Stage: Closing up
[ INFO ] Stage: Clean up
Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20200715155739-3o20a7.log
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ INFO ] Execution of provisiondb completed successfully
2020-07-15 15:58:14 6437: OUTPUT: - extra user 'ovirt_engine_history' having grants on database engine, created with a random password
[ INFO ] Stage: Initializing
[ INFO ] Stage: Environment setup
Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf, /tmp/engine-backup.5tvwGDx3qs/pg-provision-answer-file
Log file: /var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20200715155815-j4ba3l.log
Version: otopi-1.9.2 (otopi-1.9.2-1.el8)
[ INFO ] Stage: Environment packages setup
[ INFO ] Stage: Programs detection
[ INFO ] Stage: Environment customization
[ INFO ] Stage: Setup validation
[ INFO ] Stage: Transaction setup
[ INFO ] Stage: Misc configuration (early)
[ INFO ] Stage: Package installation
[ INFO ] Stage: Misc configuration
[ INFO ] Install selinux module /usr/share/ovirt-engine/selinux/ansible-runner-service.cil
[ INFO ] Stage: Transaction commit
[ INFO ] Stage: Closing up
[ INFO ] Stage: Clean up
Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20200715155815-j4ba3l.log
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ INFO ] Execution of provisiondb completed successfully
2020-07-15 15:58:46 6437: provisionDB: user ovirt_engine_history host localhost port 5432 database ovirt_engine_history secured False secured_host_validation False
2020-07-15 15:58:46 6437: OUTPUT: - user 'ovirt_engine_history', database 'ovirt_engine_history'
[ INFO ] Stage: Initializing
[ INFO ] Stage: Environment setup
Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf, /tmp/engine-backup.5tvwGDx3qs/pg-provision-answer-file
Log file: /var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20200715155847-qk7ipy.log
Version: otopi-1.9.2 (otopi-1.9.2-1.el8)
[ INFO ] Stage: Environment packages setup
[ INFO ] Stage: Programs detection
[ INFO ] Stage: Environment customization
[ INFO ] Stage: Setup validation
[ INFO ] Stage: Transaction setup
[ INFO ] Stage: Misc configuration (early)
[ INFO ] Stage: Package installation
[ INFO ] Stage: Misc configuration
[ INFO ] Creating PostgreSQL 'ovirt_engine_history' database
[ INFO ] Configuring PostgreSQL
[ ERROR ] Failed to execute stage 'Misc configuration': Existing resources found, new ones created:
database ovirt_engine_history_20200715155850 user ovirt_engine_history_20200715155850
[ INFO ] Stage: Clean up
Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20200715155847-qk7ipy.log
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ ERROR ] Execution of provisiondb failed
2020-07-15 15:58:52 6437: FATAL: Existing database 'ovirt_engine_history' or user 'ovirt_engine_history' found and temporary ones created - Please clean up everything and try again
