6:26 a.m.
Currently, our upgrade to 4.4 fails with error:
FATAL: Existing database 'ovirt_engine_history' or user
'ovirt_engine_history' found and temporary ones created
We have upgraded the running 4.3 installation to the latest version and also use the
latest packages for the upgrade on the new CentOS 8.2 installation. The back-up is made
following the Hosted Engine upgrade steps in the manual, using: `engine-backup --scope=all
--mode=backup --file=backup.bck --log=backuplog.log`
The upgrade is performed after copying the backup.bck file to the new server and using
`hosted-engine --deploy --restore-from-file=backup.bck`
After creating the Engine VM, the installation process hangs when the backup is restored.
We tried it several times, using a complete or a partial back-up.
Old/current oVirt version: 4.3.10.4-1.el7
New version: 4.4.1.8
ovirt-ansible-hosted-engine-setup: 1.1.6
Did anyone get the same error while upgrading an existing installation?
Thanks!
Error log Ansible on Host:
2020-07-15 12:34:09,361+0200 INFO otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup : Run engine-backup]
2020-07-15 12:35:28,778+0200 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:103 {'msg': 'non-zero return code',
'cmd': 'engine-backup --mode=restore
--log=/var/log/ovirt-engine/setup/restore-backup-$(date -u +%Y%m%d%H%M%S).log
--file=/root/engine_backup --provision-all-databases --restore-permissions',
'stdout': "Start of engine-backup with mode 'restore'\nscope:
all\narchive file: /root/engine_backup\nlog file:
/var/log/ovirt-engine/setup/restore-backup-20200715103410.log\nPreparing to restore:\n-
Unpacking file '/root/engine_backup'\nRestoring:\n-
Files\n------------------------------------------------------------------------------\nPlease
note:\n\nOperating system is different from the one used during backup.\nCurrent operating
system: centos8\nOperating system at backup: centos7\n\nApache httpd configuration will
not be restored.\nYou will be asked about it on the next engine-setup
run.\n----------------------------------------------------------
--------------------\nProvisioning PostgreSQL users/databases:\n- user 'engine',
database 'engine'\n- extra user 'ovirt_engine_history' having grants on
database engine, created with a random password\n- user 'ovirt_engine_history',
database 'ovirt_engine_history'", 'stderr': "FATAL: Existing
database 'ovirt_engine_history' or user 'ovirt_engine_history' found and
temporary ones created - Please clean up everything and try again", 'rc': 1,
'start': '2020-07-15 12:34:10.824630', 'end': '2020-07-15
12:35:28.488261', 'delta': '0:01:17.663631', 'changed': True,
'invocation': {'module_args': {'_raw_params': 'engine-backup
--mode=restore --log=/var/log/ovirt-engine/setup/restore-backup-$(date -u
+%Y%m%d%H%M%S).log --file=/root/engine_backup --provision-all-databases
--restore-permissions', '_uses_shell': True, 'warn': True,
'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None,
'chdir': None, 'executable': None, 'creates': None,
'removes': None, 'stdin': None}}, 'stdout_lines'
: ["Start of engine-backup with mode 'restore'", 'scope: all',
'archive file: /root/engine_backup', 'log file:
/var/log/ovirt-engine/setup/restore-backup-20200715103410.log', 'Preparing to
restore:', "- Unpacking file '/root/engine_backup'",
'Restoring:', '- Files',
'------------------------------------------------------------------------------',
'Please note:', '', 'Operating system is different from the one used
during backup.', 'Current operating system: centos8', 'Operating system at
backup: centos7', '', 'Apache httpd configuration will not be
restored.', 'You will be asked about it on the next engine-setup run.',
'------------------------------------------------------------------------------',
'Provisioning PostgreSQL users/databases:', "- user 'engine',
database 'engine'", "- extra user 'ovirt_engine_history' having
grants on database engine, created with a random password", "- user
'ovirt_engine_history', database 'ovirt_engine_history'"],
'stderr_lines': ["FATAL: Existing d
atabase 'ovirt_engine_history' or user 'ovirt_engine_history' found and
temporary ones created - Please clean up everything and try again"],
'_ansible_no_log': False, '_ansible_delegated_vars':
{'ansible_host': 'ovirt-management.dc1.triplon', 'ansible_port':
None, 'ansible_user': 'root'}}
2020-07-15 12:35:28,879+0200 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:107 fatal: [localhost -> ovirt-management.dc1.triplon]:
FAILED! => {"changed": true, "cmd": "engine-backup
--mode=restore --log=/var/log/ovirt-engine/setup/restore-backup-$(date -u
+%Y%m%d%H%M%S).log --file=/root/engine_backup --provision-all-databases
--restore-permissions", "delta": "0:01:17.663631",
"end": "2020-07-15 12:35:28.488261", "msg": "non-zero
return code", "rc": 1, "start": "2020-07-15
12:34:10.824630", "stderr": "FATAL: Existing database
'ovirt_engine_history' or user 'ovirt_engine_history' found and temporary
ones created - Please clean up everything and try again", "stderr_lines":
["FATAL: Existing database 'ovirt_engine_history' or user
'ovirt_engine_history' found and temporary ones created - Please clean up
everything and try again"], "stdout": "Start of engine-backup with
mode 'restore'\nscope: all\narchive file: /root/engine_backup\nlog file:
/var/log/ov
irt-engine/setup/restore-backup-20200715103410.log\nPreparing to restore:\n- Unpacking
file '/root/engine_backup'\nRestoring:\n-
Files\n------------------------------------------------------------------------------\nPlease
note:\n\nOperating system is different from the one used during backup.\nCurrent operating
system: centos8\nOperating system at backup: centos7\n\nApache httpd configuration will
not be restored.\nYou will be asked about it on the next engine-setup
run.\n------------------------------------------------------------------------------\nProvisioning
PostgreSQL users/databases:\n- user 'engine', database 'engine'\n- extra
user 'ovirt_engine_history' having grants on database engine, created with a
random password\n- user 'ovirt_engine_history', database
'ovirt_engine_history'", "stdout_lines": ["Start of
engine-backup with mode 'restore'", "scope: all", "archive
file: /root/engine_backup", "log file:
/var/log/ovirt-engine/setup/restore-backup-20200715103410.log", "Prep
aring to restore:", "- Unpacking file '/root/engine_backup'",
"Restoring:", "- Files",
"------------------------------------------------------------------------------",
"Please note:", "", "Operating system is different from the one
used during backup.", "Current operating system: centos8", "Operating
system at backup: centos7", "", "Apache httpd configuration will not
be restored.", "You will be asked about it on the next engine-setup run.",
"------------------------------------------------------------------------------",
"Provisioning PostgreSQL users/databases:", "- user 'engine',
database 'engine'", "- extra user 'ovirt_engine_history' having
grants on database engine, created with a random password", "- user
'ovirt_engine_history', database 'ovirt_engine_history'"]}
restore-backup-xx.log file on the created Hosted Engine (retrieved by Ansible after
failing):
2020-07-15 15:57:36 6437: Start of engine-backup mode restore scope all file
/root/engine_backup
2020-07-15 15:57:36 6437: OUTPUT: Start of engine-backup with mode 'restore'
2020-07-15 15:57:36 6437: OUTPUT: scope: all
2020-07-15 15:57:36 6437: OUTPUT: archive file: /root/engine_backup
2020-07-15 15:57:36 6437: OUTPUT: log file:
/var/log/ovirt-engine/setup/restore-backup-20200715135736.log
2020-07-15 15:57:36 6437: OUTPUT: Preparing to restore:
2020-07-15 15:57:36 6437: OUTPUT: - Unpacking file '/root/engine_backup'
2020-07-15 15:57:36 6437: Opening tarball /root/engine_backup to
/tmp/engine-backup.5tvwGDx3qs
2020-07-15 15:57:37 6437: Verifying hash
2020-07-15 15:57:37 6437: Verifying version
2020-07-15 15:57:37 6437: Reading config
2020-07-15 15:57:37 6437: OUTPUT: Restoring:
2020-07-15 15:57:37 6437: OUTPUT: - Files
2020-07-15 15:57:37 6437: Restoring files
2020-07-15 15:57:38 6437: Reloading configuration
2020-07-15 15:57:38 6437: OUTPUT: Provisioning PostgreSQL users/databases:
2020-07-15 15:57:38 6437: provisionDB: user engine host localhost port 5432 database
engine secured False secured_host_validation False
2020-07-15 15:57:38 6437: OUTPUT: - user 'engine', database 'engine'
[ INFO ] Stage: Initializing
[ INFO ] Stage: Environment setup
Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf,
/etc/ovirt-engine-setup.conf.d/10-packaging.conf,
/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf,
/tmp/engine-backup.5tvwGDx3qs/pg-provision-answer-file
Log file:
/var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20200715155739-3o20a7.log
Version: otopi-1.9.2 (otopi-1.9.2-1.el8)
[ INFO ] Stage: Environment packages setup
[ INFO ] Stage: Programs detection
[ INFO ] Stage: Environment customization
[ INFO ] Stage: Setup validation
[ INFO ] Stage: Transaction setup
[ INFO ] Stage: Misc configuration (early)
[ INFO ] Stage: Package installation
[ INFO ] Stage: Misc configuration
[ INFO ] Initializing PostgreSQL
[ INFO ] Creating PostgreSQL 'engine' database
[ INFO ] Configuring PostgreSQL
[ INFO ] Install selinux module
/usr/share/ovirt-engine/selinux/ansible-runner-service.cil
[ INFO ] Stage: Transaction commit
[ INFO ] Stage: Closing up
[ INFO ] Stage: Clean up
Log file is located at
/var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20200715155739-3o20a7.log
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ INFO ] Execution of provisiondb completed successfully
2020-07-15 15:58:14 6437: OUTPUT: - extra user 'ovirt_engine_history' having
grants on database engine, created with a random password
[ INFO ] Stage: Initializing
[ INFO ] Stage: Environment setup
Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf,
/etc/ovirt-engine-setup.conf.d/10-packaging.conf,
/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf,
/tmp/engine-backup.5tvwGDx3qs/pg-provision-answer-file
Log file:
/var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20200715155815-j4ba3l.log
Version: otopi-1.9.2 (otopi-1.9.2-1.el8)
[ INFO ] Stage: Environment packages setup
[ INFO ] Stage: Programs detection
[ INFO ] Stage: Environment customization
[ INFO ] Stage: Setup validation
[ INFO ] Stage: Transaction setup
[ INFO ] Stage: Misc configuration (early)
[ INFO ] Stage: Package installation
[ INFO ] Stage: Misc configuration
[ INFO ] Install selinux module
/usr/share/ovirt-engine/selinux/ansible-runner-service.cil
[ INFO ] Stage: Transaction commit
[ INFO ] Stage: Closing up
[ INFO ] Stage: Clean up
Log file is located at
/var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20200715155815-j4ba3l.log
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ INFO ] Execution of provisiondb completed successfully
2020-07-15 15:58:46 6437: provisionDB: user ovirt_engine_history host localhost port 5432
database ovirt_engine_history secured False secured_host_validation False
2020-07-15 15:58:46 6437: OUTPUT: - user 'ovirt_engine_history', database
'ovirt_engine_history'
[ INFO ] Stage: Initializing
[ INFO ] Stage: Environment setup
Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf,
/etc/ovirt-engine-setup.conf.d/10-packaging.conf,
/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf,
/tmp/engine-backup.5tvwGDx3qs/pg-provision-answer-file
Log file:
/var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20200715155847-qk7ipy.log
Version: otopi-1.9.2 (otopi-1.9.2-1.el8)
[ INFO ] Stage: Environment packages setup
[ INFO ] Stage: Programs detection
[ INFO ] Stage: Environment customization
[ INFO ] Stage: Setup validation
[ INFO ] Stage: Transaction setup
[ INFO ] Stage: Misc configuration (early)
[ INFO ] Stage: Package installation
[ INFO ] Stage: Misc configuration
[ INFO ] Creating PostgreSQL 'ovirt_engine_history' database
[ INFO ] Configuring PostgreSQL
[ ERROR ] Failed to execute stage 'Misc configuration': Existing resources found,
new ones created:
database ovirt_engine_history_20200715155850 user
ovirt_engine_history_20200715155850
[ INFO ] Stage: Clean up
Log file is located at
/var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20200715155847-qk7ipy.log
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ ERROR ] Execution of provisiondb failed
2020-07-15 15:58:52 6437: FATAL: Existing database 'ovirt_engine_history' or user
'ovirt_engine_history' found and temporary ones created - Please clean up
everything and try again
9:30 a.m.
New subject: Upgrade from 4.3 to 4.4 fails with db or user ovirt_engine_history already exists
A small update: After checking the back-upped database structure (searching for
'ovirt_engine_history' inside the 'engine' database), we found that the
ovirt_engine_history user was also having permissions to a table in the engine database,
probably causing the user to be created before the restoration of the history database
took place. After removing the user permissions in the active environment (4.3) we were
able to bypass this problem.
9:35 a.m.
New subject: Upgrade from 4.3 to 4.4 fails with db or user ovirt_engine_history already exists
On Tue, Jul 21, 2020 at 4:32 PM None via Users <users(a)ovirt.org> wrote:
A small update: After checking the back-upped database structure (searching for
'ovirt_engine_history' inside the 'engine' database), we found that the
ovirt_engine_history user was also having permissions to a table in the engine database,
probably causing the user to be created before the restoration of the history database
took place. After removing the user permissions in the active environment (4.3) we were
able to bypass this problem.
Thanks a lot for the report!
Any idea how this happened (history user having permissions on engine db table)?
Best regards,
--
Didi
9:58 a.m.
New subject: Upgrade from 4.3 to 4.4 fails with db or user ovirt_engine_history already exists
Hi Didi,
I don't know. We are were running the same oVirt instance since 2017 and updated it a
lot of times, including solving some bugs/problems during the years. The user was set on
the 'vm_device_history'-table and the 'disk_vm_device_history_seq'
sequence. Maybe we did it ourselves, but I couldn't find anything about it in our
logs.
Since we are new here: After this problem, we ran into some other problems. Do you want me
to post an article about it, including the steps we have taken to solve them?
We got stuck on the latest problem, and we stopped our upgrade to 4.4 for now, but it
could be related to the other thread about Storage Domains. Running the upgrade and
accessing the 4.4 GUI before continuing (you can set this option at the beginning), we are
unable to update any OVF Disks (automatically or forced), and the active SPM host keeps
resetting (also after changing it to the new 4.4 host). It does not matter if we try to
update an existing SD, the old hosted engine domain, or add a new one (NFS or Gluster).
When we ignore the problem, the installation fails when it checks the health of the new
hosted engine domain. Restarting the old HE again, and the problem is (luckily) solved and
not permanent - so I guess it is a HE 4.4 specific problem. I analyzed the logs but
couldn't find anything that made sense (on the host, HE VM, or another 4.3 host
running SPM). The error(s):
- Failed to update VMs/Templates OVF data for storage domain XX in Data Center Default
- Failed to update OVF disks xGUIDx, xGUIDx, OVF data isn't updated on those OVF
stores (Data Center Default, Storage domain XX)
The error disappeared on most SDs (except the old HE domain, or any new added one) when we
force an OVF update via the 4.3 GUI (before starting the upgrade). The old HE VM was
turned off (and the platform set to global maintenance) before we started the upgrade.
1:35 p.m.
New subject: Upgrade from 4.3 to 4.4 fails with db or user ovirt_engine_history already exists
Hi wilderink＠triplon.nl
Could you please share the steps that you usage to do the upgrade from oVirt 4.3.x to
4.4.x ?
I dont get success in my previous tests, once the current oVirt documentation show the
steps to Upgrade from Standalone Engine and not Hosted Engine.
Regards
Carlos
1391
days inactive
1398
days old
4 comments
3 participants
participants (3)
-
Carlos C -
wilderink＠triplon.nl -
Yedidyah Bar David