2:23 a.m.
On Sat, Apr 27, 2019 at 3:21 AM <magnus(a)boden.one> wrote:
Hello,
I had a ovirt-engine-4.3.0 that I am migrating to a new machine. I have done a backup of
that machine but the restore fails on the new machine.
Fresh install of centos and the 4.3.3 ovirt-engine.
[root@oengine ~]# engine-backup --mode=restore
--file=ovirt-engine-backup-20190426214711.backup --log=02.17.log --provision-db
--provision-dwh-db --restore-permissions
Start of engine-backup with mode 'restore'
scope: all
archive file: ovirt-engine-backup-20190426214711.backup
log file: 02.17.log
Preparing to restore:
- Unpacking file 'ovirt-engine-backup-20190426214711.backup'
Restoring:
- Files
Provisioning PostgreSQL users/databases:
- user 'engine', database 'engine'
- user 'ovirt_engine_history', database 'ovirt_engine_history'
Restoring:
- Engine database 'engine'
FATAL: Errors while restoring database engine
[root@oengine ~]# cat 02.17.log
2019-04-27 02:17:55 32343: Start of engine-backup mode restore scope all file
ovirt-engine-backup-20190426214711.backup
2019-04-27 02:17:55 32343: OUTPUT: Start of engine-backup with mode 'restore'
2019-04-27 02:17:55 32343: OUTPUT: scope: all
2019-04-27 02:17:55 32343: OUTPUT: archive file:
ovirt-engine-backup-20190426214711.backup
2019-04-27 02:17:55 32343: OUTPUT: log file: 02.17.log
2019-04-27 02:17:55 32343: Setting scl env for rh-postgresql10
2019-04-27 02:17:55 32343: OUTPUT: Preparing to restore:
2019-04-27 02:17:55 32343: OUTPUT: - Unpacking file
'ovirt-engine-backup-20190426214711.backup'
2019-04-27 02:17:55 32343: Opening tarball ovirt-engine-backup-20190426214711.backup to
/tmp/engine-backup.NVv4Bxjb2k
2019-04-27 02:17:55 32343: Verifying hash
2019-04-27 02:17:55 32343: Verifying version
2019-04-27 02:17:55 32343: Reading config
2019-04-27 02:17:55 32343: OUTPUT: Restoring:
2019-04-27 02:17:55 32343: OUTPUT: - Files
2019-04-27 02:17:55 32343: Restoring files
2019-04-27 02:17:56 32343: Reloading configuration
2019-04-27 02:17:56 32343: OUTPUT: Provisioning PostgreSQL users/databases:
2019-04-27 02:17:56 32343: provisionDB: user engine host localhost port 5432 database
engine secured False secured_host_validation False
2019-04-27 02:17:56 32343: OUTPUT: - user 'engine', database 'engine'
[ INFO ] Stage: Initializing
[ INFO ] Stage: Environment setup
Configuration files:
['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf',
'/etc/ovirt-engine-setup.conf.d/10-packaging.conf',
'/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf',
'/tmp/engine-backup.NVv4Bxjb2k/pg-provision-answer-file']
Log file:
/var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20190427021756-hxbrea.log
Version: otopi-1.8.1 (otopi-1.8.1-1.el7)
[ INFO ] Stage: Environment packages setup
[ INFO ] Stage: Programs detection
[ INFO ] Stage: Environment customization
[ INFO ] Stage: Setup validation
[ INFO ] Stage: Transaction setup
[ INFO ] Stage: Misc configuration (early)
[ INFO ] Stage: Package installation
[ INFO ] Stage: Misc configuration
[ INFO ] Creating PostgreSQL 'engine' database
[ INFO ] Configuring PostgreSQL
[ INFO ] Stage: Transaction commit
[ INFO ] Stage: Closing up
[ INFO ] Stage: Clean up
Log file is located at
/var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20190427021756-hxbrea.log
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ INFO ] Execution of provisiondb completed successfully
2019-04-27 02:17:58 32343: provisionDB: user ovirt_engine_history host localhost port
5432 database ovirt_engine_history secured False secured_host_validation False
2019-04-27 02:17:58 32343: OUTPUT: - user 'ovirt_engine_history', database
'ovirt_engine_history'
[ INFO ] Stage: Initializing
[ INFO ] Stage: Environment setup
Configuration files:
['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf',
'/etc/ovirt-engine-setup.conf.d/10-packaging.conf',
'/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf',
'/tmp/engine-backup.NVv4Bxjb2k/pg-provision-answer-file']
Log file:
/var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20190427021758-s2av1b.log
Version: otopi-1.8.1 (otopi-1.8.1-1.el7)
[ INFO ] Stage: Environment packages setup
[ INFO ] Stage: Programs detection
[ INFO ] Stage: Environment customization
[ INFO ] Stage: Setup validation
[ INFO ] Stage: Transaction setup
[ INFO ] Stage: Misc configuration (early)
[ INFO ] Stage: Package installation
[ INFO ] Stage: Misc configuration
[ INFO ] Creating PostgreSQL 'ovirt_engine_history' database
[ INFO ] Configuring PostgreSQL
[ INFO ] Stage: Transaction commit
[ INFO ] Stage: Closing up
[ INFO ] Stage: Clean up
Log file is located at
/var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20190427021758-s2av1b.log
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ INFO ] Execution of provisiondb completed successfully
2019-04-27 02:18:00 32343: OUTPUT: Restoring:
2019-04-27 02:18:00 32343: Generating pgpass
2019-04-27 02:18:00 32343: Verifying connection
2019-04-27 02:18:00 32343: pg_cmd running: psql -w -U engine -h localhost -p 5432 engine
-c select 1
?column?
----------
1
(1 row)
2019-04-27 02:18:00 32343: pg_cmd running: psql -w -U engine -h localhost -p 5432 engine
-t -c show lc_messages
2019-04-27 02:18:00 32343: pg_cmd running: pg_dump -w -U engine -h localhost -p 5432
engine -s
2019-04-27 02:18:01 32343: pg_cmd running: psql -w -U ovirt_engine_history -h localhost
-p 5432 ovirt_engine_history -c select 1
?column?
----------
1
(1 row)
2019-04-27 02:18:01 32343: pg_cmd running: psql -w -U ovirt_engine_history -h localhost
-p 5432 ovirt_engine_history -t -c show lc_messages
2019-04-27 02:18:01 32343: pg_cmd running: pg_dump -w -U ovirt_engine_history -h
localhost -p 5432 ovirt_engine_history -s
2019-04-27 02:18:01 32343: OUTPUT: - Engine database 'engine'
2019-04-27 02:18:01 32343: Restoring engine database backup at
/tmp/engine-backup.NVv4Bxjb2k/db/engine_backup.db
2019-04-27 02:18:01 32343: restoreDB: backupfile
/tmp/engine-backup.NVv4Bxjb2k/db/engine_backup.db user engine host localhost port 5432
database engine orig_user compressor format custom jobsnum 2
2019-04-27 02:18:01 32343: pg_cmd running: pg_restore -w -U engine -h localhost -p 5432
-d engine -j 2 /tmp/engine-backup.NVv4Bxjb2k/db/engine_backup.db
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 7624; 0 0 COMMENT EXTENSION plpgsql
pg_restore: [archiver (db)] could not execute query: ERROR: must be owner of extension
plpgsql
Command was: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
pg_restore: [archiver (db)] Error from TOC entry 7625; 0 0 COMMENT EXTENSION
"uuid-ossp"
pg_restore: [archiver (db)] could not execute query: ERROR: must be owner of extension
uuid-ossp
Command was: COMMENT ON EXTENSION "uuid-ossp" IS 'generate universally
unique identifiers (UUIDs)';
pg_restore: [archiver (db)] Error from TOC entry 6400; 2606 18298 CONSTRAINT
external_variable pk_external_variable engine
pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique
index "pk_external_variable"
DETAIL: Key (var_name)=(fence-kdump-listener-heartbeat) is duplicated.
Command was: ALTER TABLE ONLY public.external_variable
ADD CONSTRAINT pk_external_variable PRIMARY KEY (var_name);
This means that in your source database, and therefore in the backup,
you have two rows in the table 'external_variable' with var_name
'fence-kdump-listener-heartbeat'.
1. How to fix this, currently?
If you still have access to the old database, try checking this table,
these two columns, then try guessing which one is needed, and remove
the other. Or perhaps remove both, if ovirt-fence-kdump-listener can
re-add it. No idea, adding Martin.
If you only have the backup file, I guess you'll somehow have to
manually fix its content. Didn't try that myself, but I think you can
try something like this:
1.1. pg_restore should not have failed in the middle, only exited with
errors. So it did restore everything except for the constraint. So
start with the existing restored database, remove the offending row,
(re-)create the constraint, then run backup, untar the result, also
untar the original backup elsewhere, then copy the "fixed" engine db
dump to replace the one in the original backup, tar it back up, then
try to clean up the database and restore again.
1.2. Alternatively, patch engine-backup for now to ignore this error
(search for "IGNORED_ERRORS"), then try again (after cleaning up db),
then add the constraint after restoring.
Please tell us if you must go this way (not having the old engine
anymore) and need help.
2. Not sure why the constraint didn't prevent adding the second row in
the first place - it constraint existed "forever" (was added together
with the table, not after it, in 3.5, bz 1079821). Adding Eli.
3. Not sure what flow caused adding the second row. Briefly looking at
the code that adds it, can't find something obvious. You might give us
a clue by checking the _update_date col in that table, and then check
relevant logs around the time it was added.
Sorry if this isn't very helpful. It's quite a weird problem.
WARNING: errors ignored on restore: 4
2019-04-27 02:18:43 32343: Non-ignored-errors in pg_restore log:
pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique
index "pk_external_variable"
2019-04-27 02:18:43 32343: FATAL: Errors while restoring database engine
If this is a problem while restoring a 4.3.0 backup to a 4.3.3 engine is it possible to
install the 4.3.0 version?
Following the installation part of the release notes of 4.3.0 will result in 4.3.3 being
installed.
I don't think it's related. I think it would have failed with any
other version the same way.
Best regards,
--
Didi