On Sat, Apr 27, 2019 at 3:21 AM <magnus@boden.one> wrote:

Hello,

I had a ovirt-engine-4.3.0 that I am migrating to a new machine. I have done a backup of that machine but the restore fails on the new machine.

Fresh install of centos and the 4.3.3 ovirt-engine.

[root@oengine ~]# engine-backup --mode=restore --file=ovirt-engine-backup-20190426214711.backup --log=02.17.log --provision-db --provision-dwh-db --restore-permissions Start of engine-backup with mode 'restore' scope: all archive file: ovirt-engine-backup-20190426214711.backup log file: 02.17.log Preparing to restore: - Unpacking file 'ovirt-engine-backup-20190426214711.backup' Restoring: - Files Provisioning PostgreSQL users/databases: - user 'engine', database 'engine' - user 'ovirt_engine_history', database 'ovirt_engine_history' Restoring: - Engine database 'engine' FATAL: Errors while restoring database engine

[root@oengine ~]# cat 02.17.log 2019-04-27 02:17:55 32343: Start of engine-backup mode restore scope all file ovirt-engine-backup-20190426214711.backup 2019-04-27 02:17:55 32343: OUTPUT: Start of engine-backup with mode 'restore' 2019-04-27 02:17:55 32343: OUTPUT: scope: all 2019-04-27 02:17:55 32343: OUTPUT: archive file: ovirt-engine-backup-20190426214711.backup 2019-04-27 02:17:55 32343: OUTPUT: log file: 02.17.log 2019-04-27 02:17:55 32343: Setting scl env for rh-postgresql10 2019-04-27 02:17:55 32343: OUTPUT: Preparing to restore: 2019-04-27 02:17:55 32343: OUTPUT: - Unpacking file 'ovirt-engine-backup-20190426214711.backup' 2019-04-27 02:17:55 32343: Opening tarball ovirt-engine-backup-20190426214711.backup to /tmp/engine-backup.NVv4Bxjb2k 2019-04-27 02:17:55 32343: Verifying hash 2019-04-27 02:17:55 32343: Verifying version 2019-04-27 02:17:55 32343: Reading config 2019-04-27 02:17:55 32343: OUTPUT: Restoring: 2019-04-27 02:17:55 32343: OUTPUT: - Files 2019-04-27 02:17:55 32343: Restoring files 2019-04-27 02:17:56 32343: Reloading configuration 2019-04-27 02:17:56 32343: OUTPUT: Provisioning PostgreSQL users/databases: 2019-04-27 02:17:56 32343: provisionDB: user engine host localhost port 5432 database engine secured False secured_host_validation False 2019-04-27 02:17:56 32343: OUTPUT: - user 'engine', database 'engine' [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf', '/tmp/engine-backup.NVv4Bxjb2k/pg-provision-answer-file'] Log file: /var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20190427021756-hxbrea.log Version: otopi-1.8.1 (otopi-1.8.1-1.el7) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment customization [ INFO ] Stage: Setup validation [ INFO ] Stage: Transaction setup [ INFO ] Stage: Misc configuration (early) [ INFO ] Stage: Package installation [ INFO ] Stage: Misc configuration [ INFO ] Creating PostgreSQL 'engine' database [ INFO ] Configuring PostgreSQL [ INFO ] Stage: Transaction commit [ INFO ] Stage: Closing up [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20190427021756-hxbrea.log [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ INFO ] Execution of provisiondb completed successfully 2019-04-27 02:17:58 32343: provisionDB: user ovirt_engine_history host localhost port 5432 database ovirt_engine_history secured False secured_host_validation False 2019-04-27 02:17:58 32343: OUTPUT: - user 'ovirt_engine_history', database 'ovirt_engine_history' [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf', '/tmp/engine-backup.NVv4Bxjb2k/pg-provision-answer-file'] Log file: /var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20190427021758-s2av1b.log Version: otopi-1.8.1 (otopi-1.8.1-1.el7) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment customization [ INFO ] Stage: Setup validation [ INFO ] Stage: Transaction setup [ INFO ] Stage: Misc configuration (early) [ INFO ] Stage: Package installation [ INFO ] Stage: Misc configuration [ INFO ] Creating PostgreSQL 'ovirt_engine_history' database [ INFO ] Configuring PostgreSQL [ INFO ] Stage: Transaction commit [ INFO ] Stage: Closing up [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-provisiondb-20190427021758-s2av1b.log [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ INFO ] Execution of provisiondb completed successfully 2019-04-27 02:18:00 32343: OUTPUT: Restoring: 2019-04-27 02:18:00 32343: Generating pgpass 2019-04-27 02:18:00 32343: Verifying connection 2019-04-27 02:18:00 32343: pg_cmd running: psql -w -U engine -h localhost -p 5432 engine -c select 1 ?column? ---------- 1 (1 row)

2019-04-27 02:18:00 32343: pg_cmd running: psql -w -U engine -h localhost -p 5432 engine -t -c show lc_messages 2019-04-27 02:18:00 32343: pg_cmd running: pg_dump -w -U engine -h localhost -p 5432 engine -s 2019-04-27 02:18:01 32343: pg_cmd running: psql -w -U ovirt_engine_history -h localhost -p 5432 ovirt_engine_history -c select 1 ?column? ---------- 1 (1 row)

2019-04-27 02:18:01 32343: pg_cmd running: psql -w -U ovirt_engine_history -h localhost -p 5432 ovirt_engine_history -t -c show lc_messages 2019-04-27 02:18:01 32343: pg_cmd running: pg_dump -w -U ovirt_engine_history -h localhost -p 5432 ovirt_engine_history -s 2019-04-27 02:18:01 32343: OUTPUT: - Engine database 'engine' 2019-04-27 02:18:01 32343: Restoring engine database backup at /tmp/engine-backup.NVv4Bxjb2k/db/engine_backup.db 2019-04-27 02:18:01 32343: restoreDB: backupfile /tmp/engine-backup.NVv4Bxjb2k/db/engine_backup.db user engine host localhost port 5432 database engine orig_user compressor format custom jobsnum 2 2019-04-27 02:18:01 32343: pg_cmd running: pg_restore -w -U engine -h localhost -p 5432 -d engine -j 2 /tmp/engine-backup.NVv4Bxjb2k/db/engine_backup.db pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC entry 7624; 0 0 COMMENT EXTENSION plpgsql pg_restore: [archiver (db)] could not execute query: ERROR: must be owner of extension plpgsql Command was: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';

pg_restore: [archiver (db)] Error from TOC entry 7625; 0 0 COMMENT EXTENSION "uuid-ossp" pg_restore: [archiver (db)] could not execute query: ERROR: must be owner of extension uuid-ossp Command was: COMMENT ON EXTENSION "uuid-ossp" IS 'generate universally unique identifiers (UUIDs)';

pg_restore: [archiver (db)] Error from TOC entry 6400; 2606 18298 CONSTRAINT external_variable pk_external_variable engine pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique index "pk_external_variable" DETAIL: Key (var_name)=(fence-kdump-listener-heartbeat) is duplicated. Command was: ALTER TABLE ONLY public.external_variable ADD CONSTRAINT pk_external_variable PRIMARY KEY (var_name);

This means that in your source database, and therefore in the backup, you have two rows in the table 'external_variable' with var_name 'fence-kdump-listener-heartbeat'. 1. How to fix this, currently? If you still have access to the old database, try checking this table, these two columns, then try guessing which one is needed, and remove the other. Or perhaps remove both, if ovirt-fence-kdump-listener can re-add it. No idea, adding Martin. If you only have the backup file, I guess you'll somehow have to manually fix its content. Didn't try that myself, but I think you can try something like this: 1.1. pg_restore should not have failed in the middle, only exited with errors. So it did restore everything except for the constraint. So start with the existing restored database, remove the offending row, (re-)create the constraint, then run backup, untar the result, also untar the original backup elsewhere, then copy the "fixed" engine db dump to replace the one in the original backup, tar it back up, then try to clean up the database and restore again. 1.2. Alternatively, patch engine-backup for now to ignore this error (search for "IGNORED_ERRORS"), then try again (after cleaning up db), then add the constraint after restoring. Please tell us if you must go this way (not having the old engine anymore) and need help. 2. Not sure why the constraint didn't prevent adding the second row in the first place - it constraint existed "forever" (was added together with the table, not after it, in 3.5, bz 1079821). Adding Eli. 3. Not sure what flow caused adding the second row. Briefly looking at the code that adds it, can't find something obvious. You might give us a clue by checking the _update_date col in that table, and then check relevant logs around the time it was added. Sorry if this isn't very helpful. It's quite a weird problem.

WARNING: errors ignored on restore: 4 2019-04-27 02:18:43 32343: Non-ignored-errors in pg_restore log: pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique index "pk_external_variable" 2019-04-27 02:18:43 32343: FATAL: Errors while restoring database engine

If this is a problem while restoring a 4.3.0 backup to a 4.3.3 engine is it possible to install the 4.3.0 version? Following the installation part of the release notes of 4.3.0 will result in 4.3.3 being installed.

I don't think it's related. I think it would have failed with any other version the same way. Best regards, -- Didi