Please help: Failure Restoring Data on Clean Engine After Migration

Hi, I have oVirt engine 4.4.7 running on dedicated PC (not hosted engine). After several unsuccessful upgrade attempts of 4.4.7 to 4.4.10 decided to install clean 4.4.10 and migrate data. On old engine engine-backup --scope=all --mode=backup On new engine engine-backup --mode=restore --provision-all-databases --no-restore-permissions --file=ovirt-engine-backup-20220713160717.backup Result: [ ERROR ] Failed to execute stage 'Environment setup': Cannot connect to database for grafana using existing credentials: ovirt_engine_history_grafana@localhost:5432 How to Remove Grafana Completely During Engine Migration, or skip it during backup? In fact it would be nice to delete it on active 4.4.7 setup, I don’t need it anyway. Thanks in advance. # — LOG ——————————————————————— Start of engine-backup with mode 'restore' scope: all archive file: ovirt-engine-backup-20220713160717.backup log file: /var/log/ovirt-engine-backup/ovirt-engine-restore-20220714170603.log Preparing to restore: - Unpacking file 'ovirt-engine-backup-20220713160717.backup' Restoring: - Files Provisioning PostgreSQL users/databases: - user 'engine', database 'engine' - user 'ovirt_engine_history', database 'ovirt_engine_history' Restoring: - Engine database 'engine' - Cleaning up temporary tables in engine database 'engine' - Updating DbJustRestored VdcOption in engine database - Resetting DwhCurrentlyRunning in dwh_history_timekeeping in engine database - Resetting HA VM status ------------------------------------------------------------------------------ Please note: The engine database was backed up at 2022-07-13 16:07:21.000000000 +0300 . Objects that were added, removed or changed after this date, such as virtual machines, disks, etc., are missing in the engine, and will probably require recovery or recreation. ------------------------------------------------------------------------------ - DWH database 'ovirt_engine_history' - Grafana database '/var/lib/grafana/grafana.db' You should now run engine-setup. Done. [root@node00 ovirt-engine-backup]# engine-setup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20220714170757-7072xx.log Version: otopi-1.9.6 (otopi-1.9.6-1.el8) [ INFO ] The engine DB has been restored from a backup [ ERROR ] Failed to execute stage 'Environment setup': Cannot connect to database for grafana using existing credentials: ovirt_engine_history_grafana@localhost:5432 [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20220714170757-7072xx.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20220714170806-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed

I had a similar issue. for me, taking the password from /etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/10-setup-grafana-database.conf (GRAFANA_DB_PASSWORD) and set that password in postgres for the user ovirt_engine_history_grafana did the trick. Best Mo On 7/14/22 16:28, Andrei Verovski wrote:
Hi,
I have oVirt engine 4.4.7 running on dedicated PC (not hosted engine).
After several unsuccessful upgrade attempts of 4.4.7 to 4.4.10 decided to install clean 4.4.10 and migrate data.
On old engine engine-backup --scope=all --mode=backup
On new engine engine-backup --mode=restore --provision-all-databases --no-restore-permissions --file=ovirt-engine-backup-20220713160717.backup
Result: [ ERROR ] Failed to execute stage 'Environment setup': Cannot connect to database for grafana using existing credentials: ovirt_engine_history_grafana@localhost:5432
How to Remove Grafana Completely During Engine Migration, or skip it during backup? In fact it would be nice to delete it on active 4.4.7 setup, I don’t need it anyway.
Thanks in advance.
# — LOG ———————————————————————
Start of engine-backup with mode 'restore' scope: all archive file: ovirt-engine-backup-20220713160717.backup log file: /var/log/ovirt-engine-backup/ovirt-engine-restore-20220714170603.log Preparing to restore: - Unpacking file 'ovirt-engine-backup-20220713160717.backup' Restoring: - Files Provisioning PostgreSQL users/databases: - user 'engine', database 'engine' - user 'ovirt_engine_history', database 'ovirt_engine_history' Restoring: - Engine database 'engine' - Cleaning up temporary tables in engine database 'engine' - Updating DbJustRestored VdcOption in engine database - Resetting DwhCurrentlyRunning in dwh_history_timekeeping in engine database - Resetting HA VM status ------------------------------------------------------------------------------ Please note:
The engine database was backed up at 2022-07-13 16:07:21.000000000 +0300 .
Objects that were added, removed or changed after this date, such as virtual machines, disks, etc., are missing in the engine, and will probably require recovery or recreation. ------------------------------------------------------------------------------ - DWH database 'ovirt_engine_history' - Grafana database '/var/lib/grafana/grafana.db' You should now run engine-setup. Done. [root@node00 ovirt-engine-backup]# engine-setup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20220714170757-7072xx.log Version: otopi-1.9.6 (otopi-1.9.6-1.el8) [ INFO ] The engine DB has been restored from a backup [ ERROR ] Failed to execute stage 'Environment setup': Cannot connect to database for grafana using existing credentials: ovirt_engine_history_grafana@localhost:5432 [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20220714170757-7072xx.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20220714170806-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/7JORR5ZWZJTGPF...

On 7/14/22 16:37, Moritz Baumann wrote:
I had a similar issue.
for me, taking the password from /etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/10-setup-grafana-database.conf (GRAFANA_DB_PASSWORD)
and set that password in postgres for the user ovirt_engine_history_grafana did the trick.
su - postgres -s/bin/bash psql \password ovirt_engine_history_grafana # enter the password from that file

Hi, I did this and still struck at that Grafana stage. CREATE ROLE ovirt_engine_history_grafana; ALTER DEFAULT PRIVILEGES FOR ROLE ovirt_engine_history IN SCHEMA public GRANT SELECT ON TABLES TO ovirt_engine_history_grafana; ALTER ROLE ovirt_engine_history_grafana WITH PASSWORD ‘my_password’; How to delete Grafana completely from old setup??? I don’t need it. Thanks in advance.
On 14 Jul 2022, at 17:37, Moritz Baumann <moritz.baumann@inf.ethz.ch> wrote:
I had a similar issue.
for me, taking the password from /etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/10-setup-grafana-database.conf (GRAFANA_DB_PASSWORD)
and set that password in postgres for the user ovirt_engine_history_grafana did the trick.
Best Mo
On 7/14/22 16:28, Andrei Verovski wrote:
Hi, I have oVirt engine 4.4.7 running on dedicated PC (not hosted engine). After several unsuccessful upgrade attempts of 4.4.7 to 4.4.10 decided to install clean 4.4.10 and migrate data. On old engine engine-backup --scope=all --mode=backup On new engine engine-backup --mode=restore --provision-all-databases --no-restore-permissions --file=ovirt-engine-backup-20220713160717.backup Result: [ ERROR ] Failed to execute stage 'Environment setup': Cannot connect to database for grafana using existing credentials: ovirt_engine_history_grafana@localhost:5432 How to Remove Grafana Completely During Engine Migration, or skip it during backup? In fact it would be nice to delete it on active 4.4.7 setup, I don’t need it anyway. Thanks in advance. # — LOG ——————————————————————— Start of engine-backup with mode 'restore' scope: all archive file: ovirt-engine-backup-20220713160717.backup log file: /var/log/ovirt-engine-backup/ovirt-engine-restore-20220714170603.log Preparing to restore: - Unpacking file 'ovirt-engine-backup-20220713160717.backup' Restoring: - Files Provisioning PostgreSQL users/databases: - user 'engine', database 'engine' - user 'ovirt_engine_history', database 'ovirt_engine_history' Restoring: - Engine database 'engine' - Cleaning up temporary tables in engine database 'engine' - Updating DbJustRestored VdcOption in engine database - Resetting DwhCurrentlyRunning in dwh_history_timekeeping in engine database - Resetting HA VM status ------------------------------------------------------------------------------ Please note: The engine database was backed up at 2022-07-13 16:07:21.000000000 +0300 . Objects that were added, removed or changed after this date, such as virtual machines, disks, etc., are missing in the engine, and will probably require recovery or recreation. ------------------------------------------------------------------------------ - DWH database 'ovirt_engine_history' - Grafana database '/var/lib/grafana/grafana.db' You should now run engine-setup. Done. [root@node00 ovirt-engine-backup]# engine-setup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20220714170757-7072xx.log Version: otopi-1.9.6 (otopi-1.9.6-1.el8) [ INFO ] The engine DB has been restored from a backup [ ERROR ] Failed to execute stage 'Environment setup': Cannot connect to database for grafana using existing credentials: ovirt_engine_history_grafana@localhost:5432 [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20220714170757-7072xx.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20220714170806-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/7JORR5ZWZJTGPF...
Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZZOUZX65Q2EADS...

On Fri, Jul 15, 2022 at 10:31 AM Andrei Verovski <andreil1@starlett.lv> wrote:
Hi,
I did this and still struck at that Grafana stage.
CREATE ROLE ovirt_engine_history_grafana; ALTER DEFAULT PRIVILEGES FOR ROLE ovirt_engine_history IN SCHEMA public GRANT SELECT ON TABLES TO ovirt_engine_history_grafana; ALTER ROLE ovirt_engine_history_grafana WITH PASSWORD ‘my_password’;
You are probably missing pg_hba.conf configuration, see e.g. https://www.ovirt.org/documentation/data_warehouse_guide/#Allowing_Read_Only... .
How to delete Grafana completely from old setup???
I don't think we have this documented anywhere. If you only want to get rid of the setup issue, it's probably enough to edit /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf, changing the line 'OVESETUP_GRAFANA_CORE/enable=bool:True' to 'OVESETUP_GRAFANA_CORE/enable=bool:False'. This will not "delete Grafana completely", only make engine-setup ignore it.
I don’t need it.
Thanks in advance.
On 14 Jul 2022, at 17:37, Moritz Baumann <moritz.baumann@inf.ethz.ch> wrote:
I had a similar issue.
for me, taking the password from /etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/10-setup-grafana-database.conf (GRAFANA_DB_PASSWORD)
and set that password in postgres for the user ovirt_engine_history_grafana did the trick.
Best Mo
On 7/14/22 16:28, Andrei Verovski wrote:
Hi, I have oVirt engine 4.4.7 running on dedicated PC (not hosted engine). After several unsuccessful upgrade attempts of 4.4.7 to 4.4.10 decided to install clean 4.4.10 and migrate data. On old engine engine-backup --scope=all --mode=backup On new engine engine-backup --mode=restore --provision-all-databases --no-restore-permissions --file=ovirt-engine-backup-20220713160717.backup
I am sorry to note that your issue was most likely caused by '--no-restore-permissions', although the documentation (including --help/manpage) does not hint about this at all. You might want to open a doc bug to document this, or even an RFE bug, to make this a separate option. for a long time, it was mandatory to pass either --no-restore-permissions or --restore-permissions: https://bugzilla.redhat.com/show_bug.cgi?id=1220791 But I recently changed this to default to --restore-permissions: https://bugzilla.redhat.com/1821018 With --restore-permissions, if you previously manually created extra users and gave them access permissions, e.g. using the doc in above link, --mode=restore could not know the passwords for these users, and created them with random passwords, outputting "- extra user '${extrau}' having grants on database ${database}, created with a random password": https://bugzilla.redhat.com/1369757 But for grafana, this isn't true - the password is saved in the above-mentioned conf, and so --mode=restore can (and does) create the user with the saved password: https://bugzilla.redhat.com/show_bug.cgi?id=1837460 Bottom line: I now think that --restore-permissions almost always makes sense, therefore changed it to be the default. If you have scripts/procedures that pass --no-restore-permissions, I recommend rethinking these and considering dropping it altogether, relying on the default, or passing --restore-permissions. A scenario I can think of where '--no-restore-permissions' does make sense: If you do have extra users you created for some other applications to access the DWH DB, and would rather not have a restore procedure replace their passwords to random ones, but prefer having your restore procedure handle this manually - restore/setup with --no-restore-permissions, then manually add the users+passwords you need and give them permissions. Best regards, -- Didi

Hi, Thanks, I solved this problem before by means of executing these commands: su - postgres -s/bin/bash psql -d ovirt_engine_history CREATE ROLE ovirt_engine_history_grafana; ALTER DEFAULT PRIVILEGES FOR ROLE ovirt_engine_history IN SCHEMA public GRANT SELECT ON TABLES TO ovirt_engine_history_grafana; ALTER ROLE ovirt_engine_history_grafana WITH ENCRYPTED PASSWORD ‘my_password'; ALTER ROLE ovirt_engine_history_grafana WITH LOGIN; my_password from: /etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/10-setup-grafana-database.conf
On 18 Jul 2022, at 15:35, Yedidyah Bar David <didi@redhat.com> wrote:
On Fri, Jul 15, 2022 at 10:31 AM Andrei Verovski <andreil1@starlett.lv> wrote:
Hi,
I did this and still struck at that Grafana stage.
CREATE ROLE ovirt_engine_history_grafana; ALTER DEFAULT PRIVILEGES FOR ROLE ovirt_engine_history IN SCHEMA public GRANT SELECT ON TABLES TO ovirt_engine_history_grafana; ALTER ROLE ovirt_engine_history_grafana WITH PASSWORD ‘my_password’;
You are probably missing pg_hba.conf configuration, see e.g. https://www.ovirt.org/documentation/data_warehouse_guide/#Allowing_Read_Only... .
How to delete Grafana completely from old setup???
I don't think we have this documented anywhere.
If you only want to get rid of the setup issue, it's probably enough to edit /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf, changing the line 'OVESETUP_GRAFANA_CORE/enable=bool:True' to 'OVESETUP_GRAFANA_CORE/enable=bool:False'.
This will not "delete Grafana completely", only make engine-setup ignore it.
I don’t need it.
Thanks in advance.
On 14 Jul 2022, at 17:37, Moritz Baumann <moritz.baumann@inf.ethz.ch> wrote:
I had a similar issue.
for me, taking the password from /etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/10-setup-grafana-database.conf (GRAFANA_DB_PASSWORD)
and set that password in postgres for the user ovirt_engine_history_grafana did the trick.
Best Mo
On 7/14/22 16:28, Andrei Verovski wrote:
Hi, I have oVirt engine 4.4.7 running on dedicated PC (not hosted engine). After several unsuccessful upgrade attempts of 4.4.7 to 4.4.10 decided to install clean 4.4.10 and migrate data. On old engine engine-backup --scope=all --mode=backup On new engine engine-backup --mode=restore --provision-all-databases --no-restore-permissions --file=ovirt-engine-backup-20220713160717.backup
I am sorry to note that your issue was most likely caused by '--no-restore-permissions', although the documentation (including --help/manpage) does not hint about this at all. You might want to open a doc bug to document this, or even an RFE bug, to make this a separate option.
for a long time, it was mandatory to pass either --no-restore-permissions or --restore-permissions:
https://bugzilla.redhat.com/show_bug.cgi?id=1220791
But I recently changed this to default to --restore-permissions:
https://bugzilla.redhat.com/1821018
With --restore-permissions, if you previously manually created extra users and gave them access permissions, e.g. using the doc in above link, --mode=restore could not know the passwords for these users, and created them with random passwords, outputting "- extra user '${extrau}' having grants on database ${database}, created with a random password":
https://bugzilla.redhat.com/1369757
But for grafana, this isn't true - the password is saved in the above-mentioned conf, and so --mode=restore can (and does) create the user with the saved password:
https://bugzilla.redhat.com/show_bug.cgi?id=1837460
Bottom line:
I now think that --restore-permissions almost always makes sense, therefore changed it to be the default.
If you have scripts/procedures that pass --no-restore-permissions, I recommend rethinking these and considering dropping it altogether, relying on the default, or passing --restore-permissions.
A scenario I can think of where '--no-restore-permissions' does make sense: If you do have extra users you created for some other applications to access the DWH DB, and would rather not have a restore procedure replace their passwords to random ones, but prefer having your restore procedure handle this manually - restore/setup with --no-restore-permissions, then manually add the users+passwords you need and give them permissions.
Best regards, -- Didi

On Tue, Jul 19, 2022 at 9:38 AM Andrei Verovski <andreil1@starlett.lv> wrote:
Hi,
Thanks, I solved this problem before by means of executing these commands:
su - postgres -s/bin/bash psql -d ovirt_engine_history
CREATE ROLE ovirt_engine_history_grafana; ALTER DEFAULT PRIVILEGES FOR ROLE ovirt_engine_history IN SCHEMA public GRANT SELECT ON TABLES TO ovirt_engine_history_grafana; ALTER ROLE ovirt_engine_history_grafana WITH ENCRYPTED PASSWORD ‘my_password'; ALTER ROLE ovirt_engine_history_grafana WITH LOGIN;
So this last one, 'LOGIN', was the missing part. OK. Thanks for the update. Glad to hear that you solved it! Any other conclusions/suggestions you'd like to make? Best regards, -- Didi

On 19 Jul 2022, at 09:51, Yedidyah Bar David <didi@redhat.com> wrote:
On Tue, Jul 19, 2022 at 9:38 AM Andrei Verovski <andreil1@starlett.lv> wrote:
Hi,
Thanks, I solved this problem before by means of executing these commands:
su - postgres -s/bin/bash psql -d ovirt_engine_history
CREATE ROLE ovirt_engine_history_grafana; ALTER DEFAULT PRIVILEGES FOR ROLE ovirt_engine_history IN SCHEMA public GRANT SELECT ON TABLES TO ovirt_engine_history_grafana; ALTER ROLE ovirt_engine_history_grafana WITH ENCRYPTED PASSWORD ‘my_password'; ALTER ROLE ovirt_engine_history_grafana WITH LOGIN;
So this last one, 'LOGIN', was the missing part. OK.
Thanks for the update. Glad to hear that you solved it!
Any other conclusions/suggestions you'd like to make?
It was a hell like experience upgrading 4.4.7 to 4.4.10. I will post an instruction in my blog.
Best regards, -- Didi
participants (3)
-
Andrei Verovski
-
Moritz Baumann
-
Yedidyah Bar David