engine-setup failing on 4.3.2 -> 4.3.3 fails during Engine schema refresh fail

I was trying to upgrade a hyperconverged oVirt hosted engine and failed in the engine-setup command with these error and warnings. ... [ INFO ] Creating/refreshing Engine database schema [ ERROR ] schema.sh: FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/dbscripts/upgrade/04_03_0830_add_foreign_key_to_image_transfers.sql [ ERROR ] Failed to execute stage 'Misc configuration': Engine schema refresh failed ... [ INFO ] Yum Verify: 16/16: ovirt-engine-tools.noarch 0:4.3.3.5-1.el7 - e [WARNING] Rollback of DWH database postponed to Stage "Clean up" [ INFO ] Rolling back database schema ... [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed Attaching engine-setup logfile.

Hi Edward, On 4/16/19 9:23 PM, Edward Berger wrote:
I was trying to upgrade a hyperconverged oVirt hosted engine and failed in the engine-setup command with these error and warnings.
... [ INFO ] Creating/refreshing Engine database schema [ ERROR ] schema.sh: FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/dbscripts/upgrade/04_03_0830_add_foreign_key_to_image_transfers.sql The error is
ERROR: insert or update on table "image_transfers" violates foreign key constraint "fk_image_transfers_command_enitites" DETAIL: Key (command_id)=(fedc82f7-9bf7-4988-b9d9-23853e04e44e) is not present in table "command_entities". So there is some image transfer left without appropriate command entity. If you are sure that no such transfer is active. You can remove it from the table. $ scl enable rh-postgresql10 bash $ psql engine engine=# delete from image_transfers where command_id='fedc82f7-9bf7-4988-b9d9-23853e04e44e';
[ ERROR ] Failed to execute stage 'Misc configuration': Engine schema refresh failed ... [ INFO ] Yum Verify: 16/16: ovirt-engine-tools.noarch 0:4.3.3.5-1.el7 - e [WARNING] Rollback of DWH database postponed to Stage "Clean up" [ INFO ] Rolling back database schema ... [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed
Attaching engine-setup logfile.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/45EDYLIQE4QUJY... Best regards,
-- Lucie Leistnerova Quality Engineer, QE Cloud, RHVM Red Hat EMEA IRC: lleistne @ #rhev-qe

On Wed, Apr 17, 2019 at 10:30 AM Lucie Leistnerova <lleistne@redhat.com> wrote:
Hi Edward,
On 4/16/19 9:23 PM, Edward Berger wrote:
I was trying to upgrade a hyperconverged oVirt hosted engine and failed in the engine-setup command with these error and warnings.
... [ INFO ] Creating/refreshing Engine database schema [ ERROR ] schema.sh: FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/dbscripts/upgrade/04_03_0830_add_foreign_key_to_image_transfers.sql
The error is
ERROR: insert or update on table "image_transfers" violates foreign key constraint "fk_image_transfers_command_enitites" DETAIL: Key (command_id)=(fedc82f7-9bf7-4988-b9d9-23853e04e44e) is not present in table "command_entities".
So there is some image transfer left without appropriate command entity. If you are sure that no such transfer is active. You can remove it from the table.
$ scl enable rh-postgresql10 bash $ psql engine
engine=# delete from image_transfers where command_id='fedc82f7-9bf7-4988-b9d9-23853e04e44e';
Or, more generally, copying from 04_03_0270_add_foreign_key_to_image_transfers.sql: DELETE FROM image_transfers WHERE command_id NOT IN (SELECT command_entities.command_id FROM command_entities); and then try again. Adding Also Bhushan, who reported this in private as well. Adding Daniel, author of the patch causing this: https://gerrit.ovirt.org/#/q/I65906d09ba201c93da65e9a2b67cef5f9f177747,n,z Daniel: This already happened in the past, and we fixed it by adding 04_03_0265_remove_foreign_key_in_image_transfers.sql . Not sure why we needed to add 04_03_0830_add_foreign_key_to_image_transfers.sql again, but perhaps add now another copy of 04_03_0265_remove_foreign_key_in_image_transfers.sql before it as well. Also: Perhaps push a patch to OST to verify this flow. Thanks and best regards,
[ ERROR ] Failed to execute stage 'Misc configuration': Engine schema refresh failed ... [ INFO ] Yum Verify: 16/16: ovirt-engine-tools.noarch 0:4.3.3.5-1.el7 - e [WARNING] Rollback of DWH database postponed to Stage "Clean up" [ INFO ] Rolling back database schema ... [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed
Attaching engine-setup logfile.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/45EDYLIQE4QUJY...
Best regards,
-- Lucie Leistnerova Quality Engineer, QE Cloud, RHVM Red Hat EMEA
IRC: lleistne @ #rhev-qe
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2UT7ANKVQKB5NS...
-- Didi

On Wed, Apr 17, 2019 at 10:48 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Apr 17, 2019 at 10:30 AM Lucie Leistnerova <lleistne@redhat.com> wrote:
Hi Edward,
On 4/16/19 9:23 PM, Edward Berger wrote:
I was trying to upgrade a hyperconverged oVirt hosted engine and failed in the engine-setup command with these error and warnings.
... [ INFO ] Creating/refreshing Engine database schema [ ERROR ] schema.sh: FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/dbscripts/upgrade/04_03_0830_add_foreign_key_to_image_transfers.sql
The error is
ERROR: insert or update on table "image_transfers" violates foreign key constraint "fk_image_transfers_command_enitites" DETAIL: Key (command_id)=(fedc82f7-9bf7-4988-b9d9-23853e04e44e) is not present in table "command_entities".
So there is some image transfer left without appropriate command entity. If you are sure that no such transfer is active. You can remove it from the table.
$ scl enable rh-postgresql10 bash $ psql engine
engine=# delete from image_transfers where command_id='fedc82f7-9bf7-4988-b9d9-23853e04e44e';
Or, more generally, copying from 04_03_0270_add_foreign_key_to_image_transfers.sql:
DELETE FROM image_transfers WHERE command_id NOT IN (SELECT command_entities.command_id FROM command_entities);
and then try again.
Adding Also Bhushan, who reported this in private as well.
Adding Daniel, author of the patch causing this:
https://gerrit.ovirt.org/#/q/I65906d09ba201c93da65e9a2b67cef5f9f177747,n,z
Daniel:
This already happened in the past, and we fixed it by adding 04_03_0265_remove_foreign_key_in_image_transfers.sql . Not sure why we needed to add 04_03_0830_add_foreign_key_to_image_transfers.sql again, but perhaps add now another copy of 04_03_0265_remove_foreign_key_in_image_transfers.sql before it as well.
Alternatively, if we are still not fully certain this is over, consider adding an engine-setup plugin that does that, ignoring common practice of dbscripts upgrade (which ignores files it already loaded in the past).
Also: Perhaps push a patch to OST to verify this flow.
Thanks and best regards,
[ ERROR ] Failed to execute stage 'Misc configuration': Engine schema refresh failed ... [ INFO ] Yum Verify: 16/16: ovirt-engine-tools.noarch 0:4.3.3.5-1.el7 - e [WARNING] Rollback of DWH database postponed to Stage "Clean up" [ INFO ] Rolling back database schema ... [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed
Attaching engine-setup logfile.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/45EDYLIQE4QUJY...
Best regards,
-- Lucie Leistnerova Quality Engineer, QE Cloud, RHVM Red Hat EMEA
IRC: lleistne @ #rhev-qe
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2UT7ANKVQKB5NS...
-- Didi
-- Didi

On Wed, Apr 17, 2019 at 10:51 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Apr 17, 2019 at 10:48 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Apr 17, 2019 at 10:30 AM Lucie Leistnerova <lleistne@redhat.com> wrote:
Hi Edward,
On 4/16/19 9:23 PM, Edward Berger wrote:
I was trying to upgrade a hyperconverged oVirt hosted engine and failed in the engine-setup command with these error and warnings.
... [ INFO ] Creating/refreshing Engine database schema [ ERROR ] schema.sh: FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/dbscripts/upgrade/04_03_0830_add_foreign_key_to_image_transfers.sql
The error is
ERROR: insert or update on table "image_transfers" violates foreign key constraint "fk_image_transfers_command_enitites" DETAIL: Key (command_id)=(fedc82f7-9bf7-4988-b9d9-23853e04e44e) is not present in table "command_entities".
So there is some image transfer left without appropriate command entity. If you are sure that no such transfer is active. You can remove it from the table.
$ scl enable rh-postgresql10 bash $ psql engine
engine=# delete from image_transfers where command_id='fedc82f7-9bf7-4988-b9d9-23853e04e44e';
Or, more generally, copying from 04_03_0270_add_foreign_key_to_image_transfers.sql:
DELETE FROM image_transfers WHERE command_id NOT IN (SELECT command_entities.command_id FROM command_entities);
and then try again.
Adding Also Bhushan, who reported this in private as well.
Adding Daniel, author of the patch causing this:
https://gerrit.ovirt.org/#/q/I65906d09ba201c93da65e9a2b67cef5f9f177747,n,z
Daniel:
This already happened in the past, and we fixed it by adding 04_03_0265_remove_foreign_key_in_image_transfers.sql . Not sure why we needed to add 04_03_0830_add_foreign_key_to_image_transfers.sql again, but perhaps add now another copy of 04_03_0265_remove_foreign_key_in_image_transfers.sql before it as well.
Alternatively, if we are still not fully certain this is over, consider adding an engine-setup plugin that does that, ignoring common practice of dbscripts upgrade (which ignores files it already loaded in the past).
Pushed [1] to master and 4.3 branches. Feel free to take over, change Bug-Url, whatever. [1] https://gerrit.ovirt.org/#/q/I90b464eb6574b9b49daf26d365f4034098f01f64
Also: Perhaps push a patch to OST to verify this flow.
Thanks and best regards,
[ ERROR ] Failed to execute stage 'Misc configuration': Engine schema refresh failed ... [ INFO ] Yum Verify: 16/16: ovirt-engine-tools.noarch 0:4.3.3.5-1.el7 - e [WARNING] Rollback of DWH database postponed to Stage "Clean up" [ INFO ] Rolling back database schema ... [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed
Attaching engine-setup logfile.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/45EDYLIQE4QUJY...
Best regards,
-- Lucie Leistnerova Quality Engineer, QE Cloud, RHVM Red Hat EMEA
IRC: lleistne @ #rhev-qe
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2UT7ANKVQKB5NS...
-- Didi
-- Didi
-- Didi

On Wed, Apr 17, 2019 at 11:43 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Apr 17, 2019 at 10:51 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Apr 17, 2019 at 10:48 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Apr 17, 2019 at 10:30 AM Lucie Leistnerova <lleistne@redhat.com> wrote:
Hi Edward,
On 4/16/19 9:23 PM, Edward Berger wrote:
I was trying to upgrade a hyperconverged oVirt hosted engine and failed in the engine-setup command with these error and warnings.
... [ INFO ] Creating/refreshing Engine database schema [ ERROR ] schema.sh: FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/dbscripts/upgrade/04_03_0830_add_foreign_key_to_image_transfers.sql
The error is
ERROR: insert or update on table "image_transfers" violates foreign key constraint "fk_image_transfers_command_enitites" DETAIL: Key (command_id)=(fedc82f7-9bf7-4988-b9d9-23853e04e44e) is not present in table "command_entities".
So there is some image transfer left without appropriate command entity. If you are sure that no such transfer is active. You can remove it from the table.
$ scl enable rh-postgresql10 bash $ psql engine
engine=# delete from image_transfers where command_id='fedc82f7-9bf7-4988-b9d9-23853e04e44e';
Or, more generally, copying from 04_03_0270_add_foreign_key_to_image_transfers.sql:
DELETE FROM image_transfers WHERE command_id NOT IN (SELECT command_entities.command_id FROM command_entities);
and then try again.
Adding Also Bhushan, who reported this in private as well.
Adding Daniel, author of the patch causing this:
https://gerrit.ovirt.org/#/q/I65906d09ba201c93da65e9a2b67cef5f9f177747,n,z
Daniel:
This already happened in the past, and we fixed it by adding 04_03_0265_remove_foreign_key_in_image_transfers.sql . Not sure why we needed to add 04_03_0830_add_foreign_key_to_image_transfers.sql again, but perhaps add now another copy of 04_03_0265_remove_foreign_key_in_image_transfers.sql before it as well.
We needed the '04_03_0830' script, to avoid a missing FK when upgrading from 4.2.7 to 4.3, as '04_03_0270_add_foreign_key' script was skipped (and thus the FK was missing due to '04_03_0265_remove_foreign_key_in_image_transfers'). As Didi mentioned, deleting the stale image_transfers should solve the issue [*] I'll add this script to the patch: https://gerrit.ovirt.org/#/c/99497/ [*] DELETE FROM image_transfers WHERE command_id NOT IN (SELECT command_entities.command_id FROM command_entities);
Alternatively, if we are still not fully certain this is over, consider adding an engine-setup plugin that does that, ignoring common practice of dbscripts upgrade (which ignores files it already loaded in the past).
Pushed [1] to master and 4.3 branches. Feel free to take over, change Bug-Url, whatever.
[1] https://gerrit.ovirt.org/#/q/I90b464eb6574b9b49daf26d365f4034098f01f64
Also: Perhaps push a patch to OST to verify this flow.
Thanks and best regards,
[ ERROR ] Failed to execute stage 'Misc configuration': Engine schema refresh failed ... [ INFO ] Yum Verify: 16/16: ovirt-engine-tools.noarch 0:4.3.3.5-1.el7 - e [WARNING] Rollback of DWH database postponed to Stage "Clean up" [ INFO ] Rolling back database schema ... [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed
Attaching engine-setup logfile.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/45EDYLIQE4QUJY...
Best regards,
-- Lucie Leistnerova Quality Engineer, QE Cloud, RHVM Red Hat EMEA
IRC: lleistne @ #rhev-qe
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2UT7ANKVQKB5NS...
-- Didi
-- Didi
-- Didi

Thanks! That fixed the problem. engine-setup was able to complete. On Wed, Apr 17, 2019 at 3:48 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Apr 17, 2019 at 10:30 AM Lucie Leistnerova <lleistne@redhat.com> wrote:
Hi Edward,
On 4/16/19 9:23 PM, Edward Berger wrote:
I was trying to upgrade a hyperconverged oVirt hosted engine and failed
in the engine-setup command with these error and warnings.
... [ INFO ] Creating/refreshing Engine database schema [ ERROR ] schema.sh: FATAL: Cannot execute sql command:
--file=/usr/share/ovirt-engine/dbscripts/upgrade/04_03_0830_add_foreign_key_to_image_transfers.sql
The error is
ERROR: insert or update on table "image_transfers" violates foreign key
DETAIL: Key (command_id)=(fedc82f7-9bf7-4988-b9d9-23853e04e44e) is not
constraint "fk_image_transfers_command_enitites" present in table "command_entities".
So there is some image transfer left without appropriate command entity. If you are sure that no such transfer is active. You can remove it from
the table.
$ scl enable rh-postgresql10 bash $ psql engine
engine=# delete from image_transfers where
command_id='fedc82f7-9bf7-4988-b9d9-23853e04e44e';
Or, more generally, copying from 04_03_0270_add_foreign_key_to_image_transfers.sql:
DELETE FROM image_transfers WHERE command_id NOT IN (SELECT command_entities.command_id FROM command_entities);
and then try again.
Adding Also Bhushan, who reported this in private as well.
Adding Daniel, author of the patch causing this:
https://gerrit.ovirt.org/#/q/I65906d09ba201c93da65e9a2b67cef5f9f177747,n,z
Daniel:
This already happened in the past, and we fixed it by adding 04_03_0265_remove_foreign_key_in_image_transfers.sql . Not sure why we needed to add 04_03_0830_add_foreign_key_to_image_transfers.sql again, but perhaps add now another copy of 04_03_0265_remove_foreign_key_in_image_transfers.sql before it as well.
Also: Perhaps push a patch to OST to verify this flow.
Thanks and best regards,
[ ERROR ] Failed to execute stage 'Misc configuration': Engine schema
refresh failed
... [ INFO ] Yum Verify: 16/16: ovirt-engine-tools.noarch 0:4.3.3.5-1.el7 - e [WARNING] Rollback of DWH database postponed to Stage "Clean up" [ INFO ] Rolling back database schema ... [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed
Attaching engine-setup logfile.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/45EDYLIQE4QUJY...
Best regards,
-- Lucie Leistnerova Quality Engineer, QE Cloud, RHVM Red Hat EMEA
IRC: lleistne @ #rhev-qe
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2UT7ANKVQKB5NS...
-- Didi
participants (4)
-
Daniel Erez
-
Edward Berger
-
Lucie Leistnerova
-
Yedidyah Bar David