Can't upgrade ovirt 4.4.3 to 4.4.9
Hello, I tried to upgrade ovirt to 4.4.9 from 4.4.3 when i do 'engine-setup' i get error -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ ERROR ] Failed to execute stage 'Misc configuration': Command '/usr/share/ovirt-engine-dwh/dbscripts/schema.sh' failed to execute [ INFO ] DNF Performing DNF transaction rollback [ ERROR ] DNF module 'dnf.history' has no attribute 'open_history' -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- from the log i found that column "count_threads_as_cores" does not exist --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 193375 2022-01-18 16:36:26,309+0900 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine_dwh.db.schema plugin.execute :926 execute-output: ['/usr/share/ovirt-engine-dwh/dbscripts/schema.sh', '-s', 'localhost', '-p', '5432', '-u ', 'ovirt_engine_history', '-d', 'ovirt_engine_history', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setu p-20220118162426-z0o4xg.log', '-c', 'apply'] stderr: 193376 psql:/usr/share/ovirt-engine-dwh/dbscripts/create_views_4_4.sql:148: ERROR: column "count_threads_as_cores" does not exist 193377 LINE 10: count_threads_as_cores as count_threads_as_cores, 193378 ^ 193379 FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine-dwh/dbscripts/create_views_4_4.sql 193380 193381 2022-01-18 16:36:26,309+0900 DEBUG otopi.context context._executeMethod:145 method exception 193382 Traceback (most recent call last): 193383 File "/usr/lib/python3.6/site-packages/otopi/context.py", line 132, in _executeMethod 193384 method['method']() 193385 File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine-dwh/db/schema.py", line 367, in _misc 193386 odwhcons.DBEnv.PGPASS_FILE 193387 File "/usr/lib/python3.6/site-packages/otopi/plugin.py", line 931, in execute 193388 command=args[0], 193389 RuntimeError: Command '/usr/share/ovirt-engine-dwh/dbscripts/schema.sh' failed to execute 193390 2022-01-18 16:36:26,330+0900 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Misc con figuration': Command '/usr/share/ovirt-engine-dwh/dbscripts/schema.sh' failed to execute 193391 2022-01-18 16:36:26,330+0900 DEBUG otopi.transaction transaction.abort:119 aborting 'DNF Transaction' 193392 2022-01-18 16:36:26,331+0900 DEBUG otopi.plugins.otopi.packagers.dnfpackager dnfpackager.verbose:75 DNF Closi ng transaction with rollback 193393 2022-01-18 16:36:26,731+0900 INFO otopi.plugins.otopi.packagers.dnfpackager dnfpackager.info:79 DNF Performin g DNF transaction rollback 193394 2022-01-18 16:36:27,570+0900 ERROR otopi.plugins.otopi.packagers.dnfpackager dnfpackager.error:84 DNF module 'dnf.history' has no attribute 'open_history' 193395 2022-01-18 16:36:27,571+0900 DEBUG otopi.transaction transaction.abort:125 Unexpected exception from abort() of 'DNF Transaction' -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- when i search for this problem, i found that column "count_threads_as_cores" is added from ovirt-engine 4.4.6.7. and ovirt-dwh 4.4.7. do i need to update 4.4.3 to 4.4.6( or 4.4.7) then 4.4.6(or 4.4.7) to 4.4.9? Thanks, Jihwan
Unfortunately I have no answer to your problem. But I'd like to know: where does that leave you? Are youre severs still running with normal operational tasks performing, are you just not able to handle migrations, restarts or is your environment down until this gets fixed? Or were you able to go back to 4.3 and await a fix?
Hello, when i check --vm-status, engine is up but health is bad status
and after few minutes, the status of engine is like {"vm": "down_unexpected", "health": "bad", "detail": "Down", "reason": "bad vm status"}
On Fri, Jan 21, 2022 at 12:02 PM <jihwahn1018@naver.com> wrote:
Hello,
I tried to upgrade ovirt to 4.4.9 from 4.4.3
when i do 'engine-setup' i get error -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ ERROR ] Failed to execute stage 'Misc configuration': Command '/usr/share/ovirt-engine-dwh/dbscripts/schema.sh' failed to execute [ INFO ] DNF Performing DNF transaction rollback
This rollback attempt, done due to the failure above, failed:
[ ERROR ] DNF module 'dnf.history' has no attribute 'open_history'
This was fixed in otopi-1.9.6, but ovirt-engine-setup only requires otopi >= 1.9.0. We should either patch it, or change the upgrade documentation to include otopi. Perhaps you'd like to create a bug for this?
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- from the log i found that column "count_threads_as_cores" does not exist
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 193375 2022-01-18 16:36:26,309+0900 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine_dwh.db.schema plugin.execute :926 execute-output: ['/usr/share/ovirt-engine-dwh/dbscripts/schema.sh', '-s', 'localhost', '-p', '5432', '-u ', 'ovirt_engine_history', '-d', 'ovirt_engine_history', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setu p-20220118162426-z0o4xg.log', '-c', 'apply'] stderr: 193376 psql:/usr/share/ovirt-engine-dwh/dbscripts/create_views_4_4.sql:148: ERROR: column "count_threads_as_cores" does not exist 193377 LINE 10: count_threads_as_cores as count_threads_as_cores,
This indeed looks like the root cause for the failure. Can you please share the full setup log? Thanks.
193378 ^ 193379 FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine-dwh/dbscripts/create_views_4_4.sql 193380 193381 2022-01-18 16:36:26,309+0900 DEBUG otopi.context context._executeMethod:145 method exception 193382 Traceback (most recent call last): 193383 File "/usr/lib/python3.6/site-packages/otopi/context.py", line 132, in _executeMethod 193384 method['method']() 193385 File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine-dwh/db/schema.py", line 367, in _misc 193386 odwhcons.DBEnv.PGPASS_FILE 193387 File "/usr/lib/python3.6/site-packages/otopi/plugin.py", line 931, in execute 193388 command=args[0], 193389 RuntimeError: Command '/usr/share/ovirt-engine-dwh/dbscripts/schema.sh' failed to execute 193390 2022-01-18 16:36:26,330+0900 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Misc con figuration': Command '/usr/share/ovirt-engine-dwh/dbscripts/schema.sh' failed to execute 193391 2022-01-18 16:36:26,330+0900 DEBUG otopi.transaction transaction.abort:119 aborting 'DNF Transaction' 193392 2022-01-18 16:36:26,331+0900 DEBUG otopi.plugins.otopi.packagers.dnfpackager dnfpackager.verbose:75 DNF Closi ng transaction with rollback 193393 2022-01-18 16:36:26,731+0900 INFO otopi.plugins.otopi.packagers.dnfpackager dnfpackager.info:79 DNF Performin g DNF transaction rollback 193394 2022-01-18 16:36:27,570+0900 ERROR otopi.plugins.otopi.packagers.dnfpackager dnfpackager.error:84 DNF module 'dnf.history' has no attribute 'open_history' 193395 2022-01-18 16:36:27,571+0900 DEBUG otopi.transaction transaction.abort:125 Unexpected exception from abort() of 'DNF Transaction' --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
when i search for this problem, i found that column "count_threads_as_cores" is added from ovirt-engine 4.4.6.7. and ovirt-dwh 4.4.7.
Correct. Was added in an upgrade script. There might be a bug/issue/change in the order that the upgrade script runs vs create_views_* above. Adding Aviv and Shirly.
do i need to update 4.4.3 to 4.4.6( or 4.4.7) then 4.4.6(or 4.4.7) to 4.4.9?
If you have a test system with 4.4.3, you can try there. On Fri, Jan 21, 2022 at 10:54 PM Thomas Hoberg <thomas@hoberg.net> wrote:
Unfortunately I have no answer to your problem.
But I'd like to know: where does that leave you?
That's a good question, Thomas - thanks for raising it. I can't be sure without checking the setup log. Since the rollback failed, it depends on whether the rpm packages were already upgraded at that point or not. If you took a backup (engine-backup) before the upgrade, I think it would be best to do something like: 1. Check with 'dnf history' what was changed. Hopefully you'll find there exactly a single dnf transaction done by engine-setup, and exactly one done by you when upgrading the setup packages. Suppose they have IDs 11 and 10, respectively. 2. engine-cleanup 3. "dnf history undo 11" and "dnf history undo 10". Hopefully none of these will fail, because the oVirt repo keeps all released versions of the engine and dwh. 4. Restore the backup with engine-backup. If you didn't take a backup, I'd first like to see the setup log, to try and suggest how to revert.
Are youre severs still running with normal operational tasks performing, are you just not able to handle migrations, restarts or is your environment down until this gets fixed?
I'd like to know as well. Generally speaking, if the engine is down, all VMs should still be up, but there is no (simple/robust) way to manage them - including stopping/starting/etc. .
Or were you able to go back to 4.3 and await a fix?
I didn't see "4.3" mentioned anywhere, only "4.4.3". Sorry for the mess, good luck, best regards, -- Didi
Hello, Thanks for your answer
This indeed looks like the root cause for the failure. Can you please share the full setup log? Thanks.
i send the log file by mail
I'd like to know as well. Generally speaking, if the engine is down, all VMs should still be up, but there is no (simple/robust) way to manage them - including stopping/starting/etc. .
when i check --vm-status, engine is up but health is bad status and when i turn off the global maintenance mode the status of engine is like {"vm": "down_unexpected", "health": "bad", "detail": "Down", "reason": "bad vm status"} Thank you. Jihwan.
Correct. Was added in an upgrade script. There might be a bug/issue/change in the order that the upgrade script runs vs create_views_* above.
Adding Aviv and Shirly.
i found this problem at log. UPDATE 0 2022-01-21 16:02:41,589+0900 Running upgrade sql script '/usr/share/ovirt-engine-dwh/dbscripts/upgrade/pre_upgrade/set_etl_minimal_version.sql'... 2022-01-21 16:02:41,591+0900 dbfunc_psql_die --file=/usr/share/ovirt-engine-dwh/dbscripts/upgrade/pre_upgrade/set_etl_minimal_version.sql ********* QUERY ********** INSERT INTO history_configuration(var_name,var_value) SELECT 'MinimalETLVersion','4.4.7' WHERE not exists (SELECT var_name FROM history_configuration WHERE var_name = 'MinimalETLVersion'); ************************** INSERT 0 0 ********* QUERY ********** UPDATE history_configuration SET var_value = '4.4.7' WHERE var_name = 'MinimalETLVersion'; ************************** ...... 2022-01-21 16:02:43,138+0900 Skipping upgrade script /usr/share/ovirt-engine-dwh/dbscripts/upgrade/04_04_0040_added_host_count_threads_as_cores.sql, its version 04040040 is <= current version 04040070 ********* QUERY ********** copy (select now()) to stdout with delimiter as '|'; ************************** Sorry for the mess, Thank you -- Jihwan
when i check 'schema_version' table in 'ovirt_engine_history' db --------------------------------------------------- '47 | 04040070 | upgrade/04_04_0070_updated_calendar_table_year_script.sql | 9cfaeaca2a50dd6d4060b488d37d277 0 | ovirt_engine_history | 2022-01-24 14:34:43.769728 | 2022-01-24 14:34:48.520829 | INSTALLED | t |' -------------------------------------- seems make problem at _dbfunc_common_get_current_version() { dbfunc_psql_statement_parsable " select version from schema_version where current = true order by id LIMIT 1 " } it takes 04040070 so it skip all upgrade script that less then version 04040070. i don't know why that value in table but when i delete that value 'engine-setup' works well Thank you.
Hi Jihwan, Just making sure, everything is working as expected after the engine-setup worked? The ovirt-engine.service, and ovirt-engine-dwhd.service are active ? Thanks, Aviv On Mon, Jan 24, 2022 at 11:34 AM <jihwahn1018@naver.com> wrote:
when i check 'schema_version' table in 'ovirt_engine_history' db
--------------------------------------------------- '47 | 04040070 | upgrade/04_04_0070_updated_calendar_table_year_script.sql | 9cfaeaca2a50dd6d4060b488d37d277 0 | ovirt_engine_history | 2022-01-24 14:34:43.769728 | 2022-01-24 14:34:48.520829 | INSTALLED | t |'
-------------------------------------- seems make problem
at
_dbfunc_common_get_current_version() { dbfunc_psql_statement_parsable " select version from schema_version where current = true order by id LIMIT 1 " }
it takes 04040070 so it skip all upgrade script that less then version 04040070.
i don't know why that value in table but when i delete that value 'engine-setup' works well
Thank you. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/JDYNLHVXDRTE4O...
Hello, i do this for upgrade test so after i check that engine-setup work well, i didn't check engine work well and cleanup. Sorry for no help. Jihwan
participants (4)
-
Aviv Litman -
jihwahn1018＠naver.com -
Thomas Hoberg -
Yedidyah Bar David