Agreed, I did revert my victim VM back the pre-upgrade snapshot and retry the upgrade with the same results with yesterdays packages.I had watch on the logs but I did not see any DB upgrade failures. Given that I figured for the heck of it I would just rebuild with this mornings latest commits to see if the issue still persisted, I actually fully expected it to fail the same. However to my surprise it did not fail, thus I assume something changed between yesterday morning this morning that allowed things to work. for grins the packages I used yesterday were generated from: engine commit: ec2b57124edd5ae1fa41e986adadfefb9c5f1aa7 otopi commit: 8e3e3c27cbcd4330143d571a3ce8caa36c6d20ef ovirt-host-deply commit: 35a5b5f5e6d6c240ef1d20904bc06498b0ed81e1 and the packages I generated this morning that worked were from: engine commit: 28251d9acd1ecbe39b9b79e7e2e69342f57c9bd7 otopi commit: 10892fbf3fe805cea416dfa3ba628f98ffa88b3e ovirt-host-deply commit: e9c98c67dcdfca21091141bcb732727f50eb97d1 - DHC On Tue, Jul 30, 2013 at 3:27 PM, Doron Fediuck <dfediuck@redhat.com> wrote:
Big thanks DHC for your time and efforts.
Since neither of us managed to reproduce the issue I suggest we drop it for now. If someone else will hit into it, we should try and get more info on a possible failure which will not complete the db upgrade.
Thanks again, Doron
----- Original Message ----- | From: "Dead Horse" <deadhorseconsulting@gmail.com> | To: "Doron Fediuck" <dfediuck@redhat.com> | Cc: "Alon Bar-Lev" <alonbl@redhat.com>, "users" <users@ovirt.org>, "Dave Chen" <wei.d.chen@intel.com>, "Eli Mesika" | <emesika@redhat.com> | Sent: Tuesday, July 30, 2013 11:20:01 PM | Subject: Re: [Users] upgrade to latest master not working | | Ok, so something from yesterday morning to today changed. I again rebuilt | engine from the latest commit this AM. Accordingly I also built this | mornings latest commits of otopi and ovirt-host-deploy. I then attempted | upgrade and this time I did not note any major issues. The upgrade appears | to have succeeded, and engine starts and runs. | The only things that caused failures during upgrade were: | - If the previous version ovirt-engine-sdk is not removed manually and | replaced with ovirt-engine-sdk-python engine-upgrade fails due to package | dependency issues. (which then upsets ovirt-iso-uploader, | ovirt-image-uploader, ovirt-log-collector dependencies on ovirt-engine-sdk) | - If there is a leftover releasepreview directory engine-upgrade will not | run | | There were some error messages in the engine log griping about "Could not | parse option AutoRecoveryAllowedTypes value". I have attached the engine | log file from the first start of the engine after upgrade. | | The other minor thing to note is that the default ovirt.brand theme is not | working. The CSS styles are not being applied. Changes to | welcome_page.template and messages.properties were picked up however. | | - DHC | | | | On Tue, Jul 30, 2013 at 11:04 AM, Doron Fediuck <dfediuck@redhat.com> wrote: | | > | > | > ----- Original Message ----- | > | From: "Dead Horse" <deadhorseconsulting@gmail.com> | > | To: "Doron Fediuck" <dfediuck@redhat.com> | > | Cc: "Alon Bar-Lev" <alonbl@redhat.com>, "users" <users@ovirt.org>, | > "Dave Chen" <wei.d.chen@intel.com>, "Eli Mesika" | > | <emesika@redhat.com> | > | Sent: Tuesday, July 30, 2013 6:55:18 PM | > | Subject: Re: [Users] upgrade to latest master not working | > | | > | The only error I can find in the upgrade log is: | > | 2013-07-29 14:06:28 DEBUG otopi.plugins.otopi.packagers.yumpackager | > | yumpackager.verbose:84 Yum Done: ovirt-engine.noarch 0:3.3.0-20.fc18 - u | > | 2013-07-29 14:06:28 INFO otopi.plugins.otopi.packagers.yumpackager | > | yumpackager.info:88 Yum update: 10/18: ovirt-engine | > | 2013-07-29 14:06:35 DEBUG otopi.plugins.otopi.packagers.yumpackager | > | yumpackager.verbose:84 Yum Done: ovirt-engine.noarch 0:3.3.0-20.fc18 - u | > | 2013-07-29 14:06:35 DEBUG otopi.plugins.otopi.packagers.yumpackager | > | yumpackager.verbose:84 Yum Done: ovirt-engine-3.3.0-20.fc18.noarch | > | 2013-07-29 14:06:37 INFO otopi.plugins.otopi.packagers.yumpackager | > | yumpackager.info:88 Yum update: 11/18: ovirt-engine-dbscripts | > | 2013-07-29 14:06:38 DEBUG otopi.plugins.otopi.packagers.yumpackager | > | yumpackager.verbose:84 Yum Done: | > ovirt-engine-dbscripts-3.3.0-20.fc18.noarch | > | 2013-07-29 14:06:38 DEBUG otopi.plugins.otopi.packagers.yumpackager | > | yumpackager.verbose:84 Yum Done: ovirt-engine-backend | > | Traceback (most recent call last): | > | File "/usr/lib/python2.7/site-packages/yum/rpmtrans.py", line 464, in | > | callback | > | self._unInstStop( bytes, total, h ) | > | File "/usr/lib/python2.7/site-packages/yum/rpmtrans.py", line 564, in | > | _unInstStop | > | self.total_actions) | > | File "/usr/lib/python2.7/site-packages/otopi/miniyum.py", line 204, in | > | event | > | package=package.name, | > | AttributeError: 'str' object has no attribute 'name' | > | FATAL ERROR: python callback <bound method RPMTransaction.callback of | > | <yum.rpmtrans.RPMTransaction instance at 0x4587560>> failed, aborting! | > | | > | To test the database upgrade I will revert to a previous snapshot of the | > | debug VM, capture the dbase from the previous engine version, re-run | > | upgrade, dump the upgraded database, restore the previous, and then test | > a | > | manual run of the database upgrade script. | > | | > | Does this sound prudent? | > | - DHC | > | | > | > Yes. Although lots of work, it may show if there's a DB upgrade bug we | > missed, or indeed the general upgrade procedure has / had an issue. | > | > | | > | On Tue, Jul 30, 2013 at 5:54 AM, Doron Fediuck <dfediuck@redhat.com> | > wrote: | > | | > | > Thanks. | > | > So this leads me further to the upgrade procedure | > | > which may have fail, thus not completing the DB upgrade. | > | > Can you please check the upgrade logs to see if something | > | > matches this flow? | > | > | > | > ----- Original Message ----- | > | > | From: "Dead Horse" <deadhorseconsulting@gmail.com> | > | > | To: "Doron Fediuck" <dfediuck@redhat.com> | > | > | Cc: "Alon Bar-Lev" <alonbl@redhat.com>, "users" <users@ovirt.org
, | > | > "Dave Chen" <wei.d.chen@intel.com>, "Eli Mesika" | > | > | <emesika@redhat.com> | > | > | Sent: Tuesday, July 30, 2013 6:37:55 AM | > | > | Subject: Re: [Users] upgrade to latest master not working | > | > | | > | > | The engine version I am attempting to upgrade from was built from: | > | > | --------------- | > | > | commit: dbc3d31110ce372a1fa7e06f64c4a5c64544c679 | > | > | tree: ee9ee41c3e855c26f491c77fda9622c05af3fc54 | > | > | parent: 82644cae97f3c546ee631ae79c925c91e7bed836 | > | > | | > | > | userportal,webadmin: Change remove message | > | > | Change-Id: Ia934e33d1a975e0235e1a1ffae0c8a4a7af66f10 | > | > | Signed-off-by: Alexander Wels <awels@redhat.com> | > | > | --------------- | > | > | Thus it is version 3.3 and not 3.2. The upgrade (built from this | > master | > | > | this AM) was attempted ovirt a fresh install of the packages built | > from | > | > the | > | > | above commit. I also confirmed the install was fully functional. | > | > | | > | > | - DHC | > | > | | > | > | | > | > | | > | > | On Mon, Jul 29, 2013 at 4:08 PM, Doron Fediuck < dfediuck@redhat.com> | > | > wrote: | > | > | | > | > | > | > | > | > | > | > | > ----- Original Message ----- | > | > | > | From: "Alon Bar-Lev" <alonbl@redhat.com> | > | > | > | To: "Doron Fediuck" <dfediuck@redhat.com> | > | > | > | Cc: "Dead Horse" <deadhorseconsulting@gmail.com>, "users" < | > | > | > users@ovirt.org>, "Dave Chen" <wei.d.chen@intel.com>, "Eli | > | > | > | Mesika" <emesika@redhat.com> | > | > | > | Sent: Monday, July 29, 2013 11:51:35 PM | > | > | > | Subject: Re: [Users] upgrade to latest master not working | > | > | > | | > | > | > | | > | > | > | | > | > | > | ----- Original Message ----- | > | > | > | > From: "Doron Fediuck" <dfediuck@redhat.com> | > | > | > | > To: "Alon Bar-Lev" <alonbl@redhat.com> | > | > | > | > Cc: "Dead Horse" <deadhorseconsulting@gmail.com>, "users" | > | > | > | > <users@ovirt.org>, "Dave Chen" <wei.d.chen@intel.com>, "Eli | > | > | > | > Mesika" <emesika@redhat.com> | > | > | > | > Sent: Monday, July 29, 2013 11:43:46 PM | > | > | > | > Subject: Re: [Users] upgrade to latest master not working | > | > | > | > | > | > | > | > | > | > | > | > | > | > | > | > ----- Original Message ----- | > | > | > | > | From: "Alon Bar-Lev" <alonbl@redhat.com> | > | > | > | > | To: "Doron Fediuck" <dfediuck@redhat.com> | > | > | > | > | Cc: "Dead Horse" <deadhorseconsulting@gmail.com>, "users" | > | > | > | > | <users@ovirt.org>, "Dave Chen" <wei.d.chen@intel.com>, "Eli | > | > | > | > | Mesika" <emesika@redhat.com> | > | > | > | > | Sent: Monday, July 29, 2013 11:31:42 PM | > | > | > | > | Subject: Re: [Users] upgrade to latest master not working | > | > | > | > | | > | > | > | > | | > | > | > | > | | > | > | > | > | ----- Original Message ----- | > | > | > | > | > From: "Doron Fediuck" <dfediuck@redhat.com> | > | > | > | > | > To: "Alon Bar-Lev" <alonbl@redhat.com> | > | > | > | > | > Cc: "Dead Horse" <deadhorseconsulting@gmail.com>, "users" | > | > | > | > | > <users@ovirt.org>, "Dave Chen" <wei.d.chen@intel.com>, | > "Eli | > | > | > | > | > Mesika" <emesika@redhat.com> | > | > | > | > | > Sent: Monday, July 29, 2013 11:27:41 PM | > | > | > | > | > Subject: Re: [Users] upgrade to latest master not working | > | > | > | > | > | > | > | > | > | > | > | > | > | > | > | > | > | > | > | > ----- Original Message ----- | > | > | > | > | > | From: "Alon Bar-Lev" <alonbl@redhat.com> | > | > | > | > | > | To: "Dead Horse" <deadhorseconsulting@gmail.com> | > | > | > | > | > | Cc: "users" <users@ovirt.org>, "Doron Fediuck" < | > | > | > dfediuck@redhat.com>, | > | > | > | > | > | "Dave | > | > | > | > | > | Chen" <wei.d.chen@intel.com>, "Eli | > | > | > | > | > | Mesika" <emesika@redhat.com> | > | > | > | > | > | Sent: Monday, July 29, 2013 10:47:01 PM | > | > | > | > | > | Subject: Re: [Users] upgrade to latest master not working | > | > | > | > | > | | > | > | > | > | > | | > | > | > | > | > | | > | > | > | > | > | ----- Original Message ----- | > | > | > | > | > | > From: "Dead Horse" <deadhorseconsulting@gmail.com> | > | > | > | > | > | > To: "Alon Bar-Lev" <alonbl@redhat.com> | > | > | > | > | > | > Cc: "users" <users@ovirt.org> | > | > | > | > | > | > Sent: Monday, July 29, 2013 10:41:34 PM | > | > | > | > | > | > Subject: Re: [Users] upgrade to latest master not | > working | > | > | > | > | > | > | > | > | > | > | > | > server.log attached | > | > | > | > | > | | > | > | > | > | > | Thanks! | > | > | > | > | > | | > | > | > | > | > | My guess is that I8ce3448a[1] broke master. | > | > | > | > | > | | > | > | > | > | > | [1] http://gerrit.ovirt.org/#/c/14605/ | > | > | > | > | > | | > | > | > | > | > | > | > | > | > | > Works perfectly on my Gentoo with master. | > | > | > | > | > | > | > | > | > | | > | > | > | > | I doubt you are upgrading from 3.2 :) | > | > | > | > | | > | > | > | > | > | > | > | > Just did: | > | > | > | > | > | > | > | > $ git checkout origin/engine_3.2 | > | > | > | > $ cd backend/manager/dbscripts/ | > | > | > | > $ ./create_db.sh -d upgrade32-test -u engine | > | > | > | > ~/dev/ovirt/ovirt-engine/backend/manager/dbscripts | > | > | > | > ~/dev/ovirt/ovirt-engine/backend/manager/dbscripts | > | > | > | > Creating the database: upgrade32-test | > | > | > | > dropdb: database removal failed: ERROR: database | > "upgrade32-test" | > | > | > does not | > | > | > | > exist | > | > | > | > user name is: engine | > | > | > | > Creating tables... | > | > | > | > Creating functions... | > | > | > | > Creating common functions... | > | > | > | > Inserting data ... | > | > | > | > Inserting pre-defined roles ... | > | > | > | > Running upgrade scripts... | > | > | > | > upgrade script detected a change in Config, View or Stored | > | > Procedure... | > | > | > | > Running upgrade sql script | > upgrade/pre_upgrade/0000_config.sql ... | > | > | > | > | > | > | > | > ... | > | > | > | > | > | > | > | > Creating stored procedures from vm_templates_sp.sql ... | > | > | > | > Running upgrade sql script | > | > | > | > | > upgrade/post_upgrade/0010_add_object_column_white_list_table.sql | > | > ... | > | > | > | > Running upgrade sql script | > | > | > | > upgrade/post_upgrade/0020_create_materialized_views.sql ... | > | > | > | > Refreshing materialized views... | > | > | > | > ~/dev/ovirt/ovirt-engine/backend/manager/dbscripts | > | > | > | > | > | > | > | > $ git checkout master | > | > | > | > $ cd ../../../packaging/dbscripts/ | > | > | > | > $ ./upgrade.sh -d upgrade32-test -u engine | > | > | > | > upgrade script detected a change in Config, View or Stored | > | > Procedure... | > | > | > | > Running upgrade sql script upgrade/pre_upgrade/0000_config.sql | > ... | > | > | > | > | > | > | > | > ... | > | > | > | > | > | > | > | > Creating stored procedures from vms_sp.sql ... | > | > | > | > Running upgrade sql script | > | > | > | > | > upgrade/post_upgrade/0010_add_object_column_white_list_table.sql | > | > ... | > | > | > | > Running upgrade sql script | > | > | > | > upgrade/post_upgrade/0020_create_materialized_views.sql ... | > | > | > | > Refreshing materialized views... | > | > | > | > Done. | > | > | > | > | > | > | > | > | > | > | > | > Seems to work well. Am I missing something? | > | > | > | > | > | > | > | | > | > | > | Start jboss? | > | > | > | | > | > | > | Please take over this issue, you have relevant logs, and a | > | > cooperative | > | > | > user. | > | > | > | | > | > | > | Thanks, | > | > | > | Alon | > | > | > | | > | > | > | > | > | > Since jboss is up and running very well after upgrade I do not see | > a | > | > | > problem. | > | > | > Is it possible that the install / upgrade process didn't complete | > due | > | > to | > | > | > other | > | > | > errors ending up with this state? | > | > | > | > | > | > In the meantime you can manually create the missing function to get | > | > your | > | > | > setup running. The only problem is that if my assumption is | > correct on | > | > | > upgrade not finishing, you will hit the next db change as an | > error. So | > | > | > I'd try and look at the install/upgrade log files to see if db | > upgrade | > | > | > failed at some point. | > | > | > | > | > | | > | > | > | | > |