Hello, If there are Disk Uploads that have not completed (i.e: currently Paused or with Error) and were not canceled the upgrade will fail.This is due to a foreign key being added on 4.2.5, relating image_transfers table to command_entities table. When the roll-back to the previous RHV-M version finishes, open the Administration Portal and navigate to Storage -> Disks. Browse all disks looking for disks with status different than OK, such as Paused by System. Right click and cancel/remove these failed or paused uploads, then try the upgrade again. If it fails then share your observations with us, so we need to do some manual changes in database. Regards, Vaibhav Pagar

On Fri, Aug 3, 2018 at 7:01 AM Jan Siml <jsiml(a)plusline.net&gt; wrote: Turns out this is a regression that we are working urgently to fix: https://bugzilla.redhat.com/show_bug.cgi?id=1609839 Thanks to all who have reported! -- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat NA <https://www.redhat.com/> gshereme(a)redhat.com IRC: gshereme <https://red.ht/sig>

Hello, that's how it worked and the update was successful. Thanks! Regards Jan

In particular, the log of the db restore, where I can find it? to understand which state of the db I have now on disk... the old, the new, a partial restore or what... at this time the image_transfers table is clearly the old one without the foreign key: engine=# \d image_transfers Table "public.image_transfers" Column | Type | Modifiers ---------------------------+--------------------------+------------------------ command_id | uuid | not null command_type | integer | not null phase | integer | not null last_updated | timestamp with time zone | not null message | character varying | vds_id | uuid | disk_id | uuid | imaged_ticket_id | uuid | proxy_uri | character varying | signed_ticket | character varying | bytes_sent | bigint | bytes_total | bigint | type | integer | not null default 0 active | boolean | not null default false daemon_uri | character varying | client_inactivity_timeout | integer | Indexes: "pk_image_transfers" PRIMARY KEY, btree (command_id) "idx_image_transfers_disk_id" btree (disk_id) and contains engine=# select * from image_transfers; command_id | command_type | phase | last_updated | m essage | vds_id | disk_id | imaged _ticket_id | proxy_uri | signed_ticket | bytes_sent | bytes_total | type | active | daemon_uri | client_inactivity_timeout --------------------------------------+--------------+-------+----------------------------+------------ -----------------+--------------------------------------+--------------------------------------+------- -----------+--------------------------------+---------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------- ----------------------------------------------+------------+-------------+------+--------+------------+ --------------------------- 7b0e8a09-d8e9-48c4-83af-fdd94268f355 | 1024 | 6 | 2017-12-05 23:09:05.307+00 | Pausing due to client error | 459f6e56-f813-4c73-9950-994a7de8dbb5 | 713df8c6-8a97-425f-a169-fa2b7104bdc4 | | https://localhost:54323/images | eyJzYWx0IjoidHdLQWdwN0lnREU9IiwiZGF0YSI6IntcbiAgXCJuYmZcI iA6IDE1MTI1MTQ5NzIsXG4gIFwiZXhwXCIgOiAxNTEyNTE4NTcyLFxuICBcImlhdFwiIDogMTUxMjUxNDk3MixcbiAgXCJ0cmFuc2Zl ci10aWNrZXRcIiA6IFwiZTVjY2JiOWMtOTNlZS00MDRkLTgwOWItNDMzNzZkYTQ4ZmE3XCIsXG4gIFwiaW1hZ2VkLXVyaVwiIDogXCJ odHRwczovL292aXJ0MDEubHV0d3luLm9yZzo1NDMyMlwiXG59Iiwic2lnbmF0dXJlIjoibiszZ0pUbmgyd1gxYzUzMWNsRWwza1c1Mk 5vSEhXOGhUZGxZZmN1T3JURlVveHlicVhwaTZTRVhsWERCN2tOOXZBT1dMb1NhT0t5TTRWT1I0VTl1djNLdldHNHlTVUp1YWhha3dFY 3J3UUJkOTFwQVkrVXM1ZGRLRkxlSTB2NkQ2eitBOEJXRkduUkpNdCtxemJUeTd5dm5wTnVXSkxIMXlFOXFRTHFDWEN4UEZTSkNHekpY aTVveUZHQlNPTVNUdEtYbjVCS0gwejdGanR5akhZd3orMkRVM3Bxa0JESUp0dTZGRFZjeG9Gelp2bjR0amtla3M1WldvNS95NFYwekZ pcE5PdWJIZXNGMlpvL2liVGIrQ0pLZ0xyOFBHNzE1VmhKbmxlQ1BLbE5zV2VkcVRDSEE4M3ZxOHpjdFMveWZCZU9seEgzZ3BEZktkS0 tDYjlVcHNRPT0iLCJkaWdlc3QiOiJzaGExIiwiY2VydGlmaWNhdGUiOiItLS0tLUJFR0lOIENFUlRJRklDQVRFLS0tLS1cbk1JSUVjR ENDQTFpZ0F3SUJBZ0lDRUF3d0RRWUpLb1pJaHZjTkFRRUxCUUF3UXpFTE1Ba0dBMVVFQmhNQ1ZWTXhFekFSQmdOVkJBb1RcclxuQ214 MWRIZDViaTV2Y21jeEh6QWRCZ05WQkFNVEZtOTJhWEowTG14MWRIZDViaTV2Y21jdU16WTFPRGN3SGhjTk1UY3dPREkxTURneFxyXG5 NekF6V2hjTk1qSXdOek14TURneE16QXpXakE5TVFzd0NRWURWUVFHRXdKVlV6RVRNQkVHQTFVRUNoTUtiSFYwZDNsdUxtOXlaekVaXH Jcbk1CY0dBMVVFQXhNUWIzWnBjblF1YkhWMGQzbHVMbTl5WnpDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnR UJcclxuQUxUM3JjQnhaOXU4Zk9EN3JzdWFXeE9CMDNucHZHVnpBU1lQeWtBRm54QVBsanpNK21xSlczWjlUcnJBb3I4Ujg4b3ZGNi9C K3QrUFxyXG5ic3BqZ1BGdGVNVWZ5ZDI2SEh1Wmw2bS9WSDg2MmlaS3owN1RNZndSZVF6bmNkYSt5dkhrWFNONEFEaXFVUmRhOFd0OEc 4MGlLUlVWXHJcbk00a2gvT3pUd0Zhd05ZZFhDcWVEckY4KzBFOFBWZFFNRG9vYnpxTjNTNWhCRDBJK291S284K0E5bkVlTzJxUW5KZ2 5OZHA1d25nTE1cclxuVFByRGhTUERQRS9WQ01FdnQ0WE5VcFJsVWswdlBWTm9wbmk1L0JUV0Z0RnFKNDRieWt4ejBDVE1lWGdZcDlYY npmNi8yY0oxdzV5a1xyXG5ydkZvTExPUnlXR1g1TTlKaENTbzdrWlRHSDFqeEtMMjFwY1JJVk1DQXdFQUFhT0NBWEl3Z2dGdU1CMEdB MVVkRGdRV0JCUXVjZGY2XHJcbkNlNXJuelpyeTVLYjhMcWFpRGJmN1RDQmhBWUlLd1lCQlFVSEFRRUVlREIyTUhRR0NDc0dBUVVGQnp BQ2htaG9kSFJ3T2k4dmIzWnBcclxuY25RdWJIVjBkM2x1TG05eVp6bzRNQzl2ZG1seWRDMWxibWRwYm1VdmMyVnlkbWxqWlhNdmNHdH BMWEpsYzI5MWNtTmxQM0psYzI5MVxyXG5jbU5sUFdOaExXTmxjblJwWm1sallYUmxKbVp2Y20xaGREMVlOVEE1TFZCRlRTMURRVEJzQ mdOVkhTTUVaVEJqZ0JTQWRjc3NPN3hrXHJcbnkwQkdtNCtibGJOaDlmWmduYUZIcEVVd1F6RUxNQWtHQTFVRUJoTUNWVk14RXpBUkJn TlZCQW9UQ214MWRIZDViaTV2Y21jeEh6QWRcclxuQmdOVkJBTVRGbTkyYVhKMExteDFkSGQ1Ymk1dmNtY3VNelkxT0RlQ0FoQUFNQWt HQTFVZEV3UUNNQUF3RGdZRFZSMFBBUUgvQkFRRFxyXG5BZ1dnTUNBR0ExVWRKUUVCL3dRV01CUUdDQ3NHQVFVRkJ3TUJCZ2dyQmdFRk JRY0RBakFiQmdOVkhSRUVGREFTZ2hCdmRtbHlkQzVzXHJcbmRYUjNlVzR1YjNKbk1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQVloT FJnb2h5dVE3SWRiaXhENTdlZDVJN1VINDFOTnFzK2M0SnVcclxuZXFiNG9XSGRtR1ZvWmR2dDBSTUNZRk4xU1p4YXRhMEcvSVFOUEtJ VW9QeHhXZ1Fod3FaUHhsQjcwMnZFdk1YNmZIdjdoTFBma1kyWlxyXG5yL2ZpLzZ4SE5GQU05dDE5K3h5SjRrTHhPc2ZpWkdKWjF6aHF 5UEFlT3l2VFJieUtiaHh6Y29STlRmL1p0cmZ2M0xWTHhQcDJnU1hFXHJcbnZ5MnN1V2t6YUZFd000RW4rTXNuajBhZ3Ixb0RRamNLRW twSjltc2pSMlJpb0xVVnRud3JpZU1XK2ljbnM0NjlpT3l1eWprY3M4a0lcclxuSkRGSXJ3aWg2MHJxVC9zSWlTSlAvQ2dXMGtYOU5Ra GlvT2kraXo2WHZYSjRmNjVpZm1jeTNlaEhMcGZ6NVdQVWx1UlB0Vy96Z3pxcFxyXG4tLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tXG4i LCJzaWduZWRGaWVsZHMiOiJzYWx0LGRhdGEsZGlnZXN0LHZhbGlkRnJvbSx2YWxpZFRvIiwidmFsaWRGcm9tIjoiMjAxNzEyMDUyMzA yNTIiLCJ2YWxpZFRvIjoiMjAxNzEyMDYwMDAyNTIifQ== | 0 | 21478375424 | 0 | f | | (1 row) engine=# can i simply delete the record and run again "engine-setup" or do "completely" roll back the db? Thanks, Gianluca On Fri, Aug 3, 2018 at 2:24 PM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote:

On Fri, Aug 3, 2018 at 3:52 PM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: the snippet of the failed engine db restore inside log file is: 2018-08-03 14:15:47,017+0200 INFO otopi.ovirt_engine_setup.engine_common.database database.restore:869 Restoring file '/var/lib/ovirt-engine/backups/engine-20180803141347.gdQfiz.dump' to database localhost: engine. 2018-08-03 14:15:47,017+0200 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.db.schema plugin.execu tePipeRaw:481 executePipeRaw: [0] popen kw={'stdin': None, 'stdout': -1, 'preexec_fn': <function _enabl eSignals at 0x35fab18>, 'env': {'PGPASSWORD': '', 'LESSOPEN': '||/usr/bin/lesspipe.sh %s', 'SSH_CLIENT' : '192.168.1.211 56098 22', 'SELINUX_USE_CURRENT_RANGE': '', 'LOGNAME': 'root', 'USER': 'root', 'HOME': '/root', 'PATH': '/opt/rh/rh-postgresql95/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/b in:/root/bin', 'LD_LIBRARY_PATH': '/opt/rh/rh-postgresql95/root/usr/lib64', 'LANG': 'en_US.UTF-8', 'TER M': 'xterm-256color', 'SHELL': '/bin/bash', 'LIBRARY_PATH': '/opt/rh/rh-postgresql95/root/usr/lib64', ' SHLVL': '5', 'HISTSIZE': '1000', 'POSTGRESQLENV': 'COMMAND/pg_dump=str:/opt/rh/rh-postgresql95/root/usr /bin/pg_dump COMMAND/psql=str:/opt/rh/rh-postgresql95/root/usr/bin/psql COMMAND/pg_re store=str:/opt/rh/rh-postgresql95/root/usr/bin/pg_restore COMMAND/postgresql-setup=str:/opt/rh /rh-postgresql95/root/usr/bin/postgresql-setup OVESETUP_PROVISIONING/postgresService=str:rh-po stgresql95-postgresql OVESETUP_PROVISIONING/postgresConf=str:/var/opt/rh/rh-postgresql95/lib/p gsql/data/postgresql.conf OVESETUP_PROVISIONING/postgresPgHba=str:/var/opt/rh/rh-postgresql95/ lib/pgsql/data/pg_hba.conf OVESETUP_PROVISIONING/postgresPgVersion=str:/var/opt/rh/rh-postgres ql95/lib/pgsql/data/PG_VERSION', 'MANPATH': '/opt/rh/rh-postgresql95/root/usr/share/man:', 'X_SCLS': 'r h-postgresql95 ', 'XDG_RUNTIME_DIR': '/run/user/0', 'PYTHONPATH': '/usr/share/ovirt-engine/setup/bin/.. ::', 'PGPASSFILE': '/tmp/tmpM8aXOE', 'SELINUX_ROLE_REQUESTED': '', 'MAIL': '/var/spool/mail/root', 'PKG _CONFIG_PATH': '/opt/rh/rh-postgresql95/root/usr/lib64/pkgconfig', 'XDG_SESSION_ID': '4386', 'sclenv': 'rh-postgresql95', 'LS_COLORS': 'rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=3 8;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5 ;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex . . . 'XDG_CONFIG_DIRS': '/etc/opt/rh/rh-postgresql95/xdg:/etc/xdg', 'SSH_TTY': '/dev/pts/1', 'HOSTNAME': 'ovirt.lutwyn.org', 'JAVACONFDIRS': '/etc/opt/rh/rh-postgresql95/java:/etc/java', 'SELINUX_LEVEL_REQUESTED': '', 'HISTCONTROL': 'ignoredups', 'XDG_DATA_DIRS': '/opt/rh/rh-postgresql95/root/usr/share', 'PWD': '/root', 'CPATH': '/opt/rh/rh-postgresql95/root/usr/include', 'OTOPI_LOGFILE': '/var/log/ovirt-engine/setup/ovirt-engine-setup-20180803140934-1thg3k.log', 'SSH_CONNECTION': '192.168.1.211 56098 192.168.1.212 22', 'OTOPI_EXECDIR': '/root'}, 'close_fds': True, 'args': ['/opt/rh/rh-postgresql95/root/usr/bin/pg_restore', '-w', '-h', 'localhost', '-p', '5432', '-U', 'engine', '-d', 'engine', '--jobs=2', '/var/lib/ovirt-engine/backups/engine-20180803141347.gdQfiz.dump'], 'stderr': -1} 2018-08-03 14:15:47,041+0200 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.db.schema plugin.executePipeRaw:486 executePipeRaw: [0] pid pid=10100 2018-08-03 14:16:03,535+0200 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.db.schema plugin.executePipeRaw:603 executePipe-result: [0] ['/opt/rh/rh-postgresql95/root/usr/bin/pg_restore', '-w', '-h', 'localhost', '-p', '5432', '-U', 'engine', '-d', 'engine', '--jobs=2', '/var/lib/ovirt-engine/backups/engine-20180803141347.gdQfiz.dump'], rc=1 2018-08-03 14:16:03,536+0200 DEBUG otopi.ovirt_engine_setup.engine_common.database database.restore:922 db restore rc 1 stderr ['pg_restore: [archiver (db)] Error while PROCESSING TOC:', 'pg_restore: [archiver (db)] Error from TOC entry 7251; 0 0 COMMENT EXTENSION plpgsql ', 'pg_restore: [archiver (db)] could not execute query: ERROR: must be owner of extension plpgsql', " Command was: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';", '', '', '', 'pg_restore: [archiver (db)] Error from TOC entry 7252; 0 0 COMMENT EXTENSION "uuid-ossp" ', 'pg_restore: [archiver (db)] could not execute query: ERROR: must be owner of extension uuid-ossp', ' Command was: COMMENT ON EXTENSION "uuid-ossp" IS \'generate universally unique identifiers (UUIDs)\';', '', '', '', 'pg_restore: WARNING: column "user_role_title" has type "unknown"', 'DETAIL: Proceeding with relation creation anyway.', 'pg_restore: WARNING: no privileges could be revoked for "public"', 'pg_restore: WARNING: no privileges could be revoked for "public"', 'pg_restore: WARNING: no privileges were granted for "public"', 'pg_restore: WARNING: no privileges were granted for "public"', 'WARNING: errors ignored on restore: 2'] 2018-08-03 14:16:03,536+0200 ERROR otopi.ovirt_engine_setup.engine_common.database database.restore:937 Errors while restoring engine database, please check the log file for details 2018-08-03 14:16:03,536+0200 DEBUG otopi.ovirt_engine_setup.engine_common.database database.restore:942 Errors unfiltered during restore: pg_restore: [archiver (db)] Error from TOC entry 7252; 0 0 COMMENT EXTENSION "uuid-ossp" pg_restore: [archiver (db)] could not execute query: ERROR: must be owner of extension uuid-ossp 2018-08-03 14:16:03,537+0200 DEBUG otopi.context context.dumpEnvironment:859 ENVIRONMENT DUMP - BEGIN 2018-08-03 14:16:03,537+0200 DEBUG otopi.context context.dumpEnvironment:869 ENV BASE/error=bool:'True' 2018-08-03 14:16:03,537+0200 DEBUG otopi.context context.dumpEnvironment:869 ENV BASE/exceptionInfo=list:'[(<type 'exceptions.RuntimeError'>, RuntimeError('Engine schema refresh failed',), <traceback object at 0x412ad40>)]' 2018-08-03 14:16:03,539+0200 DEBUG otopi.context context.dumpEnvironment:873 ENVIRONMENT DUMP - END 2018-08-03 14:16:03,539+0200 INFO otopi.context context.runSequence:741 Stage: Clean up 2018-08-03 14:16:03,539+0200 DEBUG otopi.context context.runSequence:745 STAGE cleanup 2018-08-03 14:16:03,541+0200 DEBUG otopi.context context._executeMethod:128 Stage cleanup METHOD otopi.plugins.otopi.dialog.answer_file.Plugin._generate_answer_file 2018-08-03 14:16:03,541+0200 DEBUG otopi.context context.dumpEnvironment:859 ENVIRONMENT DUMP - BEGIN 2018-08-03 14:16:03,542+0200 DEBUG otopi.context context.dumpEnvironment:869 ENV DIALOG/answerFileContent=str:'# OTOPI answer file, generated by human dialog [environment:default] QUESTION/1/ENGINE_VACUUM_FULL=str:yes QUESTION/1/OVESETUP_CORE_ENGINE_STOP=str:ok QUESTION/1/OVESETUP_DWH_PERFORM_BACKUP=str:yes QUESTION/1/OVESETUP_UPDATE_FIREWALL=str:yes QUESTION/1/OVESETUP_DIALOG_CONFIRM_SETTINGS=str:ok QUESTION/1/DWH_VACUUM_FULL=str:yes QUESTION/1/OVESETUP_RPMDISTRO_PACKAGE_UPGRADE=str:yes ' 2018-08-03 14:16:03,543+0200 DEBUG otopi.context context.dumpEnvironment:873 ENVIRONMENT DUMP - END 2018-08-03 14:16:03,544+0200 DEBUG otopi.context context._executeMethod:128 Stage cleanup METHOD otopi.plugins.ovirt_engine_common.base.core.misc.Plugin._cleanup

On Fri, Aug 3, 2018 at 4:09 PM, Jan Siml <jsiml(a)plusline.net&gt; wrote: So your workflow was: - run engine-setup and got failure was it able in your case to restore the engine db in this first run and start engine again? You wrote that there was an upload failure and that you removed it: did you mean you removed it from the GUI, and confirming that after first failure the webadmin gui was still reachable? In my case after first run failure, my engine is down and trying to reach I get: " Service Unavailable The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later. " - run engine-setup again and got failure It seems that after this second failure your engine and dwh db are down, so also the engine portal... Then you manually execute the delete statement - engine-setup again and it succeeds Correct? Gianluca

Thanks for your explanation. Just one more clarification. See below On Fri, Aug 3, 2018 at 5:06 PM, Jan Siml <jsiml(a)plusline.net&gt; wrote: Can you confirm if also in your case the rollback of the db failed? If so, how could you be sure of the consistency of the database if only a partial restore has been executed and the pg_restore command failed? So after manually starting the engine service you were able to connect and delete from GUI the transfer, correct? So after the GUI deletion of the wrong transfer you executed engine-setup again; did you choose to backup the db or not when proposed? after the error you manually deleted and executed again engine-setup So at the end you run 3 times the engine-setup program? It remains my doubt about the status of the engine database due to pg_restore failed in the middle, so some tables/objects could have the old structure and some tables the new. It should be analyzed in your case from 4.2.4 to 4.2.5 what are the db operation to execute duting upgrade and in my case when trying to pass from 4.2.3 to 4.2.5 Gianluca