[ovirt-users] Messed up upgrade to 4.2. Solution badly needed.

[Marcin Jessa]

Hi guys.

I have VM with hosted engine and a two node setup. When beta came out I was “OH! New shiny tech! Let’s try it!”. You all know the feeling.
Unfortunately the upgrade process did not go as expected. 
I created a backup of my 4.1 installation and started to upgrade the VM. All went well. I then migrated all the running VMs to my second node put the cluster in global maintenance mode and updated one of the nodes.
Then I run hosted-engine —upgrade-appliance on the node but it fails:

[ ERROR ] Failed to execute stage 'Environment customization': exceptions must be old-style classes or derived from BaseException, not str

The log file says:
2017-11-03 13:12:48,259+0100 DEBUG otopi.context context._executeMethod:143 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-upgradeappliance/engine/misc.py", line 180, in _check_spm
    'Unable to find this host in the engine, '
TypeError: exceptions must be old-style classes or derived from BaseException, not str
2017-11-03 13:12:48,259+0100 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Environment customization': exceptions must be old-style classes or derived from BaseException, not str
2017-11-03 13:12:48,260+0100 DEBUG otopi.context context.dumpEnvironment:821 ENVIRONMENT DUMP - BEGIN
2017-11-03 13:12:48,260+0100 DEBUG otopi.context context.dumpEnvironment:831 ENV BASE/error=bool:'True'
2017-11-03 13:12:48,260+0100 DEBUG otopi.context context.dumpEnvironment:831 ENV BASE/exceptionInfo=list:'[(<type 'exceptions.TypeError'>, TypeError('exceptions must be old-style classes or derived from BaseException, not str',), <traceback object at 0x41d8128>)]'
2017-11-03 13:12:48,261+0100 DEBUG otopi.context context.dumpEnvironment:835 ENVIRONMENT DUMP - END

I can login to the hosted engine but the nodes are down, the VMs are down and the storage is not connected.

Can you please advice what to do now? 
Is there a way to upgrade my setup?
Should I disconnect my second node, which is also upgraded to 4.2 now and try to install hosted engine from scratch?
Anything else I can do to save my VMs?


Cheers
Marcin.