[ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2

[Sven Achtelik]

> -----Ursprüngliche Nachricht-----
> Von: Yedidyah Bar David [mailto:didi at redhat.com]
> Gesendet: Mittwoch, 28. März 2018 10:06
> An: Sven Achtelik
> Cc: users at ovirt.org
> Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before
> upgrading to 4.2
> 
> On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik <Sven.Achtelik at eps.aero>
> wrote:
> > Hi All,
> >
> >
> >
> > I’m still facing issues with my HE engine. Here are the steps that I
> > took to end up in this situation:
> >
> >
> >
> > -         Update Engine from 4.1.7 to 4.1.9
> >
> > o   That worked as expected
> >
> > -         Automatic Backup of Engine DB in the night
> >
> > -         Upgraded Engine from 4.1.9 to 4.2.1
> >
> > o   That worked fine
> >
> > -         Noticed Issues with the HA support for HE
> >
> > o   Cause was not having the latest ovirt-ha agent/broker version on hosts
> >
> > -         After updating the first host with the latest packages for the
> > Agent/Broker engine was started twice
> >
> > o   As a result the Engine VM Disk was corrupted and there is no Backup of
> > the Disk
> >
> > o   There is also no Backup of the Engine DB with version 4.2
> >
> > -         VM disk was repaired with fsck.ext4, but DB is corrupt
> >
> > o   Can’t restore the Engine DB because the Backup DB from Engine V 4.1
> >
> > -         Rolled back all changes on Engine VM to 4.1.9 and imported Backup
> >
> > o   Checked for HA VMs to set as disabled and started the Engine
> >
> > -         Login is fine but the Engine is having trouble picking up and
> > information from the Hosts
> >
> > o   No information on running VMs or hosts status
> >
> > -         Final Situation
> >
> > o   2 Hosts have VMs still running and I can’t stop those
> >
> > o   I still have the image of my corrupted Engine VM (v4.2)
> >
> >
> >
> > Since there were no major changes after upgrading from 4.1 to 4.2,
> > would it be possible to manually restore the 4.1 DB to the 4.2 Engine
> > VM to this up and running again or are there modifications made to the
> > DB on upgrading that are relevant for this ?
> 
> engine-backup requires restoring to the same version used to take the backup,
> with a single exception - on 4.0, it can restore 3.6.
> 
> It's very easy to patch it to allow also 4.1->4.2, search inside it for
> "VALID_BACKUP_RESTORE_PAIRS". However, I do not think anyone ever
> tested this, so no idea might break. In 3.6->4.0 days, we did have to fix a few
> other things, notably apache httpd and iptables->firewalld:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1318580
> 
> > All my work on rolling back to 4.1.9 with the DB restore failed as the
> > Engine is not capable of picking up information from the hosts.
> 
> No idea why, but not sure it's related to your restore flow.
> 
> > Lessons learned is to always make a copy/snapshot of the engine VM
> > disk before upgrading anything.
> 
> If it's a hosted-engine, this isn't supported - see my reply on the list ~ 1 hour
> ago...
> 
> > What are my options on getting
> > back to a working environment ? Any help or hint is greatly appreciated.
> 
> Restore again with either methods - what you tried, or patching engine-
> backup and restore directly into 4.2 - and if the engine fails to talk to the hosts,
> try to debug/fix this.
> 
> If you suspect corruption more severe that just the db, you can install a fresh
> engine machine from scratch and restore to it. If it's a hosted-engine, you'll
> need to deploy hosted-engine from scratch, check docs about hosted-engine
> backup/restore.

Will the setup of hosted engine from scratch require a new storage domain for the new engine or can I use the one that is already there ? What about the VMs running on my hosts, will they be effected by that ? It might be best to start with a fresh VM.

> 
> Best regards,
> --
> Didi