[ovirt-users] upgrade hosted-engine os ( not hosts )

[Yedidyah Bar David]

On Thu, Jun 30, 2016 at 10:43 PM, Paul Groeneweg | Pazion
<paul at pazion.nl> wrote:
>
>
> Op do 30 jun. 2016 om 21:17 schreef Yedidyah Bar David <didi at redhat.com>:
>>
>> On Thu, Jun 30, 2016 at 9:46 PM, Paul Groeneweg | Pazion <paul at pazion.nl>
>> wrote:
>> > Things are getting more clear to me, thanks!
>> > And this whole upgrade is something I should prepare well and not try to
>> > do
>> > in a few hours at night :)
>> >
>> > Resuming, there are multiple solutions:
>> >
>> > 1. upgrade el6 to el7
>> > no go => I need to upgrade postgresql and redhat warns about upgrading
>> > 6.6
>> > to 7 because of newer packages
>> > 2. install a new HE on new Host new storage domain
>> > a lot of work  => I have to turn off VMs, import storage domain into new
>> > setup and reinstall hosts.
>> > 3. install a new HE on new Host existing storage domain
>> > possibly not without issues => I have to manually update references to
>> > the
>> > hosted storage or better wait till there is an upgrade tool.
>> > 4. (re)install HE in current VMS container.
>> > => I can keep my Hosts and reference to the storage domain.
>> >
>> > So I guess my best bet is to go for option 4.
>> >
>> > The flow I am planning to follow would be:
>> >
>> > 1. Move to global maintenance ( keep VMs running )
>> > 2. backup ( I have also a complete disk install of old hosted-engine )
>> > 3. reboot with different conf, to boot from cd
>> > 4. acces console and run install + import the backup in the engine vm
>> > ? 5. run engine-setup to configure new install with engine restore data
>> > 6. reboot hosted-engine
>> > 7. Remove from global maintenance
>> > 8. launch web gui and I should be able to manage all still running VMs
>> > again.
>> >
>> > Would above be working or am I missing something and I  doubt step 5. Is
>> > this needed?
>>
>> Yes.
>>
>> In a normal backup/restore flow, it's less important, although officially
>> it's mandatory. But in above flow, it's not just backup/restore, but also
>> upgrade from 3.6 to 4.0.
>>
>> In principle you can also do all of the above with 3.6, at least inside
>> the engine, verify that all seems good, then upgrade to 4.0 later. See
>> also:
>>
>> https://bugzilla.redhat.com/1332463
>>
>> But I do not remember reports from people trying it.
>>
> Good point, I planned to first get 3.6 running on el7 and from there upgrade
> hosted-engine to 4.0 ( between step 6 and 7 ). Altough this way I keep old
> 3.6 files on my fresh installl...what would you advice? directly install 4.0
> on new el7 host? ( I am running 3.6.7, so above bug is probably fixed ).

If you look at the bug page you see it's fixed in 3.6.6. And even if you had
something earlier you should have been fine, as the fix works also for earlier
3.6 upgraded to 3.6.6 (or later).

The question whether to first migrate from el6/3.6 to el7/3.6 and later to 4.0
or do this in a single jump is a matter of your motivation and your style.

Do you prefer to do large changes at once, or gradually?

Do you have some specific new features you want in 4.0?

>
>
>
>>
>> One very important thing you did not mention: Try this on a test system,
>> including rollback to your backup, after you finished everything and
>> (supposedly) found out something broken. Better safe than sorry :-)
>
>
> A test setup might provide a good practice. But requires extra spare
> hosts...

You can test using nested-kvm.

>I do have a bare metal restore option. So I am able ( with cd boot
> and console mode) to restore hosted engine within an hour to previous state
> :-)

Including the shared storage? Just mentioning. With decent shared storage
you probably just revert to a snapshot.

Hey, these are you own systems, you know how much you trust your tools etc.
Just pointing out. Also, mainly, for other potential readers of this thread...

>
> Thanks a lot for all info!
>
>>
>> Best,
>>
>> >
>> > Best Regards,
>> > Paul
>> >
>> >
>> >
>> > Op do 30 jun. 2016 om 14:44 schreef Simone Tiraboschi
>> > <stirabos at redhat.com>:
>> >>
>> >> On Thu, Jun 30, 2016 at 8:34 AM, Paul Groeneweg | Pazion
>> >> <paul at pazion.nl>
>> >> wrote:
>> >> > Hi Yedidyah,
>> >> >
>> >> > Thank you for the comprehensive answers.
>> >> >
>> >> > I think I go for a complete reinstall ( read also OS upgrade tool is
>> >> > not
>> >> > adviced on 6.6 or higher as there might be newer packages as on 7 ).
>> >> > No
>> >> > doubting to re-use current VM or setup from scratch ( fresh host with
>> >> > new
>> >> > hosted-engine and existing storage domein ).
>> >>
>> >> If you are planning to redeploy hosted-engien from scratch and
>> >> restoring on the new engine DB a backup of the previous one, please
>> >> carefully consider this:
>> >> https://bugzilla.redhat.com/show_bug.cgi?id=1240466#c21
>> >>
>> >> So, if you choose that path you'll have also to manually remove any
>> >> reference to the previous hosted-engine from the restored DB.
>> >> I'd strongly suggest you to wait for the upgrade tool to be fixed
>> >> since manually doing this upgrade can be really error prone.
>> >>
>> >> > You explain the steps ( 1 to 6 ), but then don't talk about storage
>> >> > domain
>> >> > import.
>> >> > Does it mean, when I reinstall the hosted-engine in the current he VM
>> >> > and
>> >> > restore an engine-backup ( step 5 ) I am able to start vm from Host
>> >> > and
>> >> > it
>> >> > is still connected to the master storage ( so no need for storage
>> >> > import) ?
>> >> >
>> >> > Best Regards,
>> >> > Paul Groeneweg
>> >> >
>> >> >
>> >> > Op do 30 jun. 2016 om 08:00 schreef Yedidyah Bar David
>> >> > <didi at redhat.com>:
>> >> >>
>> >> >> On Wed, Jun 29, 2016 at 10:07 PM, Paul Groeneweg | Pazion
>> >> >> <paul at pazion.nl> wrote:
>> >> >> >
>> >> >> > I am looking for a way to get my hosted-engine running on el7 so I
>> >> >> > can
>> >> >> > upgrade to oVirt 4.0. Currently my hosts already run el7, but my
>> >> >> > hosted-engine is still el6.
>> >> >> >
>> >> >> > I read
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > https://www.ovirt.org/documentation/how-to/hosted-engine-host-OS-upgrade/
>> >> >> > but this is only about the hosts.
>> >> >> >
>> >> >> > I read https://www.ovirt.org/documentation/how-to/hosted-engine/,
>> >> >> > but
>> >> >> > it
>> >> >> > only mentions upgrade of the hosted-engine software, not the OS.
>> >> >> >
>> >> >> > I understood I can do a fresh hosted-engine install, and then
>> >> >> > import
>> >> >> > my
>> >> >> > storage domain to the new hosted engine, but:
>> >> >> >
>> >> >> > - Do I need to restore my hosted engine database? ( like described
>> >> >> > here:
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > http://www.ovirt.org/develop/developer-guide/engine/migrate-to-hosted-engine/
>> >> >> > )
>> >> >>
>> >> >> You might not have to, if you only care about the imported VMs from
>> >> >> your
>> >> >> storage. This will not keep other configuration, such as
>> >> >> users/roles/permissions
>> >> >> etc.
>> >> >>
>> >> >> > - Can I directly install hosted-engine 4.0 and then import the
>> >> >> > storage
>> >> >> > domain? Or should I install same hosted-engine version?
>> >> >>
>> >> >> AFAIK 4.0 engine can import 3.6 storage domains without problem.
>> >> >>
>> >> >> > - Do I first need another master storage domain or can I directly
>> >> >> > import
>> >> >> > my
>> >> >> > old master storage domain?
>> >> >>
>> >> >> No idea. Even if you do, you can create a small empty one and later
>> >> >> remove
>> >> >> it.
>> >> >>
>> >> >> > - When importing the storage domain what is the risk it fails ( I
>> >> >> > have
>> >> >> > backups, but it would cost a day to restore all )
>> >> >>
>> >> >> No idea, but IIRC we got many successful reports and at most few
>> >> >> failures
>> >> >> for this.
>> >> >>
>> >> >> > - How long would import take? few minutes or hours? ( I want to
>> >> >> > keep
>> >> >> > down
>> >> >> > time as low as possible ).
>> >> >>
>> >> >> Again no idea. Perhaps do some test?
>> >> >>
>> >> >> >
>> >> >> > Another option would be upgrade the OS ( with redhat-upgrade-tool
>> >> >> > )
>> >> >> > or
>> >> >> > is
>> >> >> > this a path for disaster?
>> >> >>
>> >> >> Didn't work for us well, so we decided to not support it. If you
>> >> >> decide
>> >> >> to
>> >> >> try,
>> >> >> make sure you test carefully beforehand. From ovirt's POV:
>> >> >> 1. You'll need to handle postgresql upgrade.
>> >> >> 2. Right after OS upgrade, you'll still have (I think) el6 packages
>> >> >> of the engine. It will hopefully be in a good-enough state for
>> >> >> upgrade
>> >> >> to 4.0, but we didn't test this.
>> >> >> 3. Specifically, if upgrade fails, rollback will most likely not
>> >> >> work,
>> >> >> so you'll have to manually handle this - take a full vm backup and
>> >> >> make
>> >> >> sure you can restore it.
>> >> >>
>> >> >> >
>> >> >> > I hope someone can tell me how I can smoothly upgrade my
>> >> >> > hosted-engine
>> >> >> > up to
>> >> >> > el7 and run oVirt 4.
>> >> >>
>> >> >> We are working on a tool/wizard to help with this process. It used
>> >> >> to
>> >> >> work,
>> >> >> but at some point it was decided that one of the actions it does is
>> >> >> risky
>> >> >> and was blocked, thus the tool is broken currently.
>> >> >>
>> >> >> You can invoke the tool by running: 'hosted-engine
>> >> >> --upgrade-appliance'.
>> >> >> As noted above, this is currently broken.
>> >> >>
>> >> >> There are several open bugs about it, e.g.:
>> >> >>
>> >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1319457
>> >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1343425
>> >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1343593 (closed, this is
>> >> >> what broke the tool)
>> >> >>
>> >> >> Basically, you can manually do what the tool is supposed to do:
>> >> >> 1. Make sure state is clean and stable (no running/pending storage
>> >> >> actions,
>> >> >> no VMs in the middle of migration etc), all clusters are compat
>> >> >> level
>> >> >> 3.6,
>> >> >> etc.
>> >> >> 2. Move to global maintenance
>> >> >> 3. backup the engine using engine-backup and keep the backup
>> >> >> elsewhere
>> >> >> 4. Reinstall engine vm with el7 and 4.0 engine (the tool will use
>> >> >> the
>> >> >> engine
>> >> >> appliance, you might too but not sure how exactly).
>> >> >> 5. Restore the backup and run engine-setup.
>> >> >> 6. If all looks ok, leave global maintenance.
>> >> >>
>> >> >> If you manually keep a full backup of the engine vm before step 4,
>> >> >> you might be able to restore this backup if there are problems.
>> >> >> Doing this in the provided tool is currently the main blocking issue
>> >> >> for it. Hopefully will be provided in 4.0.1.
>> >> >>
>> >> >> Best,
>> >> >> --
>> >> >> Didi
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > Users mailing list
>> >> > Users at ovirt.org
>> >> > http://lists.ovirt.org/mailman/listinfo/users
>> >> >
>>
>>
>>
>> --
>> Didi



-- 
Didi