[ovirt-users] upgrade hosted-engine os ( not hosts )

[Paul Groeneweg | Pazion]

Op do 30 jun. 2016 om 21:55 schreef Yedidyah Bar David <didi at redhat.com>:

> On Thu, Jun 30, 2016 at 10:43 PM, Paul Groeneweg | Pazion
> <paul at pazion.nl> wrote:
> >
> >
> > Op do 30 jun. 2016 om 21:17 schreef Yedidyah Bar David <didi at redhat.com
> >:
> >>
> >> On Thu, Jun 30, 2016 at 9:46 PM, Paul Groeneweg | Pazion <
> paul at pazion.nl>
> >> wrote:
> >> > Things are getting more clear to me, thanks!
> >> > And this whole upgrade is something I should prepare well and not try
> to
> >> > do
> >> > in a few hours at night :)
> >> >
> >> > Resuming, there are multiple solutions:
> >> >
> >> > 1. upgrade el6 to el7
> >> > no go => I need to upgrade postgresql and redhat warns about upgrading
> >> > 6.6
> >> > to 7 because of newer packages
> >> > 2. install a new HE on new Host new storage domain
> >> > a lot of work  => I have to turn off VMs, import storage domain into
> new
> >> > setup and reinstall hosts.
> >> > 3. install a new HE on new Host existing storage domain
> >> > possibly not without issues => I have to manually update references to
> >> > the
> >> > hosted storage or better wait till there is an upgrade tool.
> >> > 4. (re)install HE in current VMS container.
> >> > => I can keep my Hosts and reference to the storage domain.
> >> >
> >> > So I guess my best bet is to go for option 4.
> >> >
> >> > The flow I am planning to follow would be:
> >> >
> >> > 1. Move to global maintenance ( keep VMs running )
> >> > 2. backup ( I have also a complete disk install of old hosted-engine )
> >> > 3. reboot with different conf, to boot from cd
> >> > 4. acces console and run install + import the backup in the engine vm
> >> > ? 5. run engine-setup to configure new install with engine restore
> data
> >> > 6. reboot hosted-engine
> >> > 7. Remove from global maintenance
> >> > 8. launch web gui and I should be able to manage all still running VMs
> >> > again.
> >> >
> >> > Would above be working or am I missing something and I  doubt step 5.
> Is
> >> > this needed?
> >>
> >> Yes.
> >>
> >> In a normal backup/restore flow, it's less important, although
> officially
> >> it's mandatory. But in above flow, it's not just backup/restore, but
> also
> >> upgrade from 3.6 to 4.0.
> >>
> >> In principle you can also do all of the above with 3.6, at least inside
> >> the engine, verify that all seems good, then upgrade to 4.0 later. See
> >> also:
> >>
> >> https://bugzilla.redhat.com/1332463
> >>
> >> But I do not remember reports from people trying it.
> >>
> > Good point, I planned to first get 3.6 running on el7 and from there
> upgrade
> > hosted-engine to 4.0 ( between step 6 and 7 ). Altough this way I keep
> old
> > 3.6 files on my fresh installl...what would you advice? directly install
> 4.0
> > on new el7 host? ( I am running 3.6.7, so above bug is probably fixed ).
>
> If you look at the bug page you see it's fixed in 3.6.6. And even if you
> had
> something earlier you should have been fine, as the fix works also for
> earlier
> 3.6 upgraded to 3.6.6 (or later).
>
> The question whether to first migrate from el6/3.6 to el7/3.6 and later to
> 4.0
> or do this in a single jump is a matter of your motivation and your style.
>
> Do you prefer to do large changes at once, or gradually?
>
> I want to be able do a roll back when thing don't work, so I think I start
with upgrade to el7 with 3.6.
Not sure if it would happen, but this way I don't change anything version
related stuff on the storage side where the VMs reside.



> Do you have some specific new features you want in 4.0?
>
>
Currently I have some issues where my OVF image files of the hosted engine
storage is not updated ( timestamp on hosts folders/ files stay the same )
I am hoping this broken thing is fixed once I have upgraded.
There are further no specific issues I want to upgrade, just to stay
current.
I am running oVirt since 3.4 and it took ( and still takes ) time to learn
about the different components, but overall I love how stable it functions
ov.


> >
> >
> >
> >>
> >> One very important thing you did not mention: Try this on a test system,
> >> including rollback to your backup, after you finished everything and
> >> (supposedly) found out something broken. Better safe than sorry :-)
> >
> >
> > A test setup might provide a good practice. But requires extra spare
> > hosts...
>
> You can test using nested-kvm.
>
> That might be a good idea, will certainly review!


> >I do have a bare metal restore option. So I am able ( with cd boot
> > and console mode) to restore hosted engine within an hour to previous
> state
> > :-)
>
> Including the shared storage? Just mentioning. With decent shared storage
> you probably just revert to a snapshot.
>
>
My Hostedengine storage is mounted as NFS, so backingup/restoring nfs
folder on server might be even easier :)

Hey, these are you own systems, you know how much you trust your tools etc.
> Just pointing out. Also, mainly, for other potential readers of this
> thread...
>
> Thanks!




> >
> > Thanks a lot for all info!
> >
> >>
> >> Best,
> >>
> >> >
> >> > Best Regards,
> >> > Paul
> >> >
> >> >
> >> >
> >> > Op do 30 jun. 2016 om 14:44 schreef Simone Tiraboschi
> >> > <stirabos at redhat.com>:
> >> >>
> >> >> On Thu, Jun 30, 2016 at 8:34 AM, Paul Groeneweg | Pazion
> >> >> <paul at pazion.nl>
> >> >> wrote:
> >> >> > Hi Yedidyah,
> >> >> >
> >> >> > Thank you for the comprehensive answers.
> >> >> >
> >> >> > I think I go for a complete reinstall ( read also OS upgrade tool
> is
> >> >> > not
> >> >> > adviced on 6.6 or higher as there might be newer packages as on 7
> ).
> >> >> > No
> >> >> > doubting to re-use current VM or setup from scratch ( fresh host
> with
> >> >> > new
> >> >> > hosted-engine and existing storage domein ).
> >> >>
> >> >> If you are planning to redeploy hosted-engien from scratch and
> >> >> restoring on the new engine DB a backup of the previous one, please
> >> >> carefully consider this:
> >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1240466#c21
> >> >>
> >> >> So, if you choose that path you'll have also to manually remove any
> >> >> reference to the previous hosted-engine from the restored DB.
> >> >> I'd strongly suggest you to wait for the upgrade tool to be fixed
> >> >> since manually doing this upgrade can be really error prone.
> >> >>
> >> >> > You explain the steps ( 1 to 6 ), but then don't talk about storage
> >> >> > domain
> >> >> > import.
> >> >> > Does it mean, when I reinstall the hosted-engine in the current he
> VM
> >> >> > and
> >> >> > restore an engine-backup ( step 5 ) I am able to start vm from Host
> >> >> > and
> >> >> > it
> >> >> > is still connected to the master storage ( so no need for storage
> >> >> > import) ?
> >> >> >
> >> >> > Best Regards,
> >> >> > Paul Groeneweg
> >> >> >
> >> >> >
> >> >> > Op do 30 jun. 2016 om 08:00 schreef Yedidyah Bar David
> >> >> > <didi at redhat.com>:
> >> >> >>
> >> >> >> On Wed, Jun 29, 2016 at 10:07 PM, Paul Groeneweg | Pazion
> >> >> >> <paul at pazion.nl> wrote:
> >> >> >> >
> >> >> >> > I am looking for a way to get my hosted-engine running on el7
> so I
> >> >> >> > can
> >> >> >> > upgrade to oVirt 4.0. Currently my hosts already run el7, but my
> >> >> >> > hosted-engine is still el6.
> >> >> >> >
> >> >> >> > I read
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> https://www.ovirt.org/documentation/how-to/hosted-engine-host-OS-upgrade/
> >> >> >> > but this is only about the hosts.
> >> >> >> >
> >> >> >> > I read
> https://www.ovirt.org/documentation/how-to/hosted-engine/,
> >> >> >> > but
> >> >> >> > it
> >> >> >> > only mentions upgrade of the hosted-engine software, not the OS.
> >> >> >> >
> >> >> >> > I understood I can do a fresh hosted-engine install, and then
> >> >> >> > import
> >> >> >> > my
> >> >> >> > storage domain to the new hosted engine, but:
> >> >> >> >
> >> >> >> > - Do I need to restore my hosted engine database? ( like
> described
> >> >> >> > here:
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> http://www.ovirt.org/develop/developer-guide/engine/migrate-to-hosted-engine/
> >> >> >> > )
> >> >> >>
> >> >> >> You might not have to, if you only care about the imported VMs
> from
> >> >> >> your
> >> >> >> storage. This will not keep other configuration, such as
> >> >> >> users/roles/permissions
> >> >> >> etc.
> >> >> >>
> >> >> >> > - Can I directly install hosted-engine 4.0 and then import the
> >> >> >> > storage
> >> >> >> > domain? Or should I install same hosted-engine version?
> >> >> >>
> >> >> >> AFAIK 4.0 engine can import 3.6 storage domains without problem.
> >> >> >>
> >> >> >> > - Do I first need another master storage domain or can I
> directly
> >> >> >> > import
> >> >> >> > my
> >> >> >> > old master storage domain?
> >> >> >>
> >> >> >> No idea. Even if you do, you can create a small empty one and
> later
> >> >> >> remove
> >> >> >> it.
> >> >> >>
> >> >> >> > - When importing the storage domain what is the risk it fails (
> I
> >> >> >> > have
> >> >> >> > backups, but it would cost a day to restore all )
> >> >> >>
> >> >> >> No idea, but IIRC we got many successful reports and at most few
> >> >> >> failures
> >> >> >> for this.
> >> >> >>
> >> >> >> > - How long would import take? few minutes or hours? ( I want to
> >> >> >> > keep
> >> >> >> > down
> >> >> >> > time as low as possible ).
> >> >> >>
> >> >> >> Again no idea. Perhaps do some test?
> >> >> >>
> >> >> >> >
> >> >> >> > Another option would be upgrade the OS ( with
> redhat-upgrade-tool
> >> >> >> > )
> >> >> >> > or
> >> >> >> > is
> >> >> >> > this a path for disaster?
> >> >> >>
> >> >> >> Didn't work for us well, so we decided to not support it. If you
> >> >> >> decide
> >> >> >> to
> >> >> >> try,
> >> >> >> make sure you test carefully beforehand. From ovirt's POV:
> >> >> >> 1. You'll need to handle postgresql upgrade.
> >> >> >> 2. Right after OS upgrade, you'll still have (I think) el6
> packages
> >> >> >> of the engine. It will hopefully be in a good-enough state for
> >> >> >> upgrade
> >> >> >> to 4.0, but we didn't test this.
> >> >> >> 3. Specifically, if upgrade fails, rollback will most likely not
> >> >> >> work,
> >> >> >> so you'll have to manually handle this - take a full vm backup and
> >> >> >> make
> >> >> >> sure you can restore it.
> >> >> >>
> >> >> >> >
> >> >> >> > I hope someone can tell me how I can smoothly upgrade my
> >> >> >> > hosted-engine
> >> >> >> > up to
> >> >> >> > el7 and run oVirt 4.
> >> >> >>
> >> >> >> We are working on a tool/wizard to help with this process. It used
> >> >> >> to
> >> >> >> work,
> >> >> >> but at some point it was decided that one of the actions it does
> is
> >> >> >> risky
> >> >> >> and was blocked, thus the tool is broken currently.
> >> >> >>
> >> >> >> You can invoke the tool by running: 'hosted-engine
> >> >> >> --upgrade-appliance'.
> >> >> >> As noted above, this is currently broken.
> >> >> >>
> >> >> >> There are several open bugs about it, e.g.:
> >> >> >>
> >> >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1319457
> >> >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1343425
> >> >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1343593 (closed,
> this is
> >> >> >> what broke the tool)
> >> >> >>
> >> >> >> Basically, you can manually do what the tool is supposed to do:
> >> >> >> 1. Make sure state is clean and stable (no running/pending storage
> >> >> >> actions,
> >> >> >> no VMs in the middle of migration etc), all clusters are compat
> >> >> >> level
> >> >> >> 3.6,
> >> >> >> etc.
> >> >> >> 2. Move to global maintenance
> >> >> >> 3. backup the engine using engine-backup and keep the backup
> >> >> >> elsewhere
> >> >> >> 4. Reinstall engine vm with el7 and 4.0 engine (the tool will use
> >> >> >> the
> >> >> >> engine
> >> >> >> appliance, you might too but not sure how exactly).
> >> >> >> 5. Restore the backup and run engine-setup.
> >> >> >> 6. If all looks ok, leave global maintenance.
> >> >> >>
> >> >> >> If you manually keep a full backup of the engine vm before step 4,
> >> >> >> you might be able to restore this backup if there are problems.
> >> >> >> Doing this in the provided tool is currently the main blocking
> issue
> >> >> >> for it. Hopefully will be provided in 4.0.1.
> >> >> >>
> >> >> >> Best,
> >> >> >> --
> >> >> >> Didi
> >> >> >
> >> >> >
> >> >> > _______________________________________________
> >> >> > Users mailing list
> >> >> > Users at ovirt.org
> >> >> > http://lists.ovirt.org/mailman/listinfo/users
> >> >> >
> >>
> >>
> >>
> >> --
> >> Didi
>
>
>
> --
> Didi
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160630/4e947efb/attachment-0001.html>