[ovirt-users] upgrade hosted-engine os ( not hosts )

[Paul Groeneweg | Pazion]

Hi,

I am finally planning to start our oVirt hosted engine upgrade. I am not
sure how stable my hosted engine storage is ( it shows wrong status in web
GUI ) So I arranged a new shared storage and plan to do a fresh install (
altough I noticed there is also an option to upgrade "hosted-engine
--upgrade-appliance" ).

After I installed the oVirt 4 hosted engine on a host node I want to move
all my running VMs to the new environment. With VMs at first running on the
current data storage domain. My storage domain is FC based.

My idea is to:

1. stop all VMs
2. detach Storage domain on old ovirt 3.6 environment ( and keep data ).
3. import and activate the storage domain on the new ovirt 4 environment.

To prevent any data loss or copy too much data I have the follwoing
questions:

1. Is this storage domain import from 3.6 to 4.0 available?
2. I read I should import all VMs, this means I need to copy all data or is
it just a matter of importing a config, so all VM data /image remains on
the imported storage domain?
3. When for some reason I want to roll back when I have imported the
storage domain without having any VM imported, can I just detach and
re-attach the storage domain to the previous 3.6 environment?

If there might be a better path to get my VMs to a clean oVirt 4.0
environment I am glad to hear.

Thanks in advance!
Kind regards,
Paul

Op do 30 jun. 2016 om 23:04 schreef Paul Groeneweg | Pazion <paul at pazion.nl
>:

>
> Op do 30 jun. 2016 om 21:55 schreef Yedidyah Bar David <didi at redhat.com>:
>
> On Thu, Jun 30, 2016 at 10:43 PM, Paul Groeneweg | Pazion
> <paul at pazion.nl> wrote:
> >
> >
> > Op do 30 jun. 2016 om 21:17 schreef Yedidyah Bar David <didi at redhat.com
> >:
> >>
> >> On Thu, Jun 30, 2016 at 9:46 PM, Paul Groeneweg | Pazion <
> paul at pazion.nl>
> >> wrote:
> >> > Things are getting more clear to me, thanks!
> >> > And this whole upgrade is something I should prepare well and not try
> to
> >> > do
> >> > in a few hours at night :)
> >> >
> >> > Resuming, there are multiple solutions:
> >> >
> >> > 1. upgrade el6 to el7
> >> > no go => I need to upgrade postgresql and redhat warns about upgrading
> >> > 6.6
> >> > to 7 because of newer packages
> >> > 2. install a new HE on new Host new storage domain
> >> > a lot of work  => I have to turn off VMs, import storage domain into
> new
> >> > setup and reinstall hosts.
> >> > 3. install a new HE on new Host existing storage domain
> >> > possibly not without issues => I have to manually update references to
> >> > the
> >> > hosted storage or better wait till there is an upgrade tool.
> >> > 4. (re)install HE in current VMS container.
> >> > => I can keep my Hosts and reference to the storage domain.
> >> >
> >> > So I guess my best bet is to go for option 4.
> >> >
> >> > The flow I am planning to follow would be:
> >> >
> >> > 1. Move to global maintenance ( keep VMs running )
> >> > 2. backup ( I have also a complete disk install of old hosted-engine )
> >> > 3. reboot with different conf, to boot from cd
> >> > 4. acces console and run install + import the backup in the engine vm
> >> > ? 5. run engine-setup to configure new install with engine restore
> data
> >> > 6. reboot hosted-engine
> >> > 7. Remove from global maintenance
> >> > 8. launch web gui and I should be able to manage all still running VMs
> >> > again.
> >> >
> >> > Would above be working or am I missing something and I  doubt step 5.
> Is
> >> > this needed?
> >>
> >> Yes.
> >>
> >> In a normal backup/restore flow, it's less important, although
> officially
> >> it's mandatory. But in above flow, it's not just backup/restore, but
> also
> >> upgrade from 3.6 to 4.0.
> >>
> >> In principle you can also do all of the above with 3.6, at least inside
> >> the engine, verify that all seems good, then upgrade to 4.0 later. See
> >> also:
> >>
> >> https://bugzilla.redhat.com/1332463
> >>
> >> But I do not remember reports from people trying it.
> >>
> > Good point, I planned to first get 3.6 running on el7 and from there
> upgrade
> > hosted-engine to 4.0 ( between step 6 and 7 ). Altough this way I keep
> old
> > 3.6 files on my fresh installl...what would you advice? directly install
> 4.0
> > on new el7 host? ( I am running 3.6.7, so above bug is probably fixed ).
>
> If you look at the bug page you see it's fixed in 3.6.6. And even if you
> had
> something earlier you should have been fine, as the fix works also for
> earlier
> 3.6 upgraded to 3.6.6 (or later).
>
> The question whether to first migrate from el6/3.6 to el7/3.6 and later to
> 4.0
> or do this in a single jump is a matter of your motivation and your style.
>
> Do you prefer to do large changes at once, or gradually?
>
> I want to be able do a roll back when thing don't work, so I think I start
> with upgrade to el7 with 3.6.
> Not sure if it would happen, but this way I don't change anything version
> related stuff on the storage side where the VMs reside.
>
>
>
> Do you have some specific new features you want in 4.0?
>
>
> Currently I have some issues where my OVF image files of the hosted engine
> storage is not updated ( timestamp on hosts folders/ files stay the same )
> I am hoping this broken thing is fixed once I have upgraded.
> There are further no specific issues I want to upgrade, just to stay
> current.
> I am running oVirt since 3.4 and it took ( and still takes ) time to learn
> about the different components, but overall I love how stable it functions
> ov.
>
>
> >
> >
> >
> >>
> >> One very important thing you did not mention: Try this on a test system,
> >> including rollback to your backup, after you finished everything and
> >> (supposedly) found out something broken. Better safe than sorry :-)
> >
> >
> > A test setup might provide a good practice. But requires extra spare
> > hosts...
>
> You can test using nested-kvm.
>
> That might be a good idea, will certainly review!
>
>
> >I do have a bare metal restore option. So I am able ( with cd boot
> > and console mode) to restore hosted engine within an hour to previous
> state
> > :-)
>
> Including the shared storage? Just mentioning. With decent shared storage
> you probably just revert to a snapshot.
>
>
> My Hostedengine storage is mounted as NFS, so backingup/restoring nfs
> folder on server might be even easier :)
>
> Hey, these are you own systems, you know how much you trust your tools etc.
> Just pointing out. Also, mainly, for other potential readers of this
> thread...
>
> Thanks!
>
>
>
>
> >
> > Thanks a lot for all info!
> >
> >>
> >> Best,
> >>
> >> >
> >> > Best Regards,
> >> > Paul
> >> >
> >> >
> >> >
> >> > Op do 30 jun. 2016 om 14:44 schreef Simone Tiraboschi
> >> > <stirabos at redhat.com>:
> >> >>
> >> >> On Thu, Jun 30, 2016 at 8:34 AM, Paul Groeneweg | Pazion
> >> >> <paul at pazion.nl>
> >> >> wrote:
> >> >> > Hi Yedidyah,
> >> >> >
> >> >> > Thank you for the comprehensive answers.
> >> >> >
> >> >> > I think I go for a complete reinstall ( read also OS upgrade tool
> is
> >> >> > not
> >> >> > adviced on 6.6 or higher as there might be newer packages as on 7
> ).
> >> >> > No
> >> >> > doubting to re-use current VM or setup from scratch ( fresh host
> with
> >> >> > new
> >> >> > hosted-engine and existing storage domein ).
> >> >>
> >> >> If you are planning to redeploy hosted-engien from scratch and
> >> >> restoring on the new engine DB a backup of the previous one, please
> >> >> carefully consider this:
> >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1240466#c21
> >> >>
> >> >> So, if you choose that path you'll have also to manually remove any
> >> >> reference to the previous hosted-engine from the restored DB.
> >> >> I'd strongly suggest you to wait for the upgrade tool to be fixed
> >> >> since manually doing this upgrade can be really error prone.
> >> >>
> >> >> > You explain the steps ( 1 to 6 ), but then don't talk about storage
> >> >> > domain
> >> >> > import.
> >> >> > Does it mean, when I reinstall the hosted-engine in the current he
> VM
> >> >> > and
> >> >> > restore an engine-backup ( step 5 ) I am able to start vm from Host
> >> >> > and
> >> >> > it
> >> >> > is still connected to the master storage ( so no need for storage
> >> >> > import) ?
> >> >> >
> >> >> > Best Regards,
> >> >> > Paul Groeneweg
> >> >> >
> >> >> >
> >> >> > Op do 30 jun. 2016 om 08:00 schreef Yedidyah Bar David
> >> >> > <didi at redhat.com>:
> >> >> >>
> >> >> >> On Wed, Jun 29, 2016 at 10:07 PM, Paul Groeneweg | Pazion
> >> >> >> <paul at pazion.nl> wrote:
> >> >> >> >
> >> >> >> > I am looking for a way to get my hosted-engine running on el7
> so I
> >> >> >> > can
> >> >> >> > upgrade to oVirt 4.0. Currently my hosts already run el7, but my
> >> >> >> > hosted-engine is still el6.
> >> >> >> >
> >> >> >> > I read
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> https://www.ovirt.org/documentation/how-to/hosted-engine-host-OS-upgrade/
> >> >> >> > but this is only about the hosts.
> >> >> >> >
> >> >> >> > I read
> https://www.ovirt.org/documentation/how-to/hosted-engine/,
> >> >> >> > but
> >> >> >> > it
> >> >> >> > only mentions upgrade of the hosted-engine software, not the OS.
> >> >> >> >
> >> >> >> > I understood I can do a fresh hosted-engine install, and then
> >> >> >> > import
> >> >> >> > my
> >> >> >> > storage domain to the new hosted engine, but:
> >> >> >> >
> >> >> >> > - Do I need to restore my hosted engine database? ( like
> described
> >> >> >> > here:
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> http://www.ovirt.org/develop/developer-guide/engine/migrate-to-hosted-engine/
> >> >> >> > )
> >> >> >>
> >> >> >> You might not have to, if you only care about the imported VMs
> from
> >> >> >> your
> >> >> >> storage. This will not keep other configuration, such as
> >> >> >> users/roles/permissions
> >> >> >> etc.
> >> >> >>
> >> >> >> > - Can I directly install hosted-engine 4.0 and then import the
> >> >> >> > storage
> >> >> >> > domain? Or should I install same hosted-engine version?
> >> >> >>
> >> >> >> AFAIK 4.0 engine can import 3.6 storage domains without problem.
> >> >> >>
> >> >> >> > - Do I first need another master storage domain or can I
> directly
> >> >> >> > import
> >> >> >> > my
> >> >> >> > old master storage domain?
> >> >> >>
> >> >> >> No idea. Even if you do, you can create a small empty one and
> later
> >> >> >> remove
> >> >> >> it.
> >> >> >>
> >> >> >> > - When importing the storage domain what is the risk it fails (
> I
> >> >> >> > have
> >> >> >> > backups, but it would cost a day to restore all )
> >> >> >>
> >> >> >> No idea, but IIRC we got many successful reports and at most few
> >> >> >> failures
> >> >> >> for this.
> >> >> >>
> >> >> >> > - How long would import take? few minutes or hours? ( I want to
> >> >> >> > keep
> >> >> >> > down
> >> >> >> > time as low as possible ).
> >> >> >>
> >> >> >> Again no idea. Perhaps do some test?
> >> >> >>
> >> >> >> >
> >> >> >> > Another option would be upgrade the OS ( with
> redhat-upgrade-tool
> >> >> >> > )
> >> >> >> > or
> >> >> >> > is
> >> >> >> > this a path for disaster?
> >> >> >>
> >> >> >> Didn't work for us well, so we decided to not support it. If you
> >> >> >> decide
> >> >> >> to
> >> >> >> try,
> >> >> >> make sure you test carefully beforehand. From ovirt's POV:
> >> >> >> 1. You'll need to handle postgresql upgrade.
> >> >> >> 2. Right after OS upgrade, you'll still have (I think) el6
> packages
> >> >> >> of the engine. It will hopefully be in a good-enough state for
> >> >> >> upgrade
> >> >> >> to 4.0, but we didn't test this.
> >> >> >> 3. Specifically, if upgrade fails, rollback will most likely not
> >> >> >> work,
> >> >> >> so you'll have to manually handle this - take a full vm backup and
> >> >> >> make
> >> >> >> sure you can restore it.
> >> >> >>
> >> >> >> >
> >> >> >> > I hope someone can tell me how I can smoothly upgrade my
> >> >> >> > hosted-engine
> >> >> >> > up to
> >> >> >> > el7 and run oVirt 4.
> >> >> >>
> >> >> >> We are working on a tool/wizard to help with this process. It used
> >> >> >> to
> >> >> >> work,
> >> >> >> but at some point it was decided that one of the actions it does
> is
> >> >> >> risky
> >> >> >> and was blocked, thus the tool is broken currently.
> >> >> >>
> >> >> >> You can invoke the tool by running: 'hosted-engine
> >> >> >> --upgrade-appliance'.
> >> >> >> As noted above, this is currently broken.
> >> >> >>
> >> >> >> There are several open bugs about it, e.g.:
> >> >> >>
> >> >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1319457
> >> >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1343425
> >> >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1343593 (closed,
> this is
> >> >> >> what broke the tool)
> >> >> >>
> >> >> >> Basically, you can manually do what the tool is supposed to do:
> >> >> >> 1. Make sure state is clean and stable (no running/pending storage
> >> >> >> actions,
> >> >> >> no VMs in the middle of migration etc), all clusters are compat
> >> >> >> level
> >> >> >> 3.6,
> >> >> >> etc.
> >> >> >> 2. Move to global maintenance
> >> >> >> 3. backup the engine using engine-backup and keep the backup
> >> >> >> elsewhere
> >> >> >> 4. Reinstall engine vm with el7 and 4.0 engine (the tool will use
> >> >> >> the
> >> >> >> engine
> >> >> >> appliance, you might too but not sure how exactly).
> >> >> >> 5. Restore the backup and run engine-setup.
> >> >> >> 6. If all looks ok, leave global maintenance.
> >> >> >>
> >> >> >> If you manually keep a full backup of the engine vm before step 4,
> >> >> >> you might be able to restore this backup if there are problems.
> >> >> >> Doing this in the provided tool is currently the main blocking
> issue
> >> >> >> for it. Hopefully will be provided in 4.0.1.
> >> >> >>
> >> >> >> Best,
> >> >> >> --
> >> >> >> Didi
> >> >> >
> >> >> >
> >> >> > _______________________________________________
> >> >> > Users mailing list
> >> >> > Users at ovirt.org
> >> >> > http://lists.ovirt.org/mailman/listinfo/users
> >> >> >
> >>
> >>
> >>
> >> --
> >> Didi
>
>
>
> --
> Didi
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170109/2eae8e90/attachment-0001.html>