[ovirt-users] Hosted engine on gluster problem

Fri Apr 15 06:05:04 UTC 2016

On Thu, Apr 14, 2016 at 7:35 PM, Nir Soffer <nsoffer at redhat.com> wrote:

> On Wed, Apr 13, 2016 at 4:34 PM, Luiz Claudio Prazeres Goncalves
> <luizcpg at gmail.com> wrote:
> > Nir, here is the problem:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1298693
> >
> > When you do a hosted-engine --deploy and pick "glusterfs" you don't have
> a
> > way to define the mount options, therefore, the use of the
> > "backupvol-server", however when you create a storage domain from the UI
> you
> > can, like the attached screen shot.
> >
> >
> > In the hosted-engine --deploy, I would expect a flow which includes not
> only
> > the "gluster" entrypoint, but also the gluster mount options which is
> > missing today. This option would be optional, but would remove the single
> > point of failure described on the Bug 1298693.
> >
> > for example:
> >
> > Existing entry point on the "hosted-engine --deploy" flow
> > gluster1.xyz.com:/engine
>
> I agree, this feature must be supported.
>

It will, and it's currently targeted to 4.0.

>
> > Missing option on the "hosted-engine --deploy" flow :
> > backupvolfile-server=gluster2.xyz.com
> ,fetch-attempts=3,log-level=WARNING,log-file=/var/log/glusterfs/gluster_engine_domain.log
> >
> > Sandro, it seems to me a simple solution which can be easily fixed.
> >
> > What do you think?
> >
> > Regards
> > -Luiz
> >
> >
> >
> > 2016-04-13 4:15 GMT-03:00 Sandro Bonazzola <sbonazzo at redhat.com>:
> >>
> >>
> >>
> >> On Tue, Apr 12, 2016 at 6:47 PM, Nir Soffer <nsoffer at redhat.com> wrote:
> >>>
> >>> On Tue, Apr 12, 2016 at 3:05 PM, Luiz Claudio Prazeres Goncalves
> >>> <luizcpg at gmail.com> wrote:
> >>> > Hi Sandro, I've been using gluster with 3 external hosts for a while
> >>> > and
> >>> > things are working pretty well, however this single point of failure
> >>> > looks
> >>> > like a simple feature to implement,but critical to anyone who wants
> to
> >>> > use
> >>> > gluster on production  . This is not hyperconvergency which has other
> >>> > issues/implications. So , why not have this feature out on 3.6
> branch?
> >>> > It
> >>> > looks like just let vdsm use the 'backupvol-server' option when
> >>> > mounting the
> >>> > engine domain and make the property tests.
> >>>
> >>> Can you explain what is the problem, and what is the suggested
> solution?
> >>>
> >>> Engine and vdsm already support the backupvol-server option - you can
> >>> define this option in the storage domain options when you create a
> >>> gluster
> >>> storage domain. With this option vdsm should be able to connect to
> >>> gluster
> >>> storage domain even if a brick is down.
> >>>
> >>> If you don't have this option in engine , you probably cannot add it
> with
> >>> hosted
> >>> engine setup, since for editing it you must put the storage domain in
> >>> maintenance
> >>> and if you do this the engine vm will be killed :-) This is is one of
> >>> the issues with
> >>> engine managing the storage domain it runs on.
> >>>
> >>> I think the best way to avoid this issue, is to add a DNS entry
> >>> providing the addresses
> >>> of all the gluster bricks, and use this address for the gluster
> >>> storage domain. This way
> >>> the glusterfs mount helper can mount the domain even if one of the
> >>> gluster bricks
> >>> are down.
> >>>
> >>> Again, we will need some magic from the hosted engine developers to
> >>> modify the
> >>> address of the hosted engine gluster domain on existing system.
> >>
> >>
> >> Magic won't happen without a bz :-) please open one describing what's
> >> requested.
> >>
> >>
> >>>
> >>>
> >>> Nir
> >>>
> >>> >
> >>> > Could you add this feature to the next release of 3.6 branch?
> >>> >
> >>> > Thanks
> >>> > Luiz
> >>> >
> >>> > Em ter, 12 de abr de 2016 05:03, Sandro Bonazzola <
> sbonazzo at redhat.com>
> >>> > escreveu:
> >>> >>
> >>> >> On Mon, Apr 11, 2016 at 11:44 PM, Bond, Darryl <dbond at nrggos.com.au
> >
> >>> >> wrote:
> >>> >>>
> >>> >>> My setup is hyperconverged. I have placed my test results in
> >>> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1298693
> >>> >>>
> >>> >>
> >>> >> Ok, so you're aware about the limitation of the single point of
> >>> >> failure.
> >>> >> If you drop the host referenced in hosted engine configuration for
> the
> >>> >> initial setup it won't be able to connect to shared storage even if
> >>> >> the
> >>> >> other hosts in the cluster are up since the entry point is down.
> >>> >> Note that hyperconverged deployment is not supported in 3.6.
> >>> >>
> >>> >>
> >>> >>>
> >>> >>>
> >>> >>> Short description of setup:
> >>> >>>
> >>> >>> 3 hosts with 2 disks each set up with gluster replica 3 across the
> 6
> >>> >>> disks volume name hosted-engine.
> >>> >>>
> >>> >>> Hostname hosted-storage configured in /etc//hosts to point to the
> >>> >>> host1.
> >>> >>>
> >>> >>> Installed hosted engine on host1 with the hosted engine storage
> path
> >>> >>> =
> >>> >>> hosted-storage:/hosted-engine
> >>> >>>
> >>> >>> Install first engine on h1 successful. Hosts h2 and h3 added to the
> >>> >>> hosted engine. All works fine.
> >>> >>>
> >>> >>> Additional storage and non-hosted engine hosts added etc.
> >>> >>>
> >>> >>> Additional VMs added to hosted-engine storage (oVirt Reports VM and
> >>> >>> Cinder VM). Additional VM's are hosted by other storage - cinder
> and
> >>> >>> NFS.
> >>> >>>
> >>> >>> The system is in production.
> >>> >>>
> >>> >>>
> >>> >>> Engine can be migrated around with the web interface.
> >>> >>>
> >>> >>>
> >>> >>> - 3.6.4 upgrade released, follow the upgrade guide, engine is
> >>> >>> upgraded
> >>> >>> first , new Centos kernel requires host reboot.
> >>> >>>
> >>> >>> - Engine placed on h2 -  h3 into maintenance (local) upgrade and
> >>> >>> Reboot
> >>> >>> h3 - No issues - Local maintenance removed from h3.
> >>> >>>
> >>> >>> - Engine placed on h3 -  h2 into maintenance (local) upgrade and
> >>> >>> Reboot
> >>> >>> h2 - No issues - Local maintenance removed from h2.
> >>> >>>
> >>> >>> - Engine placed on h3 -h1 into mainteance (local) upgrade and
> reboot
> >>> >>> h1 -
> >>> >>> engine crashes and does not start elsewhere, VM(cinder)  on h3 on
> >>> >>> same
> >>> >>> gluster volume pauses.
> >>> >>>
> >>> >>> - Host 1 takes about 5 minutes to reboot (Enterprise box with all
> >>> >>> it's
> >>> >>> normal BIOS probing)
> >>> >>>
> >>> >>> - Engine starts after h1 comes back and stabilises
> >>> >>>
> >>> >>> - VM(cinder) unpauses itself,  VM(reports) continued fine the whole
> >>> >>> time.
> >>> >>> I can do no diagnosis on the 2 VMs as the engine is not available.
> >>> >>>
> >>> >>> - Local maintenance removed from h1
> >>> >>>
> >>> >>>
> >>> >>> I don't believe the issue is with gluster itself as the volume
> >>> >>> remains
> >>> >>> accessible on all hosts during this time albeit with a missing
> server
> >>> >>> (gluster volume status) as each gluster server is rebooted.
> >>> >>>
> >>> >>> Gluster was upgraded as part of the process, no issues were seen
> >>> >>> here.
> >>> >>>
> >>> >>>
> >>> >>> I have been able to duplicate the issue without the upgrade by
> >>> >>> following
> >>> >>> the same sort of timeline.
> >>> >>>
> >>> >>>
> >>> >>> ________________________________
> >>> >>> From: Sandro Bonazzola <sbonazzo at redhat.com>
> >>> >>> Sent: Monday, 11 April 2016 7:11 PM
> >>> >>> To: Richard Neuboeck; Simone Tiraboschi; Roy Golan; Martin Sivak;
> >>> >>> Sahina
> >>> >>> Bose
> >>> >>> Cc: Bond, Darryl; users
> >>> >>> Subject: Re: [ovirt-users] Hosted engine on gluster problem
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>> On Mon, Apr 11, 2016 at 9:37 AM, Richard Neuboeck
> >>> >>> <hawk at tbi.univie.ac.at<mailto:hawk at tbi.univie.ac.at>> wrote:
> >>> >>> Hi Darryl,
> >>> >>>
> >>> >>> I'm still experimenting with my oVirt installation so I tried to
> >>> >>> recreate the problems you've described.
> >>> >>>
> >>> >>> My setup has three HA hosts for virtualization and three machines
> >>> >>> for the gluster replica 3 setup.
> >>> >>>
> >>> >>> I manually migrated the Engine from the initial install host (one)
> >>> >>> to host three. Then shut down host one manually and interrupted the
> >>> >>> fencing mechanisms so the host stayed down. This didn't bother the
> >>> >>> Engine VM at all.
> >>> >>>
> >>> >>> Did you move the host one to maintenance before shutting down?
> >>> >>> Or is this a crash recovery test?
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>> To make things a bit more challenging I then shut down host three
> >>> >>> while running the Engine VM. Of course the Engine was down for some
> >>> >>> time until host two detected the problem. It started the Engine VM
> >>> >>> and everything seems to be running quite well without the initial
> >>> >>> install host.
> >>> >>>
> >>> >>> Thanks for the feedback!
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>> My only problem is that the HA agent on host two and three refuse
> to
> >>> >>> start after a reboot due to the fact that the configuration of the
> >>> >>> hosted engine is missing. I wrote another mail to
> >>> >>> users at ovirt.org<mailto:users at ovirt.org>
> >>> >>> about that.
> >>> >>>
> >>> >>> This is weird. Martin,  Simone can you please investigate on this?
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>> Cheers
> >>> >>> Richard
> >>> >>>
> >>> >>> On 04/08/2016 01:38 AM, Bond, Darryl wrote:
> >>> >>> > There seems to be a pretty severe bug with using hosted engine on
> >>> >>> > gluster.
> >>> >>> >
> >>> >>> > If the host that was used as the initial hosted-engine --deploy
> >>> >>> > host
> >>> >>> > goes away, the engine VM wil crash and cannot be restarted until
> >>> >>> > the host
> >>> >>> > comes back.
> >>> >>>
> >>> >>> is this an Hyperconverged setup?
> >>> >>>
> >>> >>>
> >>> >>> >
> >>> >>> > This is regardless of which host the engine was currently
> running.
> >>> >>> >
> >>> >>> >
> >>> >>> > The issue seems to be buried in the bowels of VDSM and is not an
> >>> >>> > issue
> >>> >>> > with gluster itself.
> >>> >>>
> >>> >>> Sahina, can you please investigate on this?
> >>> >>>
> >>> >>>
> >>> >>> >
> >>> >>> > The gluster filesystem is still accessable from the host that was
> >>> >>> > running the engine. The issue has been submitted to bugzilla but
> >>> >>> > the fix is
> >>> >>> > some way off (4.1).
> >>> >>> >
> >>> >>> >
> >>> >>> > Can my hosted engine be converted to use NFS (using the gluster
> NFS
> >>> >>> > server on the same filesystem) without rebuilding my hosted
> engine
> >>> >>> > (ie
> >>> >>> > change domainType=glusterfs to domainType=nfs)?
> >>> >>>
> >>> >>> >
> >>> >>> > What effect would that have on the hosted-engine storage domain
> >>> >>> > inside
> >>> >>> > oVirt, ie would the same filesystem be mounted twice or would it
> >>> >>> > just break.
> >>> >>> >
> >>> >>> >
> >>> >>> > Will this actually fix the problem, does it have the same issue
> >>> >>> > when
> >>> >>> > the hosted engine is on NFS?
> >>> >>> >
> >>> >>> >
> >>> >>> > Darryl
> >>> >>> >
> >>> >>> >
> >>> >>> >
> >>> >>> >
> >>> >>> > ________________________________
> >>> >>> >
> >>> >>> > The contents of this electronic message and any attachments are
> >>> >>> > intended only for the addressee and may contain legally
> privileged,
> >>> >>> > personal, sensitive or confidential information. If you are not
> the
> >>> >>> > intended
> >>> >>> > addressee, and have received this email, any transmission,
> >>> >>> > distribution,
> >>> >>> > downloading, printing or photocopying of the contents of this
> >>> >>> > message or
> >>> >>> > attachments is strictly prohibited. Any legal privilege or
> >>> >>> > confidentiality
> >>> >>> > attached to this message and attachments is not waived, lost or
> >>> >>> > destroyed by
> >>> >>> > reason of delivery to any person other than intended addressee.
> If
> >>> >>> > you have
> >>> >>> > received this message and are not the intended addressee you
> should
> >>> >>> > notify
> >>> >>> > the sender by return email and destroy all copies of the message
> >>> >>> > and any
> >>> >>> > attachments. Unless expressly attributed, the views expressed in
> >>> >>> > this email
> >>> >>> > do not necessarily represent the views of the company.
> >>> >>> > _______________________________________________
> >>> >>> > Users mailing list
> >>> >>> > Users at ovirt.org<mailto:Users at ovirt.org>
> >>> >>> > http://lists.ovirt.org/mailman/listinfo/users
> >>> >>> >
> >>> >>>
> >>> >>>
> >>> >>> --
> >>> >>> /dev/null
> >>> >>>
> >>> >>>
> >>> >>> _______________________________________________
> >>> >>> Users mailing list
> >>> >>> Users at ovirt.org<mailto:Users at ovirt.org>
> >>> >>> http://lists.ovirt.org/mailman/listinfo/users
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>> --
> >>> >>> Sandro Bonazzola
> >>> >>> Better technology. Faster innovation. Powered by community
> >>> >>> collaboration.
> >>> >>> See how it works at redhat.com<http://redhat.com>
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Sandro Bonazzola
> >>> >> Better technology. Faster innovation. Powered by community
> >>> >> collaboration.
> >>> >> See how it works at redhat.com
> >>> >> _______________________________________________
> >>> >> Users mailing list
> >>> >> Users at ovirt.org
> >>> >> http://lists.ovirt.org/mailman/listinfo/users
> >>> >
> >>> >
> >>> > _______________________________________________
> >>> > Users mailing list
> >>> > Users at ovirt.org
> >>> > http://lists.ovirt.org/mailman/listinfo/users
> >>> >
> >>
> >>
> >>
> >>
> >> --
> >> Sandro Bonazzola
> >> Better technology. Faster innovation. Powered by community
> collaboration.
> >> See how it works at redhat.com
> >
> >
>

-- 
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160415/b3a1ad9c/attachment-0001.html>