[ovirt-users] Hosted engine on gluster problem

Thu Apr 14 17:07:24 UTC 2016

Sandro, any word here? Btw, I'm not talking about hyperconvergency in this
case, but 3 external gluster nodes using replica 3

Regards
Luiz

Em qua, 13 de abr de 2016 10:34, Luiz Claudio Prazeres Goncalves <
luizcpg at gmail.com> escreveu:

> Nir, here is the problem:
> https://bugzilla.redhat.com/show_bug.cgi?id=1298693
>
> When you do a hosted-engine --deploy and pick "glusterfs" you don't have a
> way to define the mount options, therefore, the use of the
> "backupvol-server", however when you create a storage domain from the UI
> you can, like the attached screen shot.
>
>
> In the hosted-engine --deploy, I would expect a flow which includes not
> only the "gluster" entrypoint, but also the gluster mount options which is
> missing today. This option would be optional, but would remove the single
> point of failure described on the Bug 1298693.
>
> for example:
>
> Existing entry point on the "hosted-engine --deploy" flow
> gluster1.xyz.com:/engine
>
>
> Missing option on the "hosted-engine --deploy" flow :
> backupvolfile-server=gluster2.xyz.com
> ,fetch-attempts=3,log-level=WARNING,log-file=/var/log/glusterfs/gluster_engine_domain.log
>
> Sandro, it seems to me a simple solution which can be easily fixed.
>
> What do you think?
>
> Regards
> -Luiz
>
>
>
> 2016-04-13 4:15 GMT-03:00 Sandro Bonazzola <sbonazzo at redhat.com>:
>
>>
>>
>> On Tue, Apr 12, 2016 at 6:47 PM, Nir Soffer <nsoffer at redhat.com> wrote:
>>
>>> On Tue, Apr 12, 2016 at 3:05 PM, Luiz Claudio Prazeres Goncalves
>>> <luizcpg at gmail.com> wrote:
>>> > Hi Sandro, I've been using gluster with 3 external hosts for a while
>>> and
>>> > things are working pretty well, however this single point of failure
>>> looks
>>> > like a simple feature to implement,but critical to anyone who wants to
>>> use
>>> > gluster on production  . This is not hyperconvergency which has other
>>> > issues/implications. So , why not have this feature out on 3.6 branch?
>>> It
>>> > looks like just let vdsm use the 'backupvol-server' option when
>>> mounting the
>>> > engine domain and make the property tests.
>>>
>>> Can you explain what is the problem, and what is the suggested solution?
>>>
>>> Engine and vdsm already support the backupvol-server option - you can
>>> define this option in the storage domain options when you create a
>>> gluster
>>> storage domain. With this option vdsm should be able to connect to
>>> gluster
>>> storage domain even if a brick is down.
>>>
>>> If you don't have this option in engine , you probably cannot add it
>>> with hosted
>>> engine setup, since for editing it you must put the storage domain in
>>> maintenance
>>> and if you do this the engine vm will be killed :-) This is is one of
>>> the issues with
>>> engine managing the storage domain it runs on.
>>>
>>> I think the best way to avoid this issue, is to add a DNS entry
>>> providing the addresses
>>> of all the gluster bricks, and use this address for the gluster
>>> storage domain. This way
>>> the glusterfs mount helper can mount the domain even if one of the
>>> gluster bricks
>>> are down.
>>>
>>> Again, we will need some magic from the hosted engine developers to
>>> modify the
>>> address of the hosted engine gluster domain on existing system.
>>>
>>
>> Magic won't happen without a bz :-) please open one describing what's
>> requested.
>>
>>
>>
>>>
>>> Nir
>>>
>>> >
>>> > Could you add this feature to the next release of 3.6 branch?
>>> >
>>> > Thanks
>>> > Luiz
>>> >
>>> > Em ter, 12 de abr de 2016 05:03, Sandro Bonazzola <sbonazzo at redhat.com
>>> >
>>> > escreveu:
>>> >>
>>> >> On Mon, Apr 11, 2016 at 11:44 PM, Bond, Darryl <dbond at nrggos.com.au>
>>> >> wrote:
>>> >>>
>>> >>> My setup is hyperconverged. I have placed my test results in
>>> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1298693
>>> >>>
>>> >>
>>> >> Ok, so you're aware about the limitation of the single point of
>>> failure.
>>> >> If you drop the host referenced in hosted engine configuration for the
>>> >> initial setup it won't be able to connect to shared storage even if
>>> the
>>> >> other hosts in the cluster are up since the entry point is down.
>>> >> Note that hyperconverged deployment is not supported in 3.6.
>>> >>
>>> >>
>>> >>>
>>> >>>
>>> >>> Short description of setup:
>>> >>>
>>> >>> 3 hosts with 2 disks each set up with gluster replica 3 across the 6
>>> >>> disks volume name hosted-engine.
>>> >>>
>>> >>> Hostname hosted-storage configured in /etc//hosts to point to the
>>> host1.
>>> >>>
>>> >>> Installed hosted engine on host1 with the hosted engine storage path
>>> =
>>> >>> hosted-storage:/hosted-engine
>>> >>>
>>> >>> Install first engine on h1 successful. Hosts h2 and h3 added to the
>>> >>> hosted engine. All works fine.
>>> >>>
>>> >>> Additional storage and non-hosted engine hosts added etc.
>>> >>>
>>> >>> Additional VMs added to hosted-engine storage (oVirt Reports VM and
>>> >>> Cinder VM). Additional VM's are hosted by other storage - cinder and
>>> NFS.
>>> >>>
>>> >>> The system is in production.
>>> >>>
>>> >>>
>>> >>> Engine can be migrated around with the web interface.
>>> >>>
>>> >>>
>>> >>> - 3.6.4 upgrade released, follow the upgrade guide, engine is
>>> upgraded
>>> >>> first , new Centos kernel requires host reboot.
>>> >>>
>>> >>> - Engine placed on h2 -  h3 into maintenance (local) upgrade and
>>> Reboot
>>> >>> h3 - No issues - Local maintenance removed from h3.
>>> >>>
>>> >>> - Engine placed on h3 -  h2 into maintenance (local) upgrade and
>>> Reboot
>>> >>> h2 - No issues - Local maintenance removed from h2.
>>> >>>
>>> >>> - Engine placed on h3 -h1 into mainteance (local) upgrade and reboot
>>> h1 -
>>> >>> engine crashes and does not start elsewhere, VM(cinder)  on h3 on
>>> same
>>> >>> gluster volume pauses.
>>> >>>
>>> >>> - Host 1 takes about 5 minutes to reboot (Enterprise box with all
>>> it's
>>> >>> normal BIOS probing)
>>> >>>
>>> >>> - Engine starts after h1 comes back and stabilises
>>> >>>
>>> >>> - VM(cinder) unpauses itself,  VM(reports) continued fine the whole
>>> time.
>>> >>> I can do no diagnosis on the 2 VMs as the engine is not available.
>>> >>>
>>> >>> - Local maintenance removed from h1
>>> >>>
>>> >>>
>>> >>> I don't believe the issue is with gluster itself as the volume
>>> remains
>>> >>> accessible on all hosts during this time albeit with a missing server
>>> >>> (gluster volume status) as each gluster server is rebooted.
>>> >>>
>>> >>> Gluster was upgraded as part of the process, no issues were seen
>>> here.
>>> >>>
>>> >>>
>>> >>> I have been able to duplicate the issue without the upgrade by
>>> following
>>> >>> the same sort of timeline.
>>> >>>
>>> >>>
>>> >>> ________________________________
>>> >>> From: Sandro Bonazzola <sbonazzo at redhat.com>
>>> >>> Sent: Monday, 11 April 2016 7:11 PM
>>> >>> To: Richard Neuboeck; Simone Tiraboschi; Roy Golan; Martin Sivak;
>>> Sahina
>>> >>> Bose
>>> >>> Cc: Bond, Darryl; users
>>> >>> Subject: Re: [ovirt-users] Hosted engine on gluster problem
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Mon, Apr 11, 2016 at 9:37 AM, Richard Neuboeck
>>> >>> <hawk at tbi.univie.ac.at<mailto:hawk at tbi.univie.ac.at>> wrote:
>>> >>> Hi Darryl,
>>> >>>
>>> >>> I'm still experimenting with my oVirt installation so I tried to
>>> >>> recreate the problems you've described.
>>> >>>
>>> >>> My setup has three HA hosts for virtualization and three machines
>>> >>> for the gluster replica 3 setup.
>>> >>>
>>> >>> I manually migrated the Engine from the initial install host (one)
>>> >>> to host three. Then shut down host one manually and interrupted the
>>> >>> fencing mechanisms so the host stayed down. This didn't bother the
>>> >>> Engine VM at all.
>>> >>>
>>> >>> Did you move the host one to maintenance before shutting down?
>>> >>> Or is this a crash recovery test?
>>> >>>
>>> >>>
>>> >>>
>>> >>> To make things a bit more challenging I then shut down host three
>>> >>> while running the Engine VM. Of course the Engine was down for some
>>> >>> time until host two detected the problem. It started the Engine VM
>>> >>> and everything seems to be running quite well without the initial
>>> >>> install host.
>>> >>>
>>> >>> Thanks for the feedback!
>>> >>>
>>> >>>
>>> >>>
>>> >>> My only problem is that the HA agent on host two and three refuse to
>>> >>> start after a reboot due to the fact that the configuration of the
>>> >>> hosted engine is missing. I wrote another mail to
>>> >>> users at ovirt.org<mailto:users at ovirt.org>
>>> >>> about that.
>>> >>>
>>> >>> This is weird. Martin,  Simone can you please investigate on this?
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> Cheers
>>> >>> Richard
>>> >>>
>>> >>> On 04/08/2016 01:38 AM, Bond, Darryl wrote:
>>> >>> > There seems to be a pretty severe bug with using hosted engine on
>>> >>> > gluster.
>>> >>> >
>>> >>> > If the host that was used as the initial hosted-engine --deploy
>>> host
>>> >>> > goes away, the engine VM wil crash and cannot be restarted until
>>> the host
>>> >>> > comes back.
>>> >>>
>>> >>> is this an Hyperconverged setup?
>>> >>>
>>> >>>
>>> >>> >
>>> >>> > This is regardless of which host the engine was currently running.
>>> >>> >
>>> >>> >
>>> >>> > The issue seems to be buried in the bowels of VDSM and is not an
>>> issue
>>> >>> > with gluster itself.
>>> >>>
>>> >>> Sahina, can you please investigate on this?
>>> >>>
>>> >>>
>>> >>> >
>>> >>> > The gluster filesystem is still accessable from the host that was
>>> >>> > running the engine. The issue has been submitted to bugzilla but
>>> the fix is
>>> >>> > some way off (4.1).
>>> >>> >
>>> >>> >
>>> >>> > Can my hosted engine be converted to use NFS (using the gluster NFS
>>> >>> > server on the same filesystem) without rebuilding my hosted engine
>>> (ie
>>> >>> > change domainType=glusterfs to domainType=nfs)?
>>> >>>
>>> >>> >
>>> >>> > What effect would that have on the hosted-engine storage domain
>>> inside
>>> >>> > oVirt, ie would the same filesystem be mounted twice or would it
>>> just break.
>>> >>> >
>>> >>> >
>>> >>> > Will this actually fix the problem, does it have the same issue
>>> when
>>> >>> > the hosted engine is on NFS?
>>> >>> >
>>> >>> >
>>> >>> > Darryl
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> > ________________________________
>>> >>> >
>>> >>> > The contents of this electronic message and any attachments are
>>> >>> > intended only for the addressee and may contain legally privileged,
>>> >>> > personal, sensitive or confidential information. If you are not
>>> the intended
>>> >>> > addressee, and have received this email, any transmission,
>>> distribution,
>>> >>> > downloading, printing or photocopying of the contents of this
>>> message or
>>> >>> > attachments is strictly prohibited. Any legal privilege or
>>> confidentiality
>>> >>> > attached to this message and attachments is not waived, lost or
>>> destroyed by
>>> >>> > reason of delivery to any person other than intended addressee. If
>>> you have
>>> >>> > received this message and are not the intended addressee you
>>> should notify
>>> >>> > the sender by return email and destroy all copies of the message
>>> and any
>>> >>> > attachments. Unless expressly attributed, the views expressed in
>>> this email
>>> >>> > do not necessarily represent the views of the company.
>>> >>> > _______________________________________________
>>> >>> > Users mailing list
>>> >>> > Users at ovirt.org<mailto:Users at ovirt.org>
>>> >>> > http://lists.ovirt.org/mailman/listinfo/users
>>> >>> >
>>> >>>
>>> >>>
>>> >>> --
>>> >>> /dev/null
>>> >>>
>>> >>>
>>> >>> _______________________________________________
>>> >>> Users mailing list
>>> >>> Users at ovirt.org<mailto:Users at ovirt.org>
>>> >>> http://lists.ovirt.org/mailman/listinfo/users
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Sandro Bonazzola
>>> >>> Better technology. Faster innovation. Powered by community
>>> collaboration.
>>> >>> See how it works at redhat.com<http://redhat.com>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Sandro Bonazzola
>>> >> Better technology. Faster innovation. Powered by community
>>> collaboration.
>>> >> See how it works at redhat.com
>>> >> _______________________________________________
>>> >> Users mailing list
>>> >> Users at ovirt.org
>>> >> http://lists.ovirt.org/mailman/listinfo/users
>>> >
>>> >
>>> > _______________________________________________
>>> > Users mailing list
>>> > Users at ovirt.org
>>> > http://lists.ovirt.org/mailman/listinfo/users
>>> >
>>>
>>
>>
>>
>> --
>> Sandro Bonazzola
>> Better technology. Faster innovation. Powered by community collaboration.
>> See how it works at redhat.com
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160414/a4ae4545/attachment-0001.html>