Re: [ovirt-users] Hosted engine on gluster problem

On Fri, Apr 15, 2016 at 8:00 AM, Luiz Claudio Prazeres Goncalves < luizcpg(a)gmail.com&gt; wrote: > I'm not planning to move to ovirt 4 until it gets stable, so would be > great to backport to 3.6 or ,ideally, gets developed on the next release of > 3.6 branch. Considering the urgency (its a single point of failure) x > complexity wouldn't be hard to make the proposed fix. > > Bumping old email sorry. Looks like https://bugzilla.redhat.com/ show_bug.cgi?id=1298693 was finished against 3.6.7 according to that RFE. So does that mean if I add appropriate lines to my /etc/ovirt-hosted-engine/hosted-engine.conf the next time I restart engine and agent/brokers to mount that storage point it will utilize the backupvol-server features? If so are appropriate settings outlined in docs somewhere? Running ovirt 3.6.7 and gluster 3.8.2 on centos 7 nodes.

I'm using today a production environment on top of gluster replica 3 and > this is the only SPF I have. > > Thanks > Luiz > > Em sex, 15 de abr de 2016 03:05, Sandro Bonazzola <sbonazzo(a)redhat.com&gt; > escreveu: > >> On Thu, Apr 14, 2016 at 7:35 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >> >>> On Wed, Apr 13, 2016 at 4:34 PM, Luiz Claudio Prazeres Goncalves >>> <luizcpg(a)gmail.com&gt; wrote: >>> > Nir, here is the problem: >>> > https://bugzilla.redhat.com/show_bug.cgi?id=1298693 >>> > >>> > When you do a hosted-engine --deploy and pick "glusterfs" you don't >>> have a >>> > way to define the mount options, therefore, the use of the >>> > "backupvol-server", however when you create a storage domain from the >>> UI you >>> > can, like the attached screen shot. >>> > >>> > >>> > In the hosted-engine --deploy, I would expect a flow which includes >>> not only >>> > the "gluster" entrypoint, but also the gluster mount options which is >>> > missing today. This option would be optional, but would remove the >>> single >>> > point of failure described on the Bug 1298693. >>> > >>> > for example: >>> > >>> > Existing entry point on the "hosted-engine --deploy" flow >>> > gluster1.xyz.com:/engine >>> >>> I agree, this feature must be supported. >>> >> >> It will, and it's currently targeted to 4.0. >> >> >> >>> >>> > Missing option on the "hosted-engine --deploy" flow : >>> > backupvolfile-server=gluster2.xyz.com,fetch-attempts=3,log-l >>> evel=WARNING,log-file=/var/log/glusterfs/gluster_engine_domain.log >>> > >>> > Sandro, it seems to me a simple solution which can be easily fixed. >>> > >>> > What do you think? >>> > >>> > Regards >>> > -Luiz >>> > >>> > >>> > >>> > 2016-04-13 4:15 GMT-03:00 Sandro Bonazzola <sbonazzo(a)redhat.com&gt;: >>> >> >>> >> >>> >> >>> >> On Tue, Apr 12, 2016 at 6:47 PM, Nir Soffer <nsoffer(a)redhat.com&gt; >>> wrote: >>> >>> >>> >>> On Tue, Apr 12, 2016 at 3:05 PM, Luiz Claudio Prazeres Goncalves >>> >>> <luizcpg(a)gmail.com&gt; wrote: >>> >>> > Hi Sandro, I've been using gluster with 3 external hosts for a >>> while >>> >>> > and >>> >>> > things are working pretty well, however this single point of >>> failure >>> >>> > looks >>> >>> > like a simple feature to implement,but critical to anyone who >>> wants to >>> >>> > use >>> >>> > gluster on production . This is not hyperconvergency which has >>> other >>> >>> > issues/implications. So , why not have this feature out on 3.6 >>> branch? >>> >>> > It >>> >>> > looks like just let vdsm use the 'backupvol-server' option when >>> >>> > mounting the >>> >>> > engine domain and make the property tests. >>> >>> >>> >>> Can you explain what is the problem, and what is the suggested >>> solution? >>> >>> >>> >>> Engine and vdsm already support the backupvol-server option - you >>> can >>> >>> define this option in the storage domain options when you create a >>> >>> gluster >>> >>> storage domain. With this option vdsm should be able to connect to >>> >>> gluster >>> >>> storage domain even if a brick is down. >>> >>> >>> >>> If you don't have this option in engine , you probably cannot add >>> it with >>> >>> hosted >>> >>> engine setup, since for editing it you must put the storage domain >>> in >>> >>> maintenance >>> >>> and if you do this the engine vm will be killed :-) This is is one >>> of >>> >>> the issues with >>> >>> engine managing the storage domain it runs on. >>> >>> >>> >>> I think the best way to avoid this issue, is to add a DNS entry >>> >>> providing the addresses >>> >>> of all the gluster bricks, and use this address for the gluster >>> >>> storage domain. This way >>> >>> the glusterfs mount helper can mount the domain even if one of the >>> >>> gluster bricks >>> >>> are down. >>> >>> >>> >>> Again, we will need some magic from the hosted engine developers to >>> >>> modify the >>> >>> address of the hosted engine gluster domain on existing system. >>> >> >>> >> >>> >> Magic won't happen without a bz :-) please open one describing what's >>> >> requested. >>> >> >>> >> >>> >>> >>> >>> >>> >>> Nir >>> >>> >>> >>> > >>> >>> > Could you add this feature to the next release of 3.6 branch? >>> >>> > >>> >>> > Thanks >>> >>> > Luiz >>> >>> > >>> >>> > Em ter, 12 de abr de 2016 05:03, Sandro Bonazzola < >>> sbonazzo(a)redhat.com&gt; >>> >>> > escreveu: >>> >>> >> >>> >>> >> On Mon, Apr 11, 2016 at 11:44 PM, Bond, Darryl < >>> dbond(a)nrggos.com.au&gt; >>> >>> >> wrote: >>> >>> >>> >>> >>> >>> My setup is hyperconverged. I have placed my test results in >>> >>> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1298693 >>> >>> >>> >>> >>> >> >>> >>> >> Ok, so you're aware about the limitation of the single point of >>> >>> >> failure. >>> >>> >> If you drop the host referenced in hosted engine configuration >>> for the >>> >>> >> initial setup it won't be able to connect to shared storage even >>> if >>> >>> >> the >>> >>> >> other hosts in the cluster are up since the entry point is down. >>> >>> >> Note that hyperconverged deployment is not supported in 3.6. >>> >>> >> >>> >>> >> >>> >>> >>> >>> >>> >>> >>> >>> >>> Short description of setup: >>> >>> >>> >>> >>> >>> 3 hosts with 2 disks each set up with gluster replica 3 across >>> the 6 >>> >>> >>> disks volume name hosted-engine. >>> >>> >>> >>> >>> >>> Hostname hosted-storage configured in /etc//hosts to point to >>> the >>> >>> >>> host1. >>> >>> >>> >>> >>> >>> Installed hosted engine on host1 with the hosted engine storage >>> path >>> >>> >>> = >>> >>> >>> hosted-storage:/hosted-engine >>> >>> >>> >>> >>> >>> Install first engine on h1 successful. Hosts h2 and h3 added to >>> the >>> >>> >>> hosted engine. All works fine. >>> >>> >>> >>> >>> >>> Additional storage and non-hosted engine hosts added etc. >>> >>> >>> >>> >>> >>> Additional VMs added to hosted-engine storage (oVirt Reports VM >>> and >>> >>> >>> Cinder VM). Additional VM's are hosted by other storage - >>> cinder and >>> >>> >>> NFS. >>> >>> >>> >>> >>> >>> The system is in production. >>> >>> >>> >>> >>> >>> >>> >>> >>> Engine can be migrated around with the web interface. >>> >>> >>> >>> >>> >>> >>> >>> >>> - 3.6.4 upgrade released, follow the upgrade guide, engine is >>> >>> >>> upgraded >>> >>> >>> first , new Centos kernel requires host reboot. >>> >>> >>> >>> >>> >>> - Engine placed on h2 - h3 into maintenance (local) upgrade and >>> >>> >>> Reboot >>> >>> >>> h3 - No issues - Local maintenance removed from h3. >>> >>> >>> >>> >>> >>> - Engine placed on h3 - h2 into maintenance (local) upgrade and >>> >>> >>> Reboot >>> >>> >>> h2 - No issues - Local maintenance removed from h2. >>> >>> >>> >>> >>> >>> - Engine placed on h3 -h1 into mainteance (local) upgrade and >>> reboot >>> >>> >>> h1 - >>> >>> >>> engine crashes and does not start elsewhere, VM(cinder) on h3 >>> on >>> >>> >>> same >>> >>> >>> gluster volume pauses. >>> >>> >>> >>> >>> >>> - Host 1 takes about 5 minutes to reboot (Enterprise box with >>> all >>> >>> >>> it's >>> >>> >>> normal BIOS probing) >>> >>> >>> >>> >>> >>> - Engine starts after h1 comes back and stabilises >>> >>> >>> >>> >>> >>> - VM(cinder) unpauses itself, VM(reports) continued fine the >>> whole >>> >>> >>> time. >>> >>> >>> I can do no diagnosis on the 2 VMs as the engine is not >>> available. >>> >>> >>> >>> >>> >>> - Local maintenance removed from h1 >>> >>> >>> >>> >>> >>> >>> >>> >>> I don't believe the issue is with gluster itself as the volume >>> >>> >>> remains >>> >>> >>> accessible on all hosts during this time albeit with a missing >>> server >>> >>> >>> (gluster volume status) as each gluster server is rebooted. >>> >>> >>> >>> >>> >>> Gluster was upgraded as part of the process, no issues were seen >>> >>> >>> here. >>> >>> >>> >>> >>> >>> >>> >>> >>> I have been able to duplicate the issue without the upgrade by >>> >>> >>> following >>> >>> >>> the same sort of timeline. >>> >>> >>> >>> >>> >>> >>> >>> >>> ________________________________ >>> >>> >>> From: Sandro Bonazzola <sbonazzo(a)redhat.com&gt; >>> >>> >>> Sent: Monday, 11 April 2016 7:11 PM >>> >>> >>> To: Richard Neuboeck; Simone Tiraboschi; Roy Golan; Martin >>> Sivak; >>> >>> >>> Sahina >>> >>> >>> Bose >>> >>> >>> Cc: Bond, Darryl; users >>> >>> >>> Subject: Re: [ovirt-users] Hosted engine on gluster problem >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Mon, Apr 11, 2016 at 9:37 AM, Richard Neuboeck >>> >>> >>> <hawk@tbi.univie.ac.at<mailto:hawk@tbi.univie.ac.at>> wrote: >>> >>> >>> Hi Darryl, >>> >>> >>> >>> >>> >>> I'm still experimenting with my oVirt installation so I tried to >>> >>> >>> recreate the problems you've described. >>> >>> >>> >>> >>> >>> My setup has three HA hosts for virtualization and three >>> machines >>> >>> >>> for the gluster replica 3 setup. >>> >>> >>> >>> >>> >>> I manually migrated the Engine from the initial install host >>> (one) >>> >>> >>> to host three. Then shut down host one manually and interrupted >>> the >>> >>> >>> fencing mechanisms so the host stayed down. This didn't bother >>> the >>> >>> >>> Engine VM at all. >>> >>> >>> >>> >>> >>> Did you move the host one to maintenance before shutting down? >>> >>> >>> Or is this a crash recovery test? >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> To make things a bit more challenging I then shut down host >>> three >>> >>> >>> while running the Engine VM. Of course the Engine was down for >>> some >>> >>> >>> time until host two detected the problem. It started the Engine >>> VM >>> >>> >>> and everything seems to be running quite well without the >>> initial >>> >>> >>> install host. >>> >>> >>> >>> >>> >>> Thanks for the feedback! >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> My only problem is that the HA agent on host two and three >>> refuse to >>> >>> >>> start after a reboot due to the fact that the configuration of >>> the >>> >>> >>> hosted engine is missing. I wrote another mail to >>> >>> >>> users@ovirt.org<mailto:users@ovirt.org> >>> >>> >>> about that. >>> >>> >>> >>> >>> >>> This is weird. Martin, Simone can you please investigate on >>> this? >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> Cheers >>> >>> >>> Richard >>> >>> >>> >>> >>> >>> On 04/08/2016 01:38 AM, Bond, Darryl wrote: >>> >>> >>> > There seems to be a pretty severe bug with using hosted >>> engine on >>> >>> >>> > gluster. >>> >>> >>> > >>> >>> >>> > If the host that was used as the initial hosted-engine >>> --deploy >>> >>> >>> > host >>> >>> >>> > goes away, the engine VM wil crash and cannot be restarted >>> until >>> >>> >>> > the host >>> >>> >>> > comes back. >>> >>> >>> >>> >>> >>> is this an Hyperconverged setup? >>> >>> >>> >>> >>> >>> >>> >>> >>> > >>> >>> >>> > This is regardless of which host the engine was currently >>> running. >>> >>> >>> > >>> >>> >>> > >>> >>> >>> > The issue seems to be buried in the bowels of VDSM and is not >>> an >>> >>> >>> > issue >>> >>> >>> > with gluster itself. >>> >>> >>> >>> >>> >>> Sahina, can you please investigate on this? >>> >>> >>> >>> >>> >>> >>> >>> >>> > >>> >>> >>> > The gluster filesystem is still accessable from the host that >>> was >>> >>> >>> > running the engine. The issue has been submitted to bugzilla >>> but >>> >>> >>> > the fix is >>> >>> >>> > some way off (4.1). >>> >>> >>> > >>> >>> >>> > >>> >>> >>> > Can my hosted engine be converted to use NFS (using the >>> gluster NFS >>> >>> >>> > server on the same filesystem) without rebuilding my hosted >>> engine >>> >>> >>> > (ie >>> >>> >>> > change domainType=glusterfs to domainType=nfs)? >>> >>> >>> >>> >>> >>> > >>> >>> >>> > What effect would that have on the hosted-engine storage >>> domain >>> >>> >>> > inside >>> >>> >>> > oVirt, ie would the same filesystem be mounted twice or would >>> it >>> >>> >>> > just break. >>> >>> >>> > >>> >>> >>> > >>> >>> >>> > Will this actually fix the problem, does it have the same >>> issue >>> >>> >>> > when >>> >>> >>> > the hosted engine is on NFS? >>> >>> >>> > >>> >>> >>> > >>> >>> >>> > Darryl >>> >>> >>> > >>> >>> >>> > >>> >>> >>> > >>> >>> >>> > >>> >>> >>> > ________________________________ >>> >>> >>> > >>> >>> >>> > The contents of this electronic message and any attachments >>> are >>> >>> >>> > intended only for the addressee and may contain legally >>> privileged, >>> >>> >>> > personal, sensitive or confidential information. If you are >>> not the >>> >>> >>> > intended >>> >>> >>> > addressee, and have received this email, any transmission, >>> >>> >>> > distribution, >>> >>> >>> > downloading, printing or photocopying of the contents of this >>> >>> >>> > message or >>> >>> >>> > attachments is strictly prohibited. Any legal privilege or >>> >>> >>> > confidentiality >>> >>> >>> > attached to this message and attachments is not waived, lost >>> or >>> >>> >>> > destroyed by >>> >>> >>> > reason of delivery to any person other than intended >>> addressee. If >>> >>> >>> > you have >>> >>> >>> > received this message and are not the intended addressee you >>> should >>> >>> >>> > notify >>> >>> >>> > the sender by return email and destroy all copies of the >>> message >>> >>> >>> > and any >>> >>> >>> > attachments. Unless expressly attributed, the views expressed >>> in >>> >>> >>> > this email >>> >>> >>> > do not necessarily represent the views of the company. >>> >>> >>> > _______________________________________________ >>> >>> >>> > Users mailing list >>> >>> >>> > Users@ovirt.org<mailto:Users@ovirt.org> >>> >>> >>> > http://lists.ovirt.org/mailman/listinfo/users >>> >>> >>> > >>> >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> >>> /dev/null >>> >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> >>> >>> Users mailing list >>> >>> >>> Users@ovirt.org<mailto:Users@ovirt.org> >>> >>> >>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> >>> Sandro Bonazzola >>> >>> >>> Better technology. Faster innovation. Powered by community >>> >>> >>> collaboration. >>> >>> >>> See how it works at redhat.com<http://redhat.com> >>> >>> >> >>> >>> >> >>> >>> >> >>> >>> >> >>> >>> >> -- >>> >>> >> Sandro Bonazzola >>> >>> >> Better technology. Faster innovation. Powered by community >>> >>> >> collaboration. >>> >>> >> See how it works at redhat.com >>> >>> >> _______________________________________________ >>> >>> >> Users mailing list >>> >>> >> Users(a)ovirt.org >>> >>> >> http://lists.ovirt.org/mailman/listinfo/users >>> >>> > >>> >>> > >>> >>> > _______________________________________________ >>> >>> > Users mailing list >>> >>> > Users(a)ovirt.org >>> >>> > http://lists.ovirt.org/mailman/listinfo/users >>> >>> > >>> >> >>> >> >>> >> >>> >> >>> >> -- >>> >> Sandro Bonazzola >>> >> Better technology. Faster innovation. Powered by community >>> collaboration. >>> >> See how it works at redhat.com >>> > >>> > >>> >> >> >> >> -- >> Sandro Bonazzola >> Better technology. Faster innovation. Powered by community collaboration. >> See how it works at redhat.com >> > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > >