Re: [ovirt-users] Hosted engine on gluster problem

On Fri, Aug 26, 2016 at 8:54 AM, Sandro Bonazzola <sbonazzo(a)redhat.com&gt; wrote: > > > On Tue, Aug 23, 2016 at 8:44 PM, David Gossage < > dgossage(a)carouselchecks.com&gt; wrote: > >> >> On Fri, Apr 15, 2016 at 8:00 AM, Luiz Claudio Prazeres Goncalves < >> luizcpg(a)gmail.com&gt; wrote: >> >>> I'm not planning to move to ovirt 4 until it gets stable, so would be >>> great to backport to 3.6 or ,ideally, gets developed on the next release of >>> 3.6 branch. Considering the urgency (its a single point of failure) x >>> complexity wouldn't be hard to make the proposed fix. >>> >>> >> Bumping old email sorry. Looks like https://bugzilla.redhat.com/sh >> ow_bug.cgi?id=1298693 was finished against 3.6.7 according to that RFE. >> >> So does that mean if I add appropriate lines to my >> /etc/ovirt-hosted-engine/hosted-engine.conf the next time I restart >> engine and agent/brokers to mount that storage point it will utilize the >> backupvol-server features? >> >> If so are appropriate settings outlined in docs somewhere? >> >> Running ovirt 3.6.7 and gluster 3.8.2 on centos 7 nodes. >> > > Adding Simone > First step, you have to edit /etc/ovirt-hosted-engine/hosted-engine.conf on all your hosted-engine hosts to ensure that the storage field always point to the same entry point (host01 for instance) Then on each host you can add something like: mnt_options=backupvolfile-server=host02.yourdomain.com:host03. <http://ost03.ovirt.forest.go.th/>yourdomain.com,fetch- attempts=2,log-level=WARNING,log-file=/var/log/engine_domain.log Then check the representation of your storage connection in the table storage_server_connections of the engine DB and make sure that connection refers to the entry point you used in hosted-engine.conf on all your hosts, you have lastly to set the value of mount_options also here. Please tune also the value of network.ping-timeout for your glusterFS volume to avoid this: https://bugzilla.redhat.com/show_bug.cgi?id=1319657#c17 You can find other information here: https://www.ovirt.org/develop/release-management/features/ engine/self-hosted-engine-gluster-support/

> > >> >> >> I'm using today a production environment on top of gluster replica 3 and >>> this is the only SPF I have. >>> >>> Thanks >>> Luiz >>> >>> Em sex, 15 de abr de 2016 03:05, Sandro Bonazzola <sbonazzo(a)redhat.com&gt; >>> escreveu: >>> >>>> On Thu, Apr 14, 2016 at 7:35 PM, Nir Soffer <nsoffer(a)redhat.com&gt; >>>> wrote: >>>> >>>>> On Wed, Apr 13, 2016 at 4:34 PM, Luiz Claudio Prazeres Goncalves >>>>> <luizcpg(a)gmail.com&gt; wrote: >>>>> > Nir, here is the problem: >>>>> > https://bugzilla.redhat.com/show_bug.cgi?id=1298693 >>>>> > >>>>> > When you do a hosted-engine --deploy and pick "glusterfs" you don't >>>>> have a >>>>> > way to define the mount options, therefore, the use of the >>>>> > "backupvol-server", however when you create a storage domain from >>>>> the UI you >>>>> > can, like the attached screen shot. >>>>> > >>>>> > >>>>> > In the hosted-engine --deploy, I would expect a flow which includes >>>>> not only >>>>> > the "gluster" entrypoint, but also the gluster mount options which >>>>> is >>>>> > missing today. This option would be optional, but would remove the >>>>> single >>>>> > point of failure described on the Bug 1298693. >>>>> > >>>>> > for example: >>>>> > >>>>> > Existing entry point on the "hosted-engine --deploy" flow >>>>> > gluster1.xyz.com:/engine >>>>> >>>>> I agree, this feature must be supported. >>>>> >>>> >>>> It will, and it's currently targeted to 4.0. >>>> >>>> >>>> >>>>> >>>>> > Missing option on the "hosted-engine --deploy" flow : >>>>> > backupvolfile-server=gluster2.xyz.com,fetch-attempts=3,log-l >>>>> evel=WARNING,log-file=/var/log/glusterfs/gluster_engine_domain.log >>>>> > >>>>> > Sandro, it seems to me a simple solution which can be easily fixed. >>>>> > >>>>> > What do you think? >>>>> > >>>>> > Regards >>>>> > -Luiz >>>>> > >>>>> > >>>>> > >>>>> > 2016-04-13 4:15 GMT-03:00 Sandro Bonazzola <sbonazzo(a)redhat.com&gt;: >>>>> >> >>>>> >> >>>>> >> >>>>> >> On Tue, Apr 12, 2016 at 6:47 PM, Nir Soffer <nsoffer(a)redhat.com&gt; >>>>> wrote: >>>>> >>> >>>>> >>> On Tue, Apr 12, 2016 at 3:05 PM, Luiz Claudio Prazeres Goncalves >>>>> >>> <luizcpg(a)gmail.com&gt; wrote: >>>>> >>> > Hi Sandro, I've been using gluster with 3 external hosts for a >>>>> while >>>>> >>> > and >>>>> >>> > things are working pretty well, however this single point of >>>>> failure >>>>> >>> > looks >>>>> >>> > like a simple feature to implement,but critical to anyone who >>>>> wants to >>>>> >>> > use >>>>> >>> > gluster on production . This is not hyperconvergency which has >>>>> other >>>>> >>> > issues/implications. So , why not have this feature out on 3.6 >>>>> branch? >>>>> >>> > It >>>>> >>> > looks like just let vdsm use the 'backupvol-server' option when >>>>> >>> > mounting the >>>>> >>> > engine domain and make the property tests. >>>>> >>> >>>>> >>> Can you explain what is the problem, and what is the suggested >>>>> solution? >>>>> >>> >>>>> >>> Engine and vdsm already support the backupvol-server option - you >>>>> can >>>>> >>> define this option in the storage domain options when you create a >>>>> >>> gluster >>>>> >>> storage domain. With this option vdsm should be able to connect to >>>>> >>> gluster >>>>> >>> storage domain even if a brick is down. >>>>> >>> >>>>> >>> If you don't have this option in engine , you probably cannot add >>>>> it with >>>>> >>> hosted >>>>> >>> engine setup, since for editing it you must put the storage >>>>> domain in >>>>> >>> maintenance >>>>> >>> and if you do this the engine vm will be killed :-) This is is >>>>> one of >>>>> >>> the issues with >>>>> >>> engine managing the storage domain it runs on. >>>>> >>> >>>>> >>> I think the best way to avoid this issue, is to add a DNS entry >>>>> >>> providing the addresses >>>>> >>> of all the gluster bricks, and use this address for the gluster >>>>> >>> storage domain. This way >>>>> >>> the glusterfs mount helper can mount the domain even if one of the >>>>> >>> gluster bricks >>>>> >>> are down. >>>>> >>> >>>>> >>> Again, we will need some magic from the hosted engine developers >>>>> to >>>>> >>> modify the >>>>> >>> address of the hosted engine gluster domain on existing system. >>>>> >> >>>>> >> >>>>> >> Magic won't happen without a bz :-) please open one describing >>>>> what's >>>>> >> requested. >>>>> >> >>>>> >> >>>>> >>> >>>>> >>> >>>>> >>> Nir >>>>> >>> >>>>> >>> > >>>>> >>> > Could you add this feature to the next release of 3.6 branch? >>>>> >>> > >>>>> >>> > Thanks >>>>> >>> > Luiz >>>>> >>> > >>>>> >>> > Em ter, 12 de abr de 2016 05:03, Sandro Bonazzola < >>>>> sbonazzo(a)redhat.com&gt; >>>>> >>> > escreveu: >>>>> >>> >> >>>>> >>> >> On Mon, Apr 11, 2016 at 11:44 PM, Bond, Darryl < >>>>> dbond(a)nrggos.com.au&gt; >>>>> >>> >> wrote: >>>>> >>> >>> >>>>> >>> >>> My setup is hyperconverged. I have placed my test results in >>>>> >>> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1298693 >>>>> >>> >>> >>>>> >>> >> >>>>> >>> >> Ok, so you're aware about the limitation of the single point of >>>>> >>> >> failure. >>>>> >>> >> If you drop the host referenced in hosted engine configuration >>>>> for the >>>>> >>> >> initial setup it won't be able to connect to shared storage >>>>> even if >>>>> >>> >> the >>>>> >>> >> other hosts in the cluster are up since the entry point is >>>>> down. >>>>> >>> >> Note that hyperconverged deployment is not supported in 3.6. >>>>> >>> >> >>>>> >>> >> >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> Short description of setup: >>>>> >>> >>> >>>>> >>> >>> 3 hosts with 2 disks each set up with gluster replica 3 >>>>> across the 6 >>>>> >>> >>> disks volume name hosted-engine. >>>>> >>> >>> >>>>> >>> >>> Hostname hosted-storage configured in /etc//hosts to point to >>>>> the >>>>> >>> >>> host1. >>>>> >>> >>> >>>>> >>> >>> Installed hosted engine on host1 with the hosted engine >>>>> storage path >>>>> >>> >>> = >>>>> >>> >>> hosted-storage:/hosted-engine >>>>> >>> >>> >>>>> >>> >>> Install first engine on h1 successful. Hosts h2 and h3 added >>>>> to the >>>>> >>> >>> hosted engine. All works fine. >>>>> >>> >>> >>>>> >>> >>> Additional storage and non-hosted engine hosts added etc. >>>>> >>> >>> >>>>> >>> >>> Additional VMs added to hosted-engine storage (oVirt Reports >>>>> VM and >>>>> >>> >>> Cinder VM). Additional VM's are hosted by other storage - >>>>> cinder and >>>>> >>> >>> NFS. >>>>> >>> >>> >>>>> >>> >>> The system is in production. >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> Engine can be migrated around with the web interface. >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> - 3.6.4 upgrade released, follow the upgrade guide, engine is >>>>> >>> >>> upgraded >>>>> >>> >>> first , new Centos kernel requires host reboot. >>>>> >>> >>> >>>>> >>> >>> - Engine placed on h2 - h3 into maintenance (local) upgrade >>>>> and >>>>> >>> >>> Reboot >>>>> >>> >>> h3 - No issues - Local maintenance removed from h3. >>>>> >>> >>> >>>>> >>> >>> - Engine placed on h3 - h2 into maintenance (local) upgrade >>>>> and >>>>> >>> >>> Reboot >>>>> >>> >>> h2 - No issues - Local maintenance removed from h2. >>>>> >>> >>> >>>>> >>> >>> - Engine placed on h3 -h1 into mainteance (local) upgrade and >>>>> reboot >>>>> >>> >>> h1 - >>>>> >>> >>> engine crashes and does not start elsewhere, VM(cinder) on >>>>> h3 on >>>>> >>> >>> same >>>>> >>> >>> gluster volume pauses. >>>>> >>> >>> >>>>> >>> >>> - Host 1 takes about 5 minutes to reboot (Enterprise box with >>>>> all >>>>> >>> >>> it's >>>>> >>> >>> normal BIOS probing) >>>>> >>> >>> >>>>> >>> >>> - Engine starts after h1 comes back and stabilises >>>>> >>> >>> >>>>> >>> >>> - VM(cinder) unpauses itself, VM(reports) continued fine the >>>>> whole >>>>> >>> >>> time. >>>>> >>> >>> I can do no diagnosis on the 2 VMs as the engine is not >>>>> available. >>>>> >>> >>> >>>>> >>> >>> - Local maintenance removed from h1 >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> I don't believe the issue is with gluster itself as the volume >>>>> >>> >>> remains >>>>> >>> >>> accessible on all hosts during this time albeit with a >>>>> missing server >>>>> >>> >>> (gluster volume status) as each gluster server is rebooted. >>>>> >>> >>> >>>>> >>> >>> Gluster was upgraded as part of the process, no issues were >>>>> seen >>>>> >>> >>> here. >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> I have been able to duplicate the issue without the upgrade by >>>>> >>> >>> following >>>>> >>> >>> the same sort of timeline. >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> ________________________________ >>>>> >>> >>> From: Sandro Bonazzola <sbonazzo(a)redhat.com&gt; >>>>> >>> >>> Sent: Monday, 11 April 2016 7:11 PM >>>>> >>> >>> To: Richard Neuboeck; Simone Tiraboschi; Roy Golan; Martin >>>>> Sivak; >>>>> >>> >>> Sahina >>>>> >>> >>> Bose >>>>> >>> >>> Cc: Bond, Darryl; users >>>>> >>> >>> Subject: Re: [ovirt-users] Hosted engine on gluster problem >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> On Mon, Apr 11, 2016 at 9:37 AM, Richard Neuboeck >>>>> >>> >>> <hawk@tbi.univie.ac.at<mailto:hawk@tbi.univie.ac.at>> wrote: >>>>> >>> >>> Hi Darryl, >>>>> >>> >>> >>>>> >>> >>> I'm still experimenting with my oVirt installation so I tried >>>>> to >>>>> >>> >>> recreate the problems you've described. >>>>> >>> >>> >>>>> >>> >>> My setup has three HA hosts for virtualization and three >>>>> machines >>>>> >>> >>> for the gluster replica 3 setup. >>>>> >>> >>> >>>>> >>> >>> I manually migrated the Engine from the initial install host >>>>> (one) >>>>> >>> >>> to host three. Then shut down host one manually and >>>>> interrupted the >>>>> >>> >>> fencing mechanisms so the host stayed down. This didn't >>>>> bother the >>>>> >>> >>> Engine VM at all. >>>>> >>> >>> >>>>> >>> >>> Did you move the host one to maintenance before shutting down? >>>>> >>> >>> Or is this a crash recovery test? >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> To make things a bit more challenging I then shut down host >>>>> three >>>>> >>> >>> while running the Engine VM. Of course the Engine was down >>>>> for some >>>>> >>> >>> time until host two detected the problem. It started the >>>>> Engine VM >>>>> >>> >>> and everything seems to be running quite well without the >>>>> initial >>>>> >>> >>> install host. >>>>> >>> >>> >>>>> >>> >>> Thanks for the feedback! >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> My only problem is that the HA agent on host two and three >>>>> refuse to >>>>> >>> >>> start after a reboot due to the fact that the configuration >>>>> of the >>>>> >>> >>> hosted engine is missing. I wrote another mail to >>>>> >>> >>> users@ovirt.org<mailto:users@ovirt.org> >>>>> >>> >>> about that. >>>>> >>> >>> >>>>> >>> >>> This is weird. Martin, Simone can you please investigate on >>>>> this? >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> Cheers >>>>> >>> >>> Richard >>>>> >>> >>> >>>>> >>> >>> On 04/08/2016 01:38 AM, Bond, Darryl wrote: >>>>> >>> >>> > There seems to be a pretty severe bug with using hosted >>>>> engine on >>>>> >>> >>> > gluster. >>>>> >>> >>> > >>>>> >>> >>> > If the host that was used as the initial hosted-engine >>>>> --deploy >>>>> >>> >>> > host >>>>> >>> >>> > goes away, the engine VM wil crash and cannot be restarted >>>>> until >>>>> >>> >>> > the host >>>>> >>> >>> > comes back. >>>>> >>> >>> >>>>> >>> >>> is this an Hyperconverged setup? >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> > >>>>> >>> >>> > This is regardless of which host the engine was currently >>>>> running. >>>>> >>> >>> > >>>>> >>> >>> > >>>>> >>> >>> > The issue seems to be buried in the bowels of VDSM and is >>>>> not an >>>>> >>> >>> > issue >>>>> >>> >>> > with gluster itself. >>>>> >>> >>> >>>>> >>> >>> Sahina, can you please investigate on this? >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> > >>>>> >>> >>> > The gluster filesystem is still accessable from the host >>>>> that was >>>>> >>> >>> > running the engine. The issue has been submitted to >>>>> bugzilla but >>>>> >>> >>> > the fix is >>>>> >>> >>> > some way off (4.1). >>>>> >>> >>> > >>>>> >>> >>> > >>>>> >>> >>> > Can my hosted engine be converted to use NFS (using the >>>>> gluster NFS >>>>> >>> >>> > server on the same filesystem) without rebuilding my hosted >>>>> engine >>>>> >>> >>> > (ie >>>>> >>> >>> > change domainType=glusterfs to domainType=nfs)? >>>>> >>> >>> >>>>> >>> >>> > >>>>> >>> >>> > What effect would that have on the hosted-engine storage >>>>> domain >>>>> >>> >>> > inside >>>>> >>> >>> > oVirt, ie would the same filesystem be mounted twice or >>>>> would it >>>>> >>> >>> > just break. >>>>> >>> >>> > >>>>> >>> >>> > >>>>> >>> >>> > Will this actually fix the problem, does it have the same >>>>> issue >>>>> >>> >>> > when >>>>> >>> >>> > the hosted engine is on NFS? >>>>> >>> >>> > >>>>> >>> >>> > >>>>> >>> >>> > Darryl >>>>> >>> >>> > >>>>> >>> >>> > >>>>> >>> >>> > >>>>> >>> >>> > >>>>> >>> >>> > ________________________________ >>>>> >>> >>> > >>>>> >>> >>> > The contents of this electronic message and any attachments >>>>> are >>>>> >>> >>> > intended only for the addressee and may contain legally >>>>> privileged, >>>>> >>> >>> > personal, sensitive or confidential information. If you are >>>>> not the >>>>> >>> >>> > intended >>>>> >>> >>> > addressee, and have received this email, any transmission, >>>>> >>> >>> > distribution, >>>>> >>> >>> > downloading, printing or photocopying of the contents of >>>>> this >>>>> >>> >>> > message or >>>>> >>> >>> > attachments is strictly prohibited. Any legal privilege or >>>>> >>> >>> > confidentiality >>>>> >>> >>> > attached to this message and attachments is not waived, >>>>> lost or >>>>> >>> >>> > destroyed by >>>>> >>> >>> > reason of delivery to any person other than intended >>>>> addressee. If >>>>> >>> >>> > you have >>>>> >>> >>> > received this message and are not the intended addressee >>>>> you should >>>>> >>> >>> > notify >>>>> >>> >>> > the sender by return email and destroy all copies of the >>>>> message >>>>> >>> >>> > and any >>>>> >>> >>> > attachments. Unless expressly attributed, the views >>>>> expressed in >>>>> >>> >>> > this email >>>>> >>> >>> > do not necessarily represent the views of the company. >>>>> >>> >>> > _______________________________________________ >>>>> >>> >>> > Users mailing list >>>>> >>> >>> > Users@ovirt.org<mailto:Users@ovirt.org> >>>>> >>> >>> > http://lists.ovirt.org/mailman/listinfo/users >>>>> >>> >>> > >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> -- >>>>> >>> >>> /dev/null >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> _______________________________________________ >>>>> >>> >>> Users mailing list >>>>> >>> >>> Users@ovirt.org<mailto:Users@ovirt.org> >>>>> >>> >>> http://lists.ovirt.org/mailman/listinfo/users >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> >>>>> >>> >>> -- >>>>> >>> >>> Sandro Bonazzola >>>>> >>> >>> Better technology. Faster innovation. Powered by community >>>>> >>> >>> collaboration. >>>>> >>> >>> See how it works at redhat.com<http://redhat.com> >>>>> >>> >> >>>>> >>> >> >>>>> >>> >> >>>>> >>> >> >>>>> >>> >> -- >>>>> >>> >> Sandro Bonazzola >>>>> >>> >> Better technology. Faster innovation. Powered by community >>>>> >>> >> collaboration. >>>>> >>> >> See how it works at redhat.com >>>>> >>> >> _______________________________________________ >>>>> >>> >> Users mailing list >>>>> >>> >> Users(a)ovirt.org >>>>> >>> >> http://lists.ovirt.org/mailman/listinfo/users >>>>> >>> > >>>>> >>> > >>>>> >>> > _______________________________________________ >>>>> >>> > Users mailing list >>>>> >>> > Users(a)ovirt.org >>>>> >>> > http://lists.ovirt.org/mailman/listinfo/users >>>>> >>> > >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> -- >>>>> >> Sandro Bonazzola >>>>> >> Better technology. Faster innovation. Powered by community >>>>> collaboration. >>>>> >> See how it works at redhat.com >>>>> > >>>>> > >>>>> >>>> >>>> >>>> >>>> -- >>>> Sandro Bonazzola >>>> Better technology. Faster innovation. Powered by community >>>> collaboration. >>>> See how it works at redhat.com >>>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >> > > > -- > Sandro Bonazzola > Better technology. Faster innovation. Powered by community collaboration. > See how it works at redhat.com >