Re: [ovirt-users] Hosted engine on gluster problem

Nir, here is the problem: https://bugzilla.redhat.com/show_bug.cgi?id=1298693 When you do a hosted-engine --deploy and pick "glusterfs" you don't have a way to define the mount options, therefore, the use of the "backupvol-server", however when you create a storage domain from the UI you can, like the attached screen shot. In the hosted-engine --deploy, I would expect a flow which includes not only the "gluster" entrypoint, but also the gluster mount options which is missing today. This option would be optional, but would remove the single point of failure described on the Bug 1298693. for example: Existing entry point on the "hosted-engine --deploy" flow gluster1.xyz.com:/engine

Missing option on the "hosted-engine --deploy" flow : backupvolfile-server=gluster2.xyz.com,fetch-attempts=3,log-level=WARNING,log-file=/var/log/glusterfs/gluster_engine_domain.log Sandro, it seems to me a simple solution which can be easily fixed. What do you think? Regards -Luiz 2016-04-13 4:15 GMT-03:00 Sandro Bonazzola <sbonazzo(a)redhat.com&gt;: > > > > On Tue, Apr 12, 2016 at 6:47 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >> >> On Tue, Apr 12, 2016 at 3:05 PM, Luiz Claudio Prazeres Goncalves >> <luizcpg(a)gmail.com&gt; wrote: >> > Hi Sandro, I've been using gluster with 3 external hosts for a while >> > and >> > things are working pretty well, however this single point of failure >> > looks >> > like a simple feature to implement,but critical to anyone who wants to >> > use >> > gluster on production . This is not hyperconvergency which has other >> > issues/implications. So , why not have this feature out on 3.6 branch? >> > It >> > looks like just let vdsm use the 'backupvol-server' option when >> > mounting the >> > engine domain and make the property tests. >> >> Can you explain what is the problem, and what is the suggested solution? >> >> Engine and vdsm already support the backupvol-server option - you can >> define this option in the storage domain options when you create a >> gluster >> storage domain. With this option vdsm should be able to connect to >> gluster >> storage domain even if a brick is down. >> >> If you don't have this option in engine , you probably cannot add it with >> hosted >> engine setup, since for editing it you must put the storage domain in >> maintenance >> and if you do this the engine vm will be killed :-) This is is one of >> the issues with >> engine managing the storage domain it runs on. >> >> I think the best way to avoid this issue, is to add a DNS entry >> providing the addresses >> of all the gluster bricks, and use this address for the gluster >> storage domain. This way >> the glusterfs mount helper can mount the domain even if one of the >> gluster bricks >> are down. >> >> Again, we will need some magic from the hosted engine developers to >> modify the >> address of the hosted engine gluster domain on existing system. > > > Magic won't happen without a bz :-) please open one describing what's > requested. > > >> >> >> Nir >> >> > >> > Could you add this feature to the next release of 3.6 branch? >> > >> > Thanks >> > Luiz >> > >> > Em ter, 12 de abr de 2016 05:03, Sandro Bonazzola <sbonazzo(a)redhat.com&gt; >> > escreveu: >> >> >> >> On Mon, Apr 11, 2016 at 11:44 PM, Bond, Darryl <dbond(a)nrggos.com.au&gt; >> >> wrote: >> >>> >> >>> My setup is hyperconverged. I have placed my test results in >> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1298693 >> >>> >> >> >> >> Ok, so you're aware about the limitation of the single point of >> >> failure. >> >> If you drop the host referenced in hosted engine configuration for the >> >> initial setup it won't be able to connect to shared storage even if >> >> the >> >> other hosts in the cluster are up since the entry point is down. >> >> Note that hyperconverged deployment is not supported in 3.6. >> >> >> >> >> >>> >> >>> >> >>> Short description of setup: >> >>> >> >>> 3 hosts with 2 disks each set up with gluster replica 3 across the 6 >> >>> disks volume name hosted-engine. >> >>> >> >>> Hostname hosted-storage configured in /etc//hosts to point to the >> >>> host1. >> >>> >> >>> Installed hosted engine on host1 with the hosted engine storage path >> >>> = >> >>> hosted-storage:/hosted-engine >> >>> >> >>> Install first engine on h1 successful. Hosts h2 and h3 added to the >> >>> hosted engine. All works fine. >> >>> >> >>> Additional storage and non-hosted engine hosts added etc. >> >>> >> >>> Additional VMs added to hosted-engine storage (oVirt Reports VM and >> >>> Cinder VM). Additional VM's are hosted by other storage - cinder and >> >>> NFS. >> >>> >> >>> The system is in production. >> >>> >> >>> >> >>> Engine can be migrated around with the web interface. >> >>> >> >>> >> >>> - 3.6.4 upgrade released, follow the upgrade guide, engine is >> >>> upgraded >> >>> first , new Centos kernel requires host reboot. >> >>> >> >>> - Engine placed on h2 - h3 into maintenance (local) upgrade and >> >>> Reboot >> >>> h3 - No issues - Local maintenance removed from h3. >> >>> >> >>> - Engine placed on h3 - h2 into maintenance (local) upgrade and >> >>> Reboot >> >>> h2 - No issues - Local maintenance removed from h2. >> >>> >> >>> - Engine placed on h3 -h1 into mainteance (local) upgrade and reboot >> >>> h1 - >> >>> engine crashes and does not start elsewhere, VM(cinder) on h3 on >> >>> same >> >>> gluster volume pauses. >> >>> >> >>> - Host 1 takes about 5 minutes to reboot (Enterprise box with all >> >>> it's >> >>> normal BIOS probing) >> >>> >> >>> - Engine starts after h1 comes back and stabilises >> >>> >> >>> - VM(cinder) unpauses itself, VM(reports) continued fine the whole >> >>> time. >> >>> I can do no diagnosis on the 2 VMs as the engine is not available. >> >>> >> >>> - Local maintenance removed from h1 >> >>> >> >>> >> >>> I don't believe the issue is with gluster itself as the volume >> >>> remains >> >>> accessible on all hosts during this time albeit with a missing server >> >>> (gluster volume status) as each gluster server is rebooted. >> >>> >> >>> Gluster was upgraded as part of the process, no issues were seen >> >>> here. >> >>> >> >>> >> >>> I have been able to duplicate the issue without the upgrade by >> >>> following >> >>> the same sort of timeline. >> >>> >> >>> >> >>> ________________________________ >> >>> From: Sandro Bonazzola <sbonazzo(a)redhat.com&gt; >> >>> Sent: Monday, 11 April 2016 7:11 PM >> >>> To: Richard Neuboeck; Simone Tiraboschi; Roy Golan; Martin Sivak; >> >>> Sahina >> >>> Bose >> >>> Cc: Bond, Darryl; users >> >>> Subject: Re: [ovirt-users] Hosted engine on gluster problem >> >>> >> >>> >> >>> >> >>> On Mon, Apr 11, 2016 at 9:37 AM, Richard Neuboeck >> >>> <hawk@tbi.univie.ac.at<mailto:hawk@tbi.univie.ac.at>> wrote: >> >>> Hi Darryl, >> >>> >> >>> I'm still experimenting with my oVirt installation so I tried to >> >>> recreate the problems you've described. >> >>> >> >>> My setup has three HA hosts for virtualization and three machines >> >>> for the gluster replica 3 setup. >> >>> >> >>> I manually migrated the Engine from the initial install host (one) >> >>> to host three. Then shut down host one manually and interrupted the >> >>> fencing mechanisms so the host stayed down. This didn't bother the >> >>> Engine VM at all. >> >>> >> >>> Did you move the host one to maintenance before shutting down? >> >>> Or is this a crash recovery test? >> >>> >> >>> >> >>> >> >>> To make things a bit more challenging I then shut down host three >> >>> while running the Engine VM. Of course the Engine was down for some >> >>> time until host two detected the problem. It started the Engine VM >> >>> and everything seems to be running quite well without the initial >> >>> install host. >> >>> >> >>> Thanks for the feedback! >> >>> >> >>> >> >>> >> >>> My only problem is that the HA agent on host two and three refuse to >> >>> start after a reboot due to the fact that the configuration of the >> >>> hosted engine is missing. I wrote another mail to >> >>> users@ovirt.org<mailto:users@ovirt.org> >> >>> about that. >> >>> >> >>> This is weird. Martin, Simone can you please investigate on this? >> >>> >> >>> >> >>> >> >>> >> >>> Cheers >> >>> Richard >> >>> >> >>> On 04/08/2016 01:38 AM, Bond, Darryl wrote: >> >>> > There seems to be a pretty severe bug with using hosted engine on >> >>> > gluster. >> >>> > >> >>> > If the host that was used as the initial hosted-engine --deploy >> >>> > host >> >>> > goes away, the engine VM wil crash and cannot be restarted until >> >>> > the host >> >>> > comes back. >> >>> >> >>> is this an Hyperconverged setup? >> >>> >> >>> >> >>> > >> >>> > This is regardless of which host the engine was currently running. >> >>> > >> >>> > >> >>> > The issue seems to be buried in the bowels of VDSM and is not an >> >>> > issue >> >>> > with gluster itself. >> >>> >> >>> Sahina, can you please investigate on this? >> >>> >> >>> >> >>> > >> >>> > The gluster filesystem is still accessable from the host that was >> >>> > running the engine. The issue has been submitted to bugzilla but >> >>> > the fix is >> >>> > some way off (4.1). >> >>> > >> >>> > >> >>> > Can my hosted engine be converted to use NFS (using the gluster NFS >> >>> > server on the same filesystem) without rebuilding my hosted engine >> >>> > (ie >> >>> > change domainType=glusterfs to domainType=nfs)? >> >>> >> >>> > >> >>> > What effect would that have on the hosted-engine storage domain >> >>> > inside >> >>> > oVirt, ie would the same filesystem be mounted twice or would it >> >>> > just break. >> >>> > >> >>> > >> >>> > Will this actually fix the problem, does it have the same issue >> >>> > when >> >>> > the hosted engine is on NFS? >> >>> > >> >>> > >> >>> > Darryl >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > ________________________________ >> >>> > >> >>> > The contents of this electronic message and any attachments are >> >>> > intended only for the addressee and may contain legally privileged, >> >>> > personal, sensitive or confidential information. If you are not the >> >>> > intended >> >>> > addressee, and have received this email, any transmission, >> >>> > distribution, >> >>> > downloading, printing or photocopying of the contents of this >> >>> > message or >> >>> > attachments is strictly prohibited. Any legal privilege or >> >>> > confidentiality >> >>> > attached to this message and attachments is not waived, lost or >> >>> > destroyed by >> >>> > reason of delivery to any person other than intended addressee. If >> >>> > you have >> >>> > received this message and are not the intended addressee you should >> >>> > notify >> >>> > the sender by return email and destroy all copies of the message >> >>> > and any >> >>> > attachments. Unless expressly attributed, the views expressed in >> >>> > this email >> >>> > do not necessarily represent the views of the company. >> >>> > _______________________________________________ >> >>> > Users mailing list >> >>> > Users@ovirt.org<mailto:Users@ovirt.org> >> >>> > http://lists.ovirt.org/mailman/listinfo/users >> >>> > >> >>> >> >>> >> >>> -- >> >>> /dev/null >> >>> >> >>> >> >>> _______________________________________________ >> >>> Users mailing list >> >>> Users@ovirt.org<mailto:Users@ovirt.org> >> >>> http://lists.ovirt.org/mailman/listinfo/users >> >>> >> >>> >> >>> >> >>> >> >>> -- >> >>> Sandro Bonazzola >> >>> Better technology. Faster innovation. Powered by community >> >>> collaboration. >> >>> See how it works at redhat.com<http://redhat.com> >> >> >> >> >> >> >> >> >> >> -- >> >> Sandro Bonazzola >> >> Better technology. Faster innovation. Powered by community >> >> collaboration. >> >> See how it works at redhat.com >> >> _______________________________________________ >> >> Users mailing list >> >> Users(a)ovirt.org >> >> http://lists.ovirt.org/mailman/listinfo/users >> > >> > >> > _______________________________________________ >> > Users mailing list >> > Users(a)ovirt.org >> > http://lists.ovirt.org/mailman/listinfo/users >> > > > > > > -- > Sandro Bonazzola > Better technology. Faster innovation. Powered by community collaboration. > See how it works at redhat.com