Re: [ovirt-users] Ovirt/Gluster

On 08/21/2015 04:32 PM, Sander Hoentjen wrote: > > > On 08/21/2015 11:30 AM, Ravishankar N wrote: >> >> >> On 08/21/2015 01:21 PM, Sander Hoentjen wrote: >>> >>> >>> On 08/21/2015 09:28 AM, Ravishankar N wrote: >>>> >>>> >>>> On 08/20/2015 02:14 PM, Sander Hoentjen wrote: >>>>> >>>>> >>>>> On 08/19/2015 09:04 AM, Ravishankar N wrote: >>>>>> >>>>>> >>>>>> On 08/18/2015 04:22 PM, Ramesh Nachimuthu wrote: >>>>>>> + Ravi from gluster. >>>>>>> >>>>>>> Regards, >>>>>>> Ramesh >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>> From: "Sander Hoentjen" <sander(a)hoentjen.eu&gt; >>>>>>> To: users(a)ovirt.org >>>>>>> Sent: Tuesday, August 18, 2015 3:30:35 PM >>>>>>> Subject: [ovirt-users] Ovirt/Gluster >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> We are looking for some easy to manage self contained VM >>>>>>> hosting. Ovirt >>>>>>> with GlusterFS seems to fit that bill perfectly. I installed it >>>>>>> and then >>>>>>> starting kicking the tires. First results looked promising, but >>>>>>> now I >>>>>>> can get a VM to pause indefinitely fairly easy: >>>>>>> >>>>>>> My setup is 3 hosts that are in a Virt and Gluster cluster. >>>>>>> Gluster is >>>>>>> setup as replica-3. The gluster export is used as the storage >>>>>>> domain for >>>>>>> the VM's. >>>>>> >>>>>> Hi, >>>>>> >>>>>> What version of gluster and ovirt are you using? >>>>> glusterfs-3.7.3-1.el7.x86_64 >>>>> vdsm-4.16.20-0.el7.centos.x86_64 >>>>> ovirt-engine-3.5.3.1-1.el7.centos.noarch >>>>>> >>>>>>> >>>>>>> Now when I start the VM all is good, performance is good enough >>>>>>> so we >>>>>>> are happy. I then start bonnie++ to generate some load. I have >>>>>>> a VM >>>>>>> running on host 1, host 2 is SPM and all 3 VM's are seeing some >>>>>>> network >>>>>>> traffic courtesy of gluster. >>>>>>> >>>>>>> Now, for fun, suddenly the network on host3 goes bad (iptables >>>>>>> -I OUTPUT >>>>>>> -m statistic --mode random --probability 0.75 -j REJECT). >>>>>>> Some time later I see the guest has a small "hickup", I'm >>>>>>> guessing that >>>>>>> is when gluster decides host 3 is not allowed to play anymore. >>>>>>> No big >>>>>>> deal anyway. >>>>>>> After a while 25% of packages just isn't good enough for Ovirt >>>>>>> anymore, >>>>>>> so the host will be fenced. >>>>>> >>>>>> I'm not sure what fencing means w.r.t ovirt and what it actually >>>>>> fences. As far is gluster is concerned, since only one node is >>>>>> blocked, the VM image should still be accessible by the VM >>>>>> running on host1. >>>>> Fencing means (at least in this case) that the IPMI of the server >>>>> does a power reset. >>>>>>> After a reboot *sometimes* the VM will be >>>>>>> paused, and even after the gluster self-heal is complete it can >>>>>>> not be >>>>>>> unpaused, has to be restarted. >>>>>> >>>>>> Could you provide the gluster mount (fuse?) logs and the brick >>>>>> logs of all 3 nodes when the VM is paused? That should give us >>>>>> some clue. >>>>>> >>>>> Logs are attached. Problem was at around 8:15 - 8:20 UTC >>>>> This time however the vm stopped even without a reboot of hyp03 >>>> >>>> >>>> The mount logs (rhev-data-center-mnt-glusterSD*) are indicating >>>> frequent disconnects to the bricks with >>>> 'clnt_ping_timer_expired', 'Client-quorum is not met' and >>>> 'Read-only file system' messages. >>>> client-quorum is enabled by default for replica 3 volumes. So if >>>> the mount cannot connect to 2 bricks at least, quorum is lost and >>>> the gluster volume becomes read-only. That seems to be the reason >>>> why the VMs are pausing. >>>> I'm not sure if the frequent disconnects are due a flaky network >>>> or the bricks not responding to the mount's ping timer due to it's >>>> epoll threads busy with I/O (unlikely). Can you also share the >>>> output of `gluster volume info <volname>` ? >>> The frequent disconnects are probably because I intentionally broke >>> the network on hyp03 (dropped 75% of outgoing packets). In my >>> opinion this should not affect the VM an hyp02. Am I wrong to think >>> that? >> >> >> For client-quorum, If a client (mount) cannot connect to the number >> of bricks to achieve quorum, the client becomes read-only. So if the >> client on hyp02 can see itself and 01, it shouldn't be affected. > But it was, and I only "broke" hyp03. Beats me then. I see "[2015-08-18 15:15:27.922998] W [MSGID: 108001] [afr-common.c:4043:afr_notify] 0-VMS-replicate-0: Client-quorum is not met" on hyp02's mount log but the time stamp is earlier than when you say you observed the hang (2015-08-20, around 8:15 - 8:20 UTC?). (they do occur in that time on hyp03 though).

>> >>> >>> [root@hyp01 ~]# gluster volume info VMS >>> >>> Volume Name: VMS >>> Type: Replicate >>> Volume ID: 9e6657e7-8520-4720-ba9d-78b14a86c8ca >>> Status: Started >>> Number of Bricks: 1 x 3 = 3 >>> Transport-type: tcp >>> Bricks: >>> Brick1: 10.99.50.20:/brick/VMS >>> Brick2: 10.99.50.21:/brick/VMS >>> Brick3: 10.99.50.22:/brick/VMS >>> Options Reconfigured: >>> performance.readdir-ahead: on >>> nfs.disable: on >>> user.cifs: disable >>> auth.allow: * >>> performance.quick-read: off >>> performance.read-ahead: off >>> performance.io-cache: off >>> performance.stat-prefetch: off >>> cluster.eager-lock: enable >>> network.remote-dio: enable >>> cluster.quorum-type: auto >>> cluster.server-quorum-type: server >> >> I see that you have enabled server-quorum too. Since you blocked >> hyp03, the if the glusterd on that node cannot see the other 2 >> nodes due to iptable rules, it would kill all brick processes. See >> the "7 How To Test " section in >> http://www.gluster.org/community/documentation/index.php/Features/Server-... >> to get a better idea of server-quorum. >> > Yes but it should only kill the bricks on hyp03, right? So then why > does the VM on hyp02 die? I don't like the fact that a problem on any > one of the hosts can bring down any VM on any host. > Right. Well from a gluster point of view, if you don't want quorum enforcement you can turn it off possibly at the risk of files ending up in split-brain