Re: [ovirt-users] Ovirt/Gluster

On 08/21/2015 11:30 AM, Ravishankar N wrote: > > > On 08/21/2015 01:21 PM, Sander Hoentjen wrote: >> >> >> On 08/21/2015 09:28 AM, Ravishankar N wrote: >>> >>> >>> On 08/20/2015 02:14 PM, Sander Hoentjen wrote: >>>> >>>> >>>> On 08/19/2015 09:04 AM, Ravishankar N wrote: >>>>> >>>>> >>>>> On 08/18/2015 04:22 PM, Ramesh Nachimuthu wrote: >>>>>> + Ravi from gluster. >>>>>> >>>>>> Regards, >>>>>> Ramesh >>>>>> >>>>>> ----- Original Message ----- >>>>>> From: "Sander Hoentjen" <sander(a)hoentjen.eu&gt; >>>>>> To: users(a)ovirt.org >>>>>> Sent: Tuesday, August 18, 2015 3:30:35 PM >>>>>> Subject: [ovirt-users] Ovirt/Gluster >>>>>> >>>>>> Hi, >>>>>> >>>>>> We are looking for some easy to manage self contained VM >>>>>> hosting. Ovirt >>>>>> with GlusterFS seems to fit that bill perfectly. I installed it >>>>>> and then >>>>>> starting kicking the tires. First results looked promising, but >>>>>> now I >>>>>> can get a VM to pause indefinitely fairly easy: >>>>>> >>>>>> My setup is 3 hosts that are in a Virt and Gluster cluster. >>>>>> Gluster is >>>>>> setup as replica-3. The gluster export is used as the storage >>>>>> domain for >>>>>> the VM's. >>>>> >>>>> Hi, >>>>> >>>>> What version of gluster and ovirt are you using? >>>> glusterfs-3.7.3-1.el7.x86_64 >>>> vdsm-4.16.20-0.el7.centos.x86_64 >>>> ovirt-engine-3.5.3.1-1.el7.centos.noarch >>>>> >>>>>> >>>>>> Now when I start the VM all is good, performance is good enough >>>>>> so we >>>>>> are happy. I then start bonnie++ to generate some load. I have a VM >>>>>> running on host 1, host 2 is SPM and all 3 VM's are seeing some >>>>>> network >>>>>> traffic courtesy of gluster. >>>>>> >>>>>> Now, for fun, suddenly the network on host3 goes bad (iptables >>>>>> -I OUTPUT >>>>>> -m statistic --mode random --probability 0.75 -j REJECT). >>>>>> Some time later I see the guest has a small "hickup", I'm >>>>>> guessing that >>>>>> is when gluster decides host 3 is not allowed to play anymore. >>>>>> No big >>>>>> deal anyway. >>>>>> After a while 25% of packages just isn't good enough for Ovirt >>>>>> anymore, >>>>>> so the host will be fenced. >>>>> >>>>> I'm not sure what fencing means w.r.t ovirt and what it actually >>>>> fences. As far is gluster is concerned, since only one node is >>>>> blocked, the VM image should still be accessible by the VM >>>>> running on host1. >>>> Fencing means (at least in this case) that the IPMI of the server >>>> does a power reset. >>>>>> After a reboot *sometimes* the VM will be >>>>>> paused, and even after the gluster self-heal is complete it can >>>>>> not be >>>>>> unpaused, has to be restarted. >>>>> >>>>> Could you provide the gluster mount (fuse?) logs and the brick >>>>> logs of all 3 nodes when the VM is paused? That should give us >>>>> some clue. >>>>> >>>> Logs are attached. Problem was at around 8:15 - 8:20 UTC >>>> This time however the vm stopped even without a reboot of hyp03 >>> >>> >>> The mount logs (rhev-data-center-mnt-glusterSD*) are indicating >>> frequent disconnects to the bricks with 'clnt_ping_timer_expired', >>> 'Client-quorum is not met' and 'Read-only file system' messages. >>> client-quorum is enabled by default for replica 3 volumes. So if >>> the mount cannot connect to 2 bricks at least, quorum is lost and >>> the gluster volume becomes read-only. That seems to be the reason >>> why the VMs are pausing. >>> I'm not sure if the frequent disconnects are due a flaky network or >>> the bricks not responding to the mount's ping timer due to it's >>> epoll threads busy with I/O (unlikely). Can you also share the >>> output of `gluster volume info <volname>` ? >> The frequent disconnects are probably because I intentionally broke >> the network on hyp03 (dropped 75% of outgoing packets). In my >> opinion this should not affect the VM an hyp02. Am I wrong to think >> that? > > > For client-quorum, If a client (mount) cannot connect to the number > of bricks to achieve quorum, the client becomes read-only. So if the > client on hyp02 can see itself and 01, it shouldn't be affected. But it was, and I only "broke" hyp03.

> >> >> [root@hyp01 ~]# gluster volume info VMS >> >> Volume Name: VMS >> Type: Replicate >> Volume ID: 9e6657e7-8520-4720-ba9d-78b14a86c8ca >> Status: Started >> Number of Bricks: 1 x 3 = 3 >> Transport-type: tcp >> Bricks: >> Brick1: 10.99.50.20:/brick/VMS >> Brick2: 10.99.50.21:/brick/VMS >> Brick3: 10.99.50.22:/brick/VMS >> Options Reconfigured: >> performance.readdir-ahead: on >> nfs.disable: on >> user.cifs: disable >> auth.allow: * >> performance.quick-read: off >> performance.read-ahead: off >> performance.io-cache: off >> performance.stat-prefetch: off >> cluster.eager-lock: enable >> network.remote-dio: enable >> cluster.quorum-type: auto >> cluster.server-quorum-type: server > > I see that you have enabled server-quorum too. Since you blocked > hyp03, the if the glusterd on that node cannot see the other 2 nodes > due to iptable rules, it would kill all brick processes. See the "7 > How To Test " section in > http://www.gluster.org/community/documentation/index.php/Features/Server-... > to get a better idea of server-quorum. > Yes but it should only kill the bricks on hyp03, right? So then why does the VM on hyp02 die? I don't like the fact that a problem on any one of the hosts can bring down any VM on any host.