[ovirt-users] Ovirt/Gluster

[Ravishankar N]

On 08/21/2015 01:21 PM, Sander Hoentjen wrote:
>
>
> On 08/21/2015 09:28 AM, Ravishankar N wrote:
>>
>>
>> On 08/20/2015 02:14 PM, Sander Hoentjen wrote:
>>>
>>>
>>> On 08/19/2015 09:04 AM, Ravishankar N wrote:
>>>>
>>>>
>>>> On 08/18/2015 04:22 PM, Ramesh Nachimuthu wrote:
>>>>> + Ravi from gluster.
>>>>>
>>>>> Regards,
>>>>> Ramesh
>>>>>
>>>>> ----- Original Message -----
>>>>> From: "Sander Hoentjen" <sander at hoentjen.eu>
>>>>> To: users at ovirt.org
>>>>> Sent: Tuesday, August 18, 2015 3:30:35 PM
>>>>> Subject: [ovirt-users] Ovirt/Gluster
>>>>>
>>>>> Hi,
>>>>>
>>>>> We are looking for some easy to manage self contained VM hosting. 
>>>>> Ovirt
>>>>> with GlusterFS seems to fit that bill perfectly. I installed it 
>>>>> and then
>>>>> starting kicking the tires. First results looked promising, but now I
>>>>> can get a VM to pause indefinitely fairly easy:
>>>>>
>>>>> My setup is 3 hosts that are in a Virt and Gluster cluster. 
>>>>> Gluster is
>>>>> setup as replica-3. The gluster export is used as the storage 
>>>>> domain for
>>>>> the VM's.
>>>>
>>>> Hi,
>>>>
>>>> What version of gluster and ovirt are you using?
>>> glusterfs-3.7.3-1.el7.x86_64
>>> vdsm-4.16.20-0.el7.centos.x86_64
>>> ovirt-engine-3.5.3.1-1.el7.centos.noarch
>>>>
>>>>>
>>>>> Now when I start the VM all is good, performance is good enough so we
>>>>> are happy. I then start bonnie++ to generate some load. I have a VM
>>>>> running on host 1, host 2 is SPM and all 3 VM's are seeing some 
>>>>> network
>>>>> traffic courtesy of gluster.
>>>>>
>>>>> Now, for fun, suddenly the network on host3 goes bad (iptables -I 
>>>>> OUTPUT
>>>>> -m statistic --mode random --probability 0.75 -j REJECT).
>>>>> Some time later I see the guest has a small "hickup", I'm guessing 
>>>>> that
>>>>> is when gluster decides host 3 is not allowed to play anymore. No big
>>>>> deal anyway.
>>>>> After a while 25% of packages just isn't good enough for Ovirt 
>>>>> anymore,
>>>>> so the host will be fenced.
>>>>
>>>> I'm not sure what fencing means w.r.t ovirt and what it actually 
>>>> fences. As far is gluster is concerned, since only one node is 
>>>> blocked, the VM image should still be accessible by the VM running 
>>>> on host1.
>>> Fencing means (at least in this case) that the IPMI of the server 
>>> does a power reset.
>>>>> After a reboot *sometimes* the VM will be
>>>>> paused, and even after the gluster self-heal is complete it can 
>>>>> not be
>>>>> unpaused, has to be restarted.
>>>>
>>>> Could you provide the gluster mount (fuse?) logs and the brick logs 
>>>> of all 3 nodes when the VM is paused? That should give us some clue.
>>>>
>>> Logs are attached. Problem was at around 8:15 - 8:20 UTC
>>> This time however the vm stopped even without a reboot of hyp03
>>
>>
>> The mount logs  (rhev-data-center-mnt-glusterSD*) are indicating 
>> frequent disconnects to the bricks  with 'clnt_ping_timer_expired', 
>> 'Client-quorum is not met' and 'Read-only file system' messages.
>> client-quorum is enabled by default for replica 3 volumes. So if the 
>> mount cannot connect to 2 bricks at least, quorum is lost and the 
>> gluster volume becomes read-only. That seems to be the reason why the 
>> VMs are pausing.
>> I'm not sure if the frequent disconnects are due a flaky network or 
>> the bricks not responding to the mount's ping timer due to it's epoll 
>> threads busy with I/O (unlikely). Can you also share the output of 
>> `gluster volume info <volname>` ?
> The frequent disconnects are probably because I intentionally broke 
> the network on hyp03 (dropped 75% of outgoing packets). In my opinion 
> this should not affect the VM an hyp02. Am I wrong to think that?


For client-quorum, If a client (mount)  cannot connect to the number of 
bricks to achieve quorum, the client becomes read-only. So if the client 
on hyp02 can see itself and 01, it shouldn't be affected.

>
> [root at hyp01 ~]# gluster volume info VMS
>
> Volume Name: VMS
> Type: Replicate
> Volume ID: 9e6657e7-8520-4720-ba9d-78b14a86c8ca
> Status: Started
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: 10.99.50.20:/brick/VMS
> Brick2: 10.99.50.21:/brick/VMS
> Brick3: 10.99.50.22:/brick/VMS
> Options Reconfigured:
> performance.readdir-ahead: on
> nfs.disable: on
> user.cifs: disable
> auth.allow: *
> performance.quick-read: off
> performance.read-ahead: off
> performance.io-cache: off
> performance.stat-prefetch: off
> cluster.eager-lock: enable
> network.remote-dio: enable
> cluster.quorum-type: auto
> cluster.server-quorum-type: server

I see that you have enabled server-quorum too. Since you blocked hyp03, 
the if the glusterd on that node cannot  see the other 2 nodes due to 
iptable rules, it would kill all brick processes. See the "7 How To Test 
" section in 
http://www.gluster.org/community/documentation/index.php/Features/Server-quorum 
to get a better idea of server-quorum.

> storage.owner-uid: 36
> storage.owner-gid: 36
>