Re: [ovirt-users] Ovirt/Gluster

On 08/21/2015 09:28 AM, Ravishankar N wrote: > > > On 08/20/2015 02:14 PM, Sander Hoentjen wrote: >> >> >> On 08/19/2015 09:04 AM, Ravishankar N wrote: >>> >>> >>> On 08/18/2015 04:22 PM, Ramesh Nachimuthu wrote: >>>> + Ravi from gluster. >>>> >>>> Regards, >>>> Ramesh >>>> >>>> ----- Original Message ----- >>>> From: "Sander Hoentjen" <sander(a)hoentjen.eu&gt; >>>> To: users(a)ovirt.org >>>> Sent: Tuesday, August 18, 2015 3:30:35 PM >>>> Subject: [ovirt-users] Ovirt/Gluster >>>> >>>> Hi, >>>> >>>> We are looking for some easy to manage self contained VM hosting. >>>> Ovirt >>>> with GlusterFS seems to fit that bill perfectly. I installed it >>>> and then >>>> starting kicking the tires. First results looked promising, but now I >>>> can get a VM to pause indefinitely fairly easy: >>>> >>>> My setup is 3 hosts that are in a Virt and Gluster cluster. >>>> Gluster is >>>> setup as replica-3. The gluster export is used as the storage >>>> domain for >>>> the VM's. >>> >>> Hi, >>> >>> What version of gluster and ovirt are you using? >> glusterfs-3.7.3-1.el7.x86_64 >> vdsm-4.16.20-0.el7.centos.x86_64 >> ovirt-engine-3.5.3.1-1.el7.centos.noarch >>> >>>> >>>> Now when I start the VM all is good, performance is good enough so we >>>> are happy. I then start bonnie++ to generate some load. I have a VM >>>> running on host 1, host 2 is SPM and all 3 VM's are seeing some >>>> network >>>> traffic courtesy of gluster. >>>> >>>> Now, for fun, suddenly the network on host3 goes bad (iptables -I >>>> OUTPUT >>>> -m statistic --mode random --probability 0.75 -j REJECT). >>>> Some time later I see the guest has a small "hickup", I'm guessing >>>> that >>>> is when gluster decides host 3 is not allowed to play anymore. No big >>>> deal anyway. >>>> After a while 25% of packages just isn't good enough for Ovirt >>>> anymore, >>>> so the host will be fenced. >>> >>> I'm not sure what fencing means w.r.t ovirt and what it actually >>> fences. As far is gluster is concerned, since only one node is >>> blocked, the VM image should still be accessible by the VM running >>> on host1. >> Fencing means (at least in this case) that the IPMI of the server >> does a power reset. >>>> After a reboot *sometimes* the VM will be >>>> paused, and even after the gluster self-heal is complete it can >>>> not be >>>> unpaused, has to be restarted. >>> >>> Could you provide the gluster mount (fuse?) logs and the brick logs >>> of all 3 nodes when the VM is paused? That should give us some clue. >>> >> Logs are attached. Problem was at around 8:15 - 8:20 UTC >> This time however the vm stopped even without a reboot of hyp03 > > > The mount logs (rhev-data-center-mnt-glusterSD*) are indicating > frequent disconnects to the bricks with 'clnt_ping_timer_expired', > 'Client-quorum is not met' and 'Read-only file system' messages. > client-quorum is enabled by default for replica 3 volumes. So if the > mount cannot connect to 2 bricks at least, quorum is lost and the > gluster volume becomes read-only. That seems to be the reason why the > VMs are pausing. > I'm not sure if the frequent disconnects are due a flaky network or > the bricks not responding to the mount's ping timer due to it's epoll > threads busy with I/O (unlikely). Can you also share the output of > `gluster volume info <volname>` ? The frequent disconnects are probably because I intentionally broke the network on hyp03 (dropped 75% of outgoing packets). In my opinion this should not affect the VM an hyp02. Am I wrong to think that?

[root@hyp01 ~]# gluster volume info VMS Volume Name: VMS Type: Replicate Volume ID: 9e6657e7-8520-4720-ba9d-78b14a86c8ca Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 10.99.50.20:/brick/VMS Brick2: 10.99.50.21:/brick/VMS Brick3: 10.99.50.22:/brick/VMS Options Reconfigured: performance.readdir-ahead: on nfs.disable: on user.cifs: disable auth.allow: * performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: enable cluster.quorum-type: auto cluster.server-quorum-type: server

storage.owner-uid: 36 storage.owner-gid: 36