<div dir="ltr"><div class="gmail_extra"><br>
<br><br><div class="gmail_quote">On Sun, Feb 23, 2014 at 3:20 PM, Ayal Baron <span dir="ltr"><<a href="mailto:abaron@redhat.com" target="_blank" onclick="window.open('https://mail.google.com/mail/?view=cm&tf=1&to=abaron@redhat.com&cc=&bcc=&su=&body=','_blank');return false;">abaron@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class=""><br>
<br>
----- Original Message -----<br>
> On Sun, Feb 23, 2014 at 4:27 AM, Ayal Baron <<a href="mailto:abaron@redhat.com" onclick="window.open('https://mail.google.com/mail/?view=cm&tf=1&to=abaron@redhat.com&cc=&bcc=&su=&body=','_blank');return false;">abaron@redhat.com</a>> wrote:<br>
><br>
> ><br>
> ><br>
> > ----- Original Message -----<br>
> > > I'm looking for some opinions on this configuration in an effort to<br>
> > increase<br>
> > > write performance:<br>
> > ><br>
> > > 3 storage nodes using glusterfs in replica 3, quorum.<br>
> ><br>
> > gluster doesn't support replica 3 yet, so I'm not sure how heavily I'd<br>
> > rely on this.<br>
> ><br>
><br>
> Glusterfs or RHSS doesn't support rep 3? How could I create a quorum<br>
> without 3+ hosts?<br>
<br>
</div>glusterfs has the capability but it hasn't been widely tested with oVirt yet and we've already found a couple of issues there.<br>
afaiu gluster has the ability to define a tie breaker (a third node which is part of the quorum but does not provide a third replica of the data).<br></blockquote><div><br></div><div>Good to know, I'll dig into this.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class=""><br>
><br>
><br>
> ><br>
> > > Ovirt storage domain via NFS<br>
> ><br>
> > why NFS and not gluster?<br>
> ><br>
><br>
> Gluster via posix SD doesn't have any performance gains over NFS, maybe the<br>
> opposite.<br>
<br>
</div>gluster via posix is mounting it using the gluster fuse client which should provide better performance + availability than NFS.<br></blockquote><div><br></div><div>Availability for sure, but performance is seriously questionable. I've run in both scenarios and haven't seen a performance improvement, the general consensus seems to be fuse is adding overhead and therefore decreasing performance vs. NFS.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class=""><br>
><br>
> Gluster 'native' SD's are broken on EL6.5 so I have been unable to test<br>
> performance. I have heard performance can be upwards of 3x NFS for raw<br>
> write.<br>
<br>
</div>Broken how?<br></blockquote><div><br></div><div>Ongoing issues, libgfapi support wasn't available, and then was disabled because snapshot support wasn't built into the kvm packages which was a dependency. There are a few threads in reference to this, and some effort to get CentOS builds to enable snapshot support in kvm.</div>
<div><br></div><div>I have installed rebuilt qemu packages with the RHEV snapshot flag enabled, and was just able to create a native gluster SD, maybe I missed something during a previous attempt. I'll test performance and see if its close to what I'm looking for.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class=""><br>
><br>
> Gluster doesn't have an async write option, so its doubtful it will ever be<br>
</div>> close to NFS async speeds.t<br>
<div class="">><br>
><br>
> ><br>
> > > Volume set nfs.trusted-sync on<br>
> > > On Ovirt, taking snapshots often enough to recover from a storage crash<br>
> ><br>
> > Note that this would have negative write performance impact<br>
> ><br>
><br>
> The difference between NFS sync (<50MB/s) and async (>300MB/s on 10g) write<br>
> speeds should more than compensate for the performance hit of taking<br>
> snapshots more often. And that's just raw speed. If we take into<br>
> consideration IOPS (guest small writes) async is leaps and bounds ahead.<br>
<br>
</div>I would test this, since qemu is already doing async I/O (using threads when native AIO is not supported) and oVirt runs it with cache=none (direct I/O) so sync ops should not happen that often (depends on guest). You may be still enjoying performance boost, but I've seen UPS systems fail before bringing down multiple nodes at once.<br>
In addition, if you do not guarantee your data is safe when you create a snapshot (and it doesn't seem like you are) then I see no reason to think your snapshots are any better off than latest state on disk.<br></blockquote>
<div><br></div><div>My logic here was if a snapshot is run, then the disk and system state should be consistent at time of snapshot once its been written to storage. If the host failed during snapshot then the snapshot would be incomplete, and the last complete snapshot would need to be used for recovery.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class=""><br>
><br>
><br>
> If we assume the site has backup UPS and generator power and we can build a<br>
> highly available storage system with 3 nodes in quorum, are there any<br>
> potential issues other than a write performance hit?<br>
><br>
> The issue I thought might be most prevalent is if an ovirt host goes down<br>
> and the VM's are automatically brought back up on another host, they could<br>
> incur disk corruption and need to be brought back down and restored to the<br>
> last snapshot state. This basically means the HA feature should be disabled.<br>
<br>
</div>I'm not sure I understand what your concern is here, what would cause the data corruption? if your node crashed then there is no I/O in flight. So starting up the VM should be perfectly safe.<br></blockquote><div>
<br></div><div>Good point, that makes sense.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class=""><br>
><br>
> Even worse, if the gluster node with CTDB NFS IP goes down, it may not have<br>
> written out and replicated to its peers. <-- I think I may have just<br>
> answered my own question.<br>
<br>
</div>If 'trusted-sync' means that the CTDB NFS node acks the I/O before it reached quorum then I'd say that's a gluster bug.</blockquote><div><br></div><div><a href="http://gluster.org/community/documentation/index.php/Gluster_3.2:_Setting_Volume_Options#nfs.trusted-sync">http://gluster.org/community/documentation/index.php/Gluster_3.2:_Setting_Volume_Options#nfs.trusted-sync</a> Specifically mentions data won't be guaranteed to be on disk, but doesn't mention if data would be replicated in memory between gluster nodes. Technically async breaks the NFS protocol standard anyways but this seems like a question for the gluster guys, I'll reach out on freenode.<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> It should ack the I/O before data hits the disc, but it should not ack it before it has quorum.<br>
However, the configuration we feel comfortable using gluster is with both server and client quorum (gluster has 2 different configs and you need to configure both to work safely).<br></blockquote><div><br></div><div>Is this specifically in relation to posix/fuse mounts? With libgfapi does the host pick up the config from the server side?</div>
<div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
<br>
><br>
><br>
> Thanks,<br>
> Steve<br>
><br>
</blockquote></div><br></div></div>