<div dir="ltr"><div><div><div><div>I don't see any other bottleneck. CPUs are quite idle. Seems that the load is mostly due to high latency on IO. <br><br></div>Reading further the gluster docs: <br></div><div><br></div><div><a href="https://github.com/gluster/glusterfs-specs/blob/master/done/GlusterFS%203.5/libgfapi%20with%20qemu%20libvirt.md">https://github.com/gluster/glusterfs-specs/blob/master/done/GlusterFS%203.5/libgfapi%20with%20qemu%20libvirt.md</a><br></div><div><br></div><div>I see that I am missing the following options: <br><br>/etc/glusterfs/glusterd.vol : <br>option rpc-auth-allow-insecure on<br><br>gluster volume set vms gluster server.allow-insecure on<br><br></div>It says that the above allow qemu to use libgfapi. <br></div>When checking the VM XML, I don't see any gluster protocol at the disk drive: <br><br><span style="font-family:monospace,monospace"><disk type='file' device='disk' snapshot='no'><br> <driver name='qemu' type='raw' cache='none' error_policy='stop' io='threads'/><br> <source file='/rhev/data-center/00000001-0001-0001-0001-000000000311/94741028-e765-4300-a618-c3eeb7dbb7c8/images/222a1312-5efa-4731-8914-9a9d24dccba5/d691e6b3-c8e7-4820-9042-555d30c8a21b'/><br> <target dev='sda' bus='scsi'/><br> <serial>222a1312-5efa-4731-8914-9a9d24dccba5</serial><br> <boot order='1'/><br> <address type='drive' controller='0' bus='0' target='0' unit='0'/><br> </disk><br></span><br><br><br></div>While at gluster docs it mentions the below type of disk: <br><br><pre><code><disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source protocol='gluster' name='distrepvol/vm3.img'>
<host name='10.70.37.106' port='24007'/>
</source>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk><br><br><span style="font-family:arial,helvetica,sans-serif"><br></span></code></pre><pre><code><span style="font-family:arial,helvetica,sans-serif">Does the above indicate that ovirt/qemu is not using libgfapi but FUSE only? <br>This could be the reason of such slow perf. <br></span><br></code></pre></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 7, 2017 at 1:47 PM, Yaniv Kaul <span dir="ltr"><<a href="mailto:ykaul@redhat.com" target="_blank">ykaul@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Thu, Sep 7, 2017 at 12:52 PM, Abi Askushi <span dir="ltr"><<a href="mailto:rightkicktech@gmail.com" target="_blank">rightkicktech@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Thu, Sep 7, 2017 at 10:30 AM, Yaniv Kaul <span dir="ltr"><<a href="mailto:ykaul@redhat.com" target="_blank">ykaul@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="m_-2506883854242430591m_-841542638229875602gmail-">On Thu, Sep 7, 2017 at 10:06 AM, Yaniv Kaul <span dir="ltr"><<a href="mailto:ykaul@redhat.com" target="_blank">ykaul@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Wed, Sep 6, 2017 at 6:08 PM, Abi Askushi <span dir="ltr"><<a href="mailto:rightkicktech@gmail.com" target="_blank">rightkicktech@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div><div><div>For a first idea I use:<br></div><div><span style="font-family:monospace,monospace"><br></span></div><div><span style="font-family:monospace,monospace">dd if=/dev/zero of=testfile bs=1GB count=1</span></div></div></div></div></div></div></div></div></div></div></div></blockquote><div><br></div></span><div>This is an incorrect way to test performance, for various reasons:</div><div>1. You are not using oflag=direct , thus not using DirectIO, but using cache.</div><div>2. It's unrealistic - it is very uncommon to write large blocks of zeros (sometimes during FS creation or wiping). Certainly not 1GB</div><div>3. It is a single thread of IO - again, unrealistic for VM's IO.</div><div><br></div></div></div></div></blockquote></span></div></div></div></blockquote></span><div>I forgot to mention that I include oflag=direct in my tests. I agree though that dd is not the correct way to test, hence I mentioned I just use it to get a first feel. More tests are done within the VM benchmarking its disk IO (with tools like IOmeter). <br></div><span><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="m_-2506883854242430591m_-841542638229875602gmail-"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div>I suggest using fio and such. See <a href="https://github.com/pcuzner/fio-tools" target="_blank">https://github.com/pcuzner<wbr>/fio-tools</a> for example.</div></div></div></div></blockquote></span></div></div></div></blockquote></span><div>Do you have any recommended config file to use for VM workload?</div></div></div></div></blockquote><div><br></div></span><div>Desktops and Servers VMs behave quite differently, so not really. But the 70/30 job is typically a good baseline.</div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="m_-2506883854242430591m_-841542638229875602gmail-"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div><div><div><br></div><div><br></div><div>When testing on the gluster mount point using above command I hardly get 10MB/s. (On the same time the network traffic hardly reaches 100Mbit). <br></div><div><br></div>When testing our of the gluster (for example at /root) I get 600 - 700MB/s. <br></div></div></div></div></div></div></div></div></div></div></blockquote><div><br></div></span><div>That's very fast - from 4 disks doing RAID5? Impressive (unless you use caching!). Are those HDDs or SSDs/NVMe?</div><span><div> </div></span></div></div></div></blockquote></span></div></div></div></blockquote></span><div>These are SAS disks. But there is also a RAID controller with 1GB cache.<br></div><span><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="m_-2506883854242430591m_-841542638229875602gmail-"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div><div></div><div><br></div>When I mount the gluster volume with NFS and test on it I get 90 - 100 MB/s, (almost 10x from gluster results) which is the max I can get considering I have only 1 Gbit network for the storage. <br><br></div>Also, when using glusterfs the general VM performance is very poor and disk write benchmarks show that is it at least 4 times slower then when the VM is hosted on the same data store when NFS mounted. <br></div><div><br></div>I don't know why I hitting such a significant performance penalty, and every possible tweak that I was able to find out there did not make any difference on the performance. <br><br></div>The hardware I am using is pretty decent for the purposes intended:<br></div>3 nodes, each node having with 32 MB of RAM, 16 physical CPU cores, 2 TB of storage in RAID5 (4 disks), of which 1.5 TB are sliced for the data store of ovirt where VMs are stored. <br></div></div></div></div></div></blockquote></span></div></div></div></blockquote><div><br></div></span><div>I forgot to ask why are you using RAID 5 with 4 disks and not RAID 10? Same usable capacity, higher performance, same protection and faster recovery, I believe.</div></div></div></div></blockquote></span><div>Correction: there are 5 disks of 600GB each. The main reason going with RAID 5 was the capacity. With RAID 10 I can use only 4 of them and get only 1.1 TB usable, with RAID 5 I get 2.2 TB usable. I agree going with RAID 10 (+ one additional drive to go with 6 drives) would be better but this is what I have now. <br></div><span><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="m_-2506883854242430591m_-841542638229875602gmail-HOEnZb"><font color="#888888"><div>Y.</div></font></span><div><div class="m_-2506883854242430591m_-841542638229875602gmail-h5"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div></div></div></div></div></div></blockquote><div><br></div></span><div>You have not mentioned your NIC speeds. Please ensure all work well, with 10g.</div><div>Is the network dedicated for Gluster traffic? How are they connected?</div><span><div> </div></span></div></div></div></blockquote></div></div></div></div></div></blockquote></span><div>I have mentioned that I have 1 Gbit dedicated for the storage. A different network is used for this and a dedicated 1Gbit switch. The throughput has been confirmed between all nodes with iperf. <br></div></div></div></div></blockquote><div><br></div></span><div>Oh.... With 1Gb, you can't get more than 100+MBps...</div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div>I know 10Gbit would be better, but when using native gluster at ovirt the network pipe was hardly reaching 100Mbps thus the bottleneck was gluster and not the network. If I can saturate 1Gbit and I still have performance issues then I may think to go with 10Gbit. With NFS on top gluster I see traffic reaching 800Mbit when testing with dd which is much better. <br></div></div></div></div></blockquote><div><br></div></span><div>Agreed. Do you see the bottleneck elsewhere? CPU?</div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><span><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="m_-2506883854242430591m_-841542638229875602gmail-h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><br></div>The gluster configuration is the following: <br></div></div></div></div></blockquote><div><br></div></span><div>Which version of Gluster are you using?</div><span><div> </div></span></div></div></div></blockquote></div></div></div></div></div></blockquote></span><div>The version is 3.8.12 <br></div></div></div></div></blockquote><div><br></div></span><div>I think it's a very old release (near end of life?). I warmly suggest 3.10.x or 3.12.</div><div>There are performance improvements (AFAIR) in both.</div><span class="HOEnZb"><font color="#888888"><div>Y.</div></font></span><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div><div class="m_-2506883854242430591h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="m_-2506883854242430591m_-841542638229875602gmail-h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><br>Volume Name: vms<br>Type: Replicate<br>Volume ID: 4513340d-7919-498b-bfe0-d836b5<wbr>cea40b<br>Status: Started<br>Snapshot Count: 0<br>Number of Bricks: 1 x (2 + 1) = 3<br>Transport-type: tcp<br>Bricks:<br>Brick1: gluster0:/gluster/vms/brick<br>Brick2: gluster1:/gluster/vms/brick<br>Brick3: gluster2:/gluster/vms/brick (arbiter)<br>Options Reconfigured:<br>nfs.export-volumes: on<br>nfs.disable: off<br>performance.readdir-ahead: on<br>transport.address-family: inet<br>performance.quick-read: off<br>performance.read-ahead: off<br>performance.io-cache: off<br>performance.stat-prefetch: on<br></div></div></div></div></blockquote><div><br></div></span><div>I think this should be off.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div>performance.low-prio-threads: 32<br>network.remote-dio: off<br></div></div></div></div></blockquote><div><br></div><div>I think this should be enabled.</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div>cluster.eager-lock: off<br>cluster.quorum-type: auto<br>cluster.server-quorum-type: server<br>cluster.data-self-heal-algorit<wbr>hm: full<br>cluster.locking-scheme: granular<br>cluster.shd-max-threads: 8<br>cluster.shd-wait-qlength: 10000<br>features.shard: on<br>user.cifs: off<br>storage.owner-uid: 36<br>storage.owner-gid: 36<br>network.ping-timeout: 30<br>performance.strict-o-direct: on<br>cluster.granular-entry-heal: enable<br>features.shard-block-size: 64MB<br></div></div></div></div></blockquote><div><br></div></span><div>I'm not sure if this should not be 512MB. I don't remember the last resolution on this.</div><div>Y.</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div>performance.client-io-threads: on<br>client.event-threads: 4<br>server.event-threads: 4<br>performance.write-behind-windo<wbr>w-size: 4MB<br>performance.cache-size: 1GB<br><br></div></div></div></div></blockquote></span></div></div></div></blockquote></div></div></div></div></div></blockquote></div></div><div>I have been playing with all above with very little difference on performance I was getting.<br></div><span><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="m_-2506883854242430591m_-841542638229875602gmail-h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div></div>In case I can provide any other details let me know. <br></div></div></div></blockquote><div><br></div></span><div>What is your tuned profile?</div><span><div> </div></span></div></div></div></blockquote></div></div></div></div></div></blockquote></span><div>the tuned profile is virtual-host</div><span><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="m_-2506883854242430591m_-841542638229875602gmail-h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div></div>At the moment I already switched to gluster based NFS but I have a similar setup with 2 nodes where the data store is mounted through gluster (and again relatively good hardware) where I might check any tweaks or improvements on this setup. <br><br></div>Thanx<br><div><br></div></div><div class="m_-2506883854242430591m_-841542638229875602gmail-m_-6797784585761107416m_-2474129227919145367gmail-m_-571567959162763644HOEnZb"><div class="m_-2506883854242430591m_-841542638229875602gmail-m_-6797784585761107416m_-2474129227919145367gmail-m_-571567959162763644h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 6, 2017 at 5:32 PM, Yaniv Kaul <span dir="ltr"><<a href="mailto:ykaul@redhat.com" target="_blank">ykaul@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Wed, Sep 6, 2017 at 3:32 PM, Abi Askushi <span dir="ltr"><<a href="mailto:rightkicktech@gmail.com" target="_blank">rightkicktech@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div><div><div>Hi All,<br><br></div>I've playing with ovirt self hosted engine setup and I even use it to production for several VM. The setup I have is 3 server with gluster storage in replica 2+1 (1 arbiter). <br></div>The data storage domain where VMs are stored is mounted with gluster through ovirt. The performance I get for the VMs is very low and I was thinking to switch and mount the same storage through NFS instead of glusterfs. <br></div></div></div></div></div></div></div></div></div></blockquote><div><br></div></span><div>I don't see how it'll improve performance.</div><div>I suggest you share the gluster configuration (as well as the storage HW) so we can understand why the performance is low.</div><div>Y.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span><div dir="ltr"><div><div><div><div><div><div><div><div><br></div>The only think I am hesitant is how can I ensure high availability of the storage when I loose one server? I was thinking to have at /etc/hosts sth like below: <br><br></div>10.100.100.1 nfsmount<br></div>10.100.100.2 nfsmount<br></div>10.100.100.3 nfsmount<br><br></div>then use nfsmount as the server name when adding this domain through ovirt GUI. <br></div>Are there any other more elegant solutions? What do you do for such cases?<br></div>Note: gluster has the back-vol-file option which provides a lean way to have redundancy on the mount point and I am using this when mounting with glusterfs. <br><br></div>Thanx<br></div>
<br></span>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
<br></blockquote></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></span></div><br></div></div>
</blockquote></div></div></div><br></div></div>
</blockquote></span></div><br></div></div>
</blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>