This is a multi-part message in MIME format. --------------9DE07E5668EB7C56D4297046 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Hello, Gluster-Performance is bad. Thats why I asked for native qemu-libgfapi access for Ovirt-VM's to gluster volumes which I thought to be possible since 3.6.x. Documentation is misleading and still in 4.1.2 Ovirt is using fuse to mount gluster-based VM-Disks. Bye Am 19.06.2017 um 17:23 schrieb Darrell Budic:
Chris-
You probably need to head over to gluster-users@gluster.org <mailto:gluster-users@gluster.org> for help with performance issues.
That said, what kind of performance are you getting, via some form or testing like bonnie++ or even dd runs? Raw bricks vs gluster performance is useful to determine what kind of performance you’re actually getting.
Beyond that, I’d recommend dropping the arbiter bricks and re-adding them as full replicas, they can’t serve distributed data in this configuration and may be slowing things down on you. If you’ve got a storage network setup, make sure it’s using the largest MTU it can, and consider adding/testing these settings that I use on my main storage volume:
performance.io <http://performance.io>-thread-count: 32 client.event-threads: 8 server.event-threads: 3 performance.stat-prefetch: on
Good luck,
-Darrell
On Jun 19, 2017, at 9:46 AM, Chris Boot <bootc@bootc.net <mailto:bootc@bootc.net>> wrote:
Hi folks,
I have 3x servers in a "hyper-converged" oVirt 4.1.2 + GlusterFS 3.10 configuration. My VMs run off a replica 3 arbiter 1 volume comprised of 6 bricks, which themselves live on two SSDs in each of the servers (one brick per SSD). The bricks are XFS on LVM thin volumes straight onto the SSDs. Connectivity is 10G Ethernet.
Performance within the VMs is pretty terrible. I experience very low throughput and random IO is really bad: it feels like a latency issue. On my oVirt nodes the SSDs are not generally very busy. The 10G network seems to run without errors (iperf3 gives bandwidth measurements of >= 9.20 Gbits/sec between the three servers).
To put this into perspective: I was getting better behaviour from NFS4 on a gigabit connection than I am with GlusterFS on 10G: that doesn't feel right at all.
My volume configuration looks like this:
Volume Name: vmssd Type: Distributed-Replicate Volume ID: d5a5ddd1-a140-4e0d-b514-701cfe464853 Status: Started Snapshot Count: 0 Number of Bricks: 2 x (2 + 1) = 6 Transport-type: tcp Bricks: Brick1: ovirt3:/gluster/ssd0_vmssd/brick Brick2: ovirt1:/gluster/ssd0_vmssd/brick Brick3: ovirt2:/gluster/ssd0_vmssd/brick (arbiter) Brick4: ovirt3:/gluster/ssd1_vmssd/brick Brick5: ovirt1:/gluster/ssd1_vmssd/brick Brick6: ovirt2:/gluster/ssd1_vmssd/brick (arbiter) Options Reconfigured: nfs.disable: on transport.address-family: inet6 performance.quick-read: off performance.read-ahead: off performance.io <http://performance.io>-cache: off performance.stat-prefetch: off performance.low-prio-threads: 32 network.remote-dio: off cluster.eager-lock: enable cluster.quorum-type: auto cluster.server-quorum-type: server cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-max-threads: 8 cluster.shd-wait-qlength: 10000 features.shard: on user.cifs: off storage.owner-uid: 36 storage.owner-gid: 36 features.shard-block-size: 128MB performance.strict-o-direct: on network.ping-timeout: 30 cluster.granular-entry-heal: enable
I would really appreciate some guidance on this to try to improve things because at this rate I will need to reconsider using GlusterFS altogether.
Cheers, Chris
-- Chris Boot bootc@bootc.net <mailto:bootc@bootc.net> _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- *Ralf Schenk* fon +49 (0) 24 05 / 40 83 70 fax +49 (0) 24 05 / 40 83 759 mail *rs@databay.de* <mailto:rs@databay.de> *Databay AG* Jens-Otto-Krag-Straße 11 D-52146 Würselen *www.databay.de* <http://www.databay.de> Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202 Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm. Philipp Hermanns Aufsichtsratsvorsitzender: Wilhelm Dohmen ------------------------------------------------------------------------ --------------9DE07E5668EB7C56D4297046 Content-Type: multipart/related; boundary="------------CE9FA4B243E602B4EF2B1DB6" --------------CE9FA4B243E602B4EF2B1DB6 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <p><font face="Helvetica, Arial, sans-serif">Hello,</font></p> <p><font face="Helvetica, Arial, sans-serif">Gluster-Performance is bad. Thats why I asked for native qemu-libgfapi access for Ovirt-VM's to gluster volumes which I thought to be possible since 3.6.x. Documentation is misleading and still in 4.1.2 Ovirt is using fuse to mount gluster-based VM-Disks.</font></p> <p><font face="Helvetica, Arial, sans-serif"></font>Bye<br> </p> <br> <div class="moz-cite-prefix">Am 19.06.2017 um 17:23 schrieb Darrell Budic:<br> </div> <blockquote type="cite" cite="mid:D8AAF9DB-02FB-4B38-9E33-174134F5377C@onholyground.com"> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> Chris- <div class=""><br class=""> </div> <div class="">You probably need to head over to <a href="mailto:gluster-users@gluster.org" class="" moz-do-not-send="true">gluster-users@gluster.org</a> for help with performance issues.</div> <div class=""><br class=""> </div> <div class="">That said, what kind of performance are you getting, via some form or testing like bonnie++ or even dd runs? Raw bricks vs gluster performance is useful to determine what kind of performance you’re actually getting.</div> <div class=""><br class=""> </div> <div class="">Beyond that, I’d recommend dropping the arbiter bricks and re-adding them as full replicas, they can’t serve distributed data in this configuration and may be slowing things down on you. If you’ve got a storage network setup, make sure it’s using the largest MTU it can, and consider adding/testing these settings that I use on my main storage volume:</div> <div class=""><br class=""> </div> <div class=""> <div style="margin: 0px; line-height: normal;" class=""><a href="http://performance.io" class="" moz-do-not-send="true">performance.io</a>-thread-count: 32</div> <div style="margin: 0px; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">client.event-threads: 8</span></div> <div style="margin: 0px; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">server.event-threads: 3</span></div> <div style="margin: 0px; line-height: normal;" class="">performance.stat-prefetch: on</div> </div> <div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""> </span></div> <div class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Good luck,</span></div> <div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""> </span></div> <div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> -Darrell</span></div> <div class=""><br class=""> </div> <div class=""><br class=""> <div> <blockquote type="cite" class=""> <div class="">On Jun 19, 2017, at 9:46 AM, Chris Boot <<a href="mailto:bootc@bootc.net" class="" moz-do-not-send="true">bootc@bootc.net</a>> wrote:</div> <br class="Apple-interchange-newline"> <div class=""> <div class="">Hi folks,<br class=""> <br class=""> I have 3x servers in a "hyper-converged" oVirt 4.1.2 + GlusterFS 3.10<br class=""> configuration. My VMs run off a replica 3 arbiter 1 volume comprised of<br class=""> 6 bricks, which themselves live on two SSDs in each of the servers (one<br class=""> brick per SSD). The bricks are XFS on LVM thin volumes straight onto the<br class=""> SSDs. Connectivity is 10G Ethernet.<br class=""> <br class=""> Performance within the VMs is pretty terrible. I experience very low<br class=""> throughput and random IO is really bad: it feels like a latency issue.<br class=""> On my oVirt nodes the SSDs are not generally very busy. The 10G network<br class=""> seems to run without errors (iperf3 gives bandwidth measurements of >=<br class=""> 9.20 Gbits/sec between the three servers).<br class=""> <br class=""> To put this into perspective: I was getting better behaviour from NFS4<br class=""> on a gigabit connection than I am with GlusterFS on 10G: that doesn't<br class=""> feel right at all.<br class=""> <br class=""> My volume configuration looks like this:<br class=""> <br class=""> Volume Name: vmssd<br class=""> Type: Distributed-Replicate<br class=""> Volume ID: d5a5ddd1-a140-4e0d-b514-701cfe464853<br class=""> Status: Started<br class=""> Snapshot Count: 0<br class=""> Number of Bricks: 2 x (2 + 1) = 6<br class=""> Transport-type: tcp<br class=""> Bricks:<br class=""> Brick1: ovirt3:/gluster/ssd0_vmssd/brick<br class=""> Brick2: ovirt1:/gluster/ssd0_vmssd/brick<br class=""> Brick3: ovirt2:/gluster/ssd0_vmssd/brick (arbiter)<br class=""> Brick4: ovirt3:/gluster/ssd1_vmssd/brick<br class=""> Brick5: ovirt1:/gluster/ssd1_vmssd/brick<br class=""> Brick6: ovirt2:/gluster/ssd1_vmssd/brick (arbiter)<br class=""> Options Reconfigured:<br class=""> nfs.disable: on<br class=""> transport.address-family: inet6<br class=""> performance.quick-read: off<br class=""> performance.read-ahead: off<br class=""> <a href="http://performance.io" class="" moz-do-not-send="true">performance.io</a>-cache: off<br class=""> performance.stat-prefetch: off<br class=""> performance.low-prio-threads: 32<br class=""> network.remote-dio: off<br class=""> cluster.eager-lock: enable<br class=""> cluster.quorum-type: auto<br class=""> cluster.server-quorum-type: server<br class=""> cluster.data-self-heal-algorithm: full<br class=""> cluster.locking-scheme: granular<br class=""> cluster.shd-max-threads: 8<br class=""> cluster.shd-wait-qlength: 10000<br class=""> features.shard: on<br class=""> user.cifs: off<br class=""> storage.owner-uid: 36<br class=""> storage.owner-gid: 36<br class=""> features.shard-block-size: 128MB<br class=""> performance.strict-o-direct: on<br class=""> network.ping-timeout: 30<br class=""> cluster.granular-entry-heal: enable<br class=""> <br class=""> I would really appreciate some guidance on this to try to improve things<br class=""> because at this rate I will need to reconsider using GlusterFS altogether.<br class=""> <br class=""> Cheers,<br class=""> Chris<br class=""> <br class=""> -- <br class=""> Chris Boot<br class=""> <a href="mailto:bootc@bootc.net" class="" moz-do-not-send="true">bootc@bootc.net</a><br class=""> _______________________________________________<br class=""> Users mailing list<br class=""> <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a><br class=""> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a><br class=""> </div> </div> </blockquote> </div> <br class=""> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <div class="moz-signature">-- <br> <p> </p> <table cellspacing="0" cellpadding="0" border="0"> <tbody> <tr> <td colspan="3"><img src="cid:part6.276D40AB.8385CD25@databay.de" height="30" width="151" border="0"></td> </tr> <tr> <td valign="top"> <font size="-1" face="Verdana, Arial, sans-serif"><br> <b>Ralf Schenk</b><br> fon +49 (0) 24 05 / 40 83 70<br> fax +49 (0) 24 05 / 40 83 759<br> mail <a href="mailto:rs@databay.de"><font color="#FF0000"><b>rs@databay.de</b></font></a><br> </font> </td> <td width="30"> </td> <td valign="top"> <font size="-1" face="Verdana, Arial, sans-serif"><br> <b>Databay AG</b><br> Jens-Otto-Krag-Straße 11<br> D-52146 Würselen<br> <a href="http://www.databay.de"><font color="#FF0000"><b>www.databay.de</b></font></a> </font> </td> </tr> <tr> <td colspan="3" valign="top"> <font size="1" face="Verdana, Arial, sans-serif"><br> Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202<br> Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm. Philipp Hermanns<br> Aufsichtsratsvorsitzender: Wilhelm Dohmen </font> </td> </tr> </tbody> </table> <hr noshade="noshade" size="1" color="#000000" width="100%"> </div> </body> </html> --------------CE9FA4B243E602B4EF2B1DB6 Content-Type: image/gif; name="geegbplfmmafjmcf.gif" Content-Transfer-Encoding: base64 Content-ID: <part6.276D40AB.8385CD25@databay.de> Content-Disposition: inline; filename="geegbplfmmafjmcf.gif" R0lGODlhlwAeAMQAAObm5v9QVf/R0oKBgfDw8NfX105MTLi3t/r6+sfHx/+rrf98gC0sLP8L EhIQEKalpf/g4ZmYmHd2dmppaf8uNP/y8v8cIv+Ym//AwkE/P46NjRwbG11cXP8ABwUDA/// /yH5BAAAAAAALAAAAACXAB4AAAX/4CeOYnUJZKqubOu+cCzPNA0tVnfVfO//wGAKk+t0Ap+K QMFUYCDCqHRKJVUWDaPRUsFktZ1G4AKtms9o1gKsFVS+7I5ll67bpd647hPQawNld4KDMQJF bA07F35aFBiEkJEpfXEBjx8KjI0Vkp2DEIdaCySgFBShbEgrCQOtrq+uEQcALQewrQUjEbe8 rgkkD7y5KhMZB3drqSoVFQhdlHGXKQYe1dbX2BvHKwzY1RMiAN7j1xEjBeTmKeIeD3cYCxRf FigvChRxFJwkBBvk5A7cpZhAjgGCDwn+kfslgto4CSoSehh2BwEEBQvowDAUR0EKdArHZTg4 4oDCXBFC/3qj9SEluZEpHnjYQFIGgpo1KgSasYjNKBImrzF4NaFbNgIjCGRQeIyVKwneOLzS cLCAg38OWI4Y4GECgQcSOEwYcADnh6/FNjAwoGFYAQ0atI4AAFeEFwsLFLiJUQEfGH0kNGAD x8+oNQdIRQg+7NCaOhIgD8sVgYADNsPVGI5YWjRqzQTdHDDIYHRDLokaUhCglkFEJi0NKJhl 0RP2TsvXUg88KiLBVWsZrF6DmMKlNYMqglqTik1guN8OBgAgkGCpB+L9ugK4iSCBvwEfECw1 kILrBpa1jVCQIQBRvbP+rlEcQVAoSevWyv6uhpwE12uEkQAAZucpVw1xIsjkgf8B863mQVYt eQATCZYJZJ5WBfij2wfpHcEeHGG8Z+BMszVWDXkfKLhceJhBSAJ+1ThH32AfRFZNayNAtUFi wFSTSwEHJIYAAQU84IADwyjIEALU9MchG+vFgIF7W2GDI2T7HfjBgNcgKQKMHmwjgnCSpeCb ULRkdxhF1CDY40RjgmUAA/v1J5FAKW2gGSZscBFDMraNgJs1AYpAAGYP5jJoNQ4Y4Gh8jpFg HH9mgbmWo1l6oA4C3Ygp6UwEIFBfNRtkMIBlKMLnAXgAXLWhXXH85EIFqMhGGZgDEKArABGA ed0HI4bk5qgnprCYSt88B6dqS0FEEAMPJDCdCJYViur/B1BlwGMJqDTwnhqxJgUpo0ceOQ4D 0yEakpMm/jqCRMgWm2I1j824Y6vLvuuPjHnqOJkIgP6xzwp5sCFNsCFp88Gxh11lrjfDcNrc CEx64/CD3iAHlQcMUEQXvcA+qBkBB4Q2X1CusjBlJdKMYAKI6g28MbKN5hJsBAXknHOwutn4 oFYqkpqAzjnPbE0u1PxmwAQGXLWBbvhuIIEGEnRjlAHO4SvhbCNAkwoGzEBwgV9U0lfu2WiX OkDEGaCdKgl0nk2YkWdPOCDabvaGdkAftL1LlgwCM+7Tq11V71IO7LkM2XE0YAHMYMhqqK6U V165CpaHukLmiXFO8XSVzzakX+UH6TrmAajPNxfqByTQec41AeBPvSwIALkmAnuiexCsca3C BajgfsROuxcPA8kHQJX4DAIwjnsAvhsvfXHWKEwDAljg7sj03L9wwAQTxOWD2AE0YP75eCkw cPfs+xACADs= --------------CE9FA4B243E602B4EF2B1DB6-- --------------9DE07E5668EB7C56D4297046--