1:09 a.m.
Strahil,
Thanks for your suggestions. The config is pretty standard HCI setup with
cockpit and hosts are oVirt node. XFS was handled by the deployment
automatically. The gluster volumes were optimized for virt store.
I tried noop on the SSDs, that made zero difference in the tests I was
running above. I took a look at the random-io-profile and it looks like it
really only sets vm.dirty_background_ratio = 2 & vm.dirty_ratio = 5 -- my
hosts already appear to have those sysctl values, and by default are
using virtual-host tuned profile.
I'm curious what a test like "dd if=/dev/zero of=test2.img bs=512
count=1000 oflag=dsync" on one of your VMs would show for results?
I haven't done much with gluster profiling but will take a look and see if
I can make sense of it. Otherwise, the setup is pretty stock oVirt HCI
deployment with SSD backed storage and 10Gbe storage network. I'm not
coming anywhere close to maxing network throughput.
The NFS export I was testing was an export from a local server exporting a
single SSD (same type as in the oVirt hosts).
I might end up switching storage to NFS and ditching gluster if performance
is really this much better...
On Fri, Mar 6, 2020 at 5:06 PM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
On March 6, 2020 6:02:03 PM GMT+02:00, Jayme <jaymef(a)gmail.com>
wrote:
>I have 3 server HCI with Gluster replica 3 storage (10GBe and SSD
>disks).
>Small file performance inner-vm is pretty terrible compared to a
>similar
>spec'ed VM using NFS mount (10GBe network, SSD disk)
>
>VM with gluster storage:
>
># dd if=/dev/zero of=test2.img bs=512 count=1000 oflag=dsync
>1000+0 records in
>1000+0 records out
>512000 bytes (512 kB) copied, 53.9616 s, 9.5 kB/s
>
>VM with NFS:
>
># dd if=/dev/zero of=test2.img bs=512 count=1000 oflag=dsync
>1000+0 records in
>1000+0 records out
>512000 bytes (512 kB) copied, 2.20059 s, 233 kB/s
>
>This is a very big difference, 2 seconds to copy 1000 files on NFS VM
>VS 53
>seconds on the other.
>
>Aside from enabling libgfapi is there anything I can tune on the
>gluster or
>VM side to improve small file performance? I have seen some guides by
>Redhat in regards to small file performance but I'm not sure what/if
>any of
>it applies to oVirt's implementation of gluster in HCI.
You can use the rhgs-random-io tuned profile from
ftp://ftp.redhat.com/redhat/linux/enterprise/7Server/en/RHS/SRPMS/redhat-...
and try with that on your hosts.
In my case, I have modified it so it's a mixture between rhgs-random-io
and the profile for Virtualization Host.
Also,ensure that your bricks are using XFS with relatime/noatime mount
option and your scheduler for the SSDs is either 'noop' or 'none' .The
default I/O scheduler for RHEL7 is deadline which is giving preference to
reads and your workload is definitely 'write'.
Ensure that the virt settings are enabled for your gluster volumes:
'gluster volume set <volname> group virt'
Also, are you running on fully allocated disks for the VM or you started
thin ?
I'm asking as creation of new shards at gluster level is a slow task.
Have you checked gluster profiling the volume? It can clarify what is
going on.
Also are you comparing apples to apples ?
For example, 1 ssd mounted and exported as NFS and a replica 3 volume
of the same type of ssd ? If not, the NFS can have more iops due to
multiple disks behind it, while Gluster has to write the same thing on all
nodes.
Best Regards,
Strahil Nikolov