On March 24, 2020 12:08:08 AM GMT+02:00, Jayme <jaymef@gmail.com> wrote:
I too struggle with speed issues in hci. Latency is a big problem with writes for me especially when dealing with small file workloads. How are you testing exactly?
Look into enabling libgfapi and try some comparisons with that. People have been saying it’s much faster, but it’s not a default option and has a few bugs. Redhat devs do not appear to be giving its implementation any priority unfortunately.
I’ve been considering switching to nfs storage because I’m seeing much better performance in testing with it. I have some nvme drives on the way and am curious how they would perform in hci but I’m thinking the issue is not a disk bottleneck (that appears very obvious in your case as well)
On Mon, Mar 23, 2020 at 6:44 PM Christian Reiss <email@christian-reiss.de> wrote:
Hey folks,
gluster related question. Having SSD in a RAID that can do 2 GB writes and Reads (actually above, but meh) in a 3-way HCI cluster connected with 10gbit connection things are pretty slow inside gluster.
I have these settings:
Options Reconfigured: cluster.server-quorum-type: server cluster.quorum-type: auto cluster.shd-max-threads: 8 features.shard: on features.shard-block-size: 64MB server.event-threads: 8 user.cifs: off cluster.shd-wait-qlength: 10000 cluster.locking-scheme: granular cluster.eager-lock: enable performance.low-prio-threads: 32 network.ping-timeout: 30 cluster.granular-entry-heal: enable storage.owner-gid: 36 storage.owner-uid: 36 cluster.choose-local: true client.event-threads: 16 performance.strict-o-direct: on network.remote-dio: enable performance.client-io-threads: on nfs.disable: on storage.fips-mode-rchecksum: on transport.address-family: inet cluster.readdir-optimize: on cluster.metadata-self-heal: on cluster.data-self-heal: on cluster.entry-self-heal: on cluster.data-self-heal-algorithm: full features.uss: enable features.show-snapshot-directory: on features.barrier: disable auto-delete: enable snap-activate-on-create: enable
Writing inside the /gluster_bricks yields those 2GB/sec writes, Reading the same.
Reading inside the /rhev/data-center/mnt/glusterSD/ dir reads go down to 366mb/sec while writes plummet to to 200mb/sec.
Summed up: Writing into the SSD Raid in the lvm/xfs gluster brick directory is fast, writing into the mounted gluster dir is horribly slow.
The above can be seen and repeated on all 3 servers. The network can do full 10gbit (tested with, among others: rsync, iperf3).
Anyone with some idea on whats missing/ going on here?
Thanks folks, as always stay safe and healthy!
-- with kind regards, mit freundlichen Gruessen,
Christian Reiss _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/OMAAERV4IUISYE...
Hey Chris,, You got some options. 1. To speedup the reads in HCI - you can use the option : cluster.choose-local: on 2. You can adjust the server and client event-threads 3. You can use NFS Ganesha (which connects to all servers via libgfapi) as a NFS Server. In such case you have to use some clustering like ctdb or pacemaker. Note:disable cluster.choose-local if you use this one 4 You can try the built-in NFS , although it's deprecated (NFS Ganesha is fully supported) 5. Create a gluster profile during the tests. I have seen numerous improperly selected tests -> so test with real-world workload. Synthetic tests are not good. Best Regards, Strahil Nikolov