On Mon, Mar 23, 2020 at 11:44 PM Christian Reiss <email@christian-reiss.de> wrote:
Hey folks,
gluster related question. Having SSD in a RAID that can do 2 GB writes and Reads (actually above, but meh) in a 3-way HCI cluster connected with 10gbit connection things are pretty slow inside gluster. I have these settings:
Options Reconfigured: cluster.server-quorum-type: server cluster.quorum-type: auto cluster.shd-max-threads: 8 features.shard: on features.shard-block-size: 64MB server.event-threads: 8 user.cifs: off cluster.shd-wait-qlength: 10000 cluster.locking-scheme: granular cluster.eager-lock: enable performance.low-prio-threads: 32 network.ping-timeout: 30 cluster.granular-entry-heal: enable storage.owner-gid: 36 storage.owner-uid: 36 cluster.choose-local: true client.event-threads: 16
These settings mean:
performance.strict-o-direct: on network.remote-dio: enable
That you are using direct I/O both on the client and server side.
performance.client-io-threads: on nfs.disable: on storage.fips-mode-rchecksum: on transport.address-family: inet cluster.readdir-optimize: on cluster.metadata-self-heal: on cluster.data-self-heal: on cluster.entry-self-heal: on cluster.data-self-heal-algorithm: full features.uss: enable features.show-snapshot-directory: on features.barrier: disable auto-delete: enable snap-activate-on-create: enable
Writing inside the /gluster_bricks yields those 2GB/sec writes, Reading the same.
How did you test this? Did you test reading from the storage on the server side using direct I/O? if not, you test accessing server buffer cache, which is pretty fast.
Reading inside the /rhev/data-center/mnt/glusterSD/ dir reads go down to 366mb/sec while writes plummet to to 200mb/sec.
This use direct I/O.
Summed up: Writing into the SSD Raid in the lvm/xfs gluster brick directory is fast, writing into the mounted gluster dir is horribly slow.
The above can be seen and repeated on all 3 servers. The network can do full 10gbit (tested with, among others: rsync, iperf3).
Anyone with some idea on whats missing/ going on here?
Please share the commands/configuration files used to perform the tests. Adding storage folks that can help with analyzing this.
Thanks folks, as always stay safe and healthy!
Nir
-- with kind regards, mit freundlichen Gruessen,
Christian Reiss _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/OMAAERV4IUISYE...