On 2019-04-13 03:15, Strahil wrote:
> Hi,
>
> What is your dirty  cache settings on the gluster servers  ?
>
> Best Regards,
> Strahil NikolovOn Apr 13, 2019 00:44, Alex McWhirter <alex@triadic.us>
> wrote:
>>
>> I have 8 machines acting as gluster servers. They each have 12 drives
>> raid 50'd together (3 sets of 4 drives raid 5'd then 0'd together as
>> one).
>>
>> They connect to the compute hosts and to each other over lacp'd 10GB
>> connections split across two cisco nexus switched with VPC.
>>
>> Gluster has the following set.
>>
>> performance.write-behind-window-size: 4MB
>> performance.flush-behind: on
>> performance.stat-prefetch: on
>> server.event-threads: 4
>> client.event-threads: 8
>> performance.io-thread-count: 32
>> network.ping-timeout: 30
>> cluster.granular-entry-heal: enable
>> performance.strict-o-direct: on
>> storage.owner-gid: 36
>> storage.owner-uid: 36
>> features.shard: on
>> cluster.shd-wait-qlength: 10000
>> cluster.shd-max-threads: 8
>> cluster.locking-scheme: granular
>> cluster.data-self-heal-algorithm: full
>> cluster.server-quorum-type: server
>> cluster.quorum-type: auto
>> cluster.eager-lock: enable
>> network.remote-dio: off
>> performance.low-prio-threads: 32
>> performance.io-cache: off
>> performance.read-ahead: off
>> performance.quick-read: off
>> auth.allow: *
>> user.cifs: off
>> transport.address-family: inet
>> nfs.disable: off
>> performance.client-io-threads: on
>>
>>
>> I have the following sysctl values on gluster client and servers,
>> using
>> libgfapi, MTU 9K
>>
>> net.core.rmem_max = 134217728
>> net.core.wmem_max = 134217728
>> net.ipv4.tcp_rmem = 4096 87380 134217728
>> net.ipv4.tcp_wmem = 4096 65536 134217728
>> net.core.netdev_max_backlog = 300000
>> net.ipv4.tcp_moderate_rcvbuf =1
>> net.ipv4.tcp_no_metrics_save = 1
>> net.ipv4.tcp_congestion_control=htcp
>>
>> reads with this setup are perfect, benchmarked in VM to be about
>> 770MB/s
>> sequential with disk access times of < 1ms. Writes on the other hand
>> are
>> all over the place. They peak around 320MB/s sequential write, which
>> is
>> what i expect but it seems as if there is some blocking going on.
>>
>> During the write test i will hit 320MB/s briefly, then 0MB/s as disk
>> access time shoot to over 3000ms, then back to 320MB/s. It averages
>> out
>> to about 110MB/s afterwards.
>>
>> Gluster version is 3.12.15 ovirt is 4.2.7.5
>>
>> Any ideas on what i could tune to eliminate or minimize that blocking?
>> _______________________________________________
>> Users mailing list -- users@ovirt.org
>> To unsubscribe send an email to users-leave@ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z7F72BKYKAGICERZETSA4KCLQYR3AORR/
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/FMB6NCNJL2WKEDWPAM4OJIRF2GIDJUUE/

Just the vdsm defaults

vm.dirty_ratio = 5
vm.dirty_background_ratio = 2

these boxes only have 8gb of ram as well, so those percentages should be
super small.