What do you mean by "small block sizes"?
inside a VM, or directly on the mounted glusterfs; dd if=/dev/zero of=tmpfile bs=1M count=100 oflag=direct of course, a terrible way to write data, but also things like compiling software inside one of the VMs was terrible slow, 5-10x slower than hardware, consisting of almost only idling. Uploading disk images never got above 30MB/s. (and I did try all options i could find; using upload_disk.py on one of the hosts, even through a unix socket or with -d option, tweaking buffer size, all of which made no difference). Adding an NFS volume and uploading to it I reach +200MB/s. I tried tuning a few parameters on glusterfs but saw no improvements until I got to network.remote-dio, which made everything listed above really fast.
Note that network.remote-dio is not the recommended configuration for ovirt, in particular if on hyperconverge setup when it can be harmful by delaying sanlock I/O.
https://github.com/oVirt/ovirt-site/blob/4a9b28aac48870343c5ea4d1e83a63c1... (Patch in discussion)
Oh, I had seen this page, thanks. Is "remote-dio=enabled" harmful as in things breaking, or just worse performance? I was a bit reluctant to turn it on, but after seeing it was part of the virt group I thought it must have been safe. Perhaps some of the other options like "performance.strict-o-direct on" would solve my performance issues in a nicer way (I will test it out first thing on monday) Thanks (I'm not much of a filesystem guy, thanks for putting up with my ignorance)