> What do you mean by "small block sizes"?
inside a VM, or directly on the mounted glusterfs;
dd if=/dev/zero of=tmpfile bs=1M count=100 oflag=direct
of course, a terrible way to write data, but also things like compiling software inside one of the VMs was terrible slow, 5-10x slower than hardware, consisting of almost only idling.
Uploading disk images never got above 30MB/s.
(and I did try all options i could find; using upload_disk.py on one of the hosts, even through a unix socket or with -d option, tweaking buffer size, all of which made no difference).
Adding an NFS volume and uploading to it I reach +200MB/s.
I tried tuning a few parameters on glusterfs but saw no improvements until I got to network.remote-dio, which made everything listed above really fast.
> Note that network.remote-dio is not the recommended configuration
> for ovirt, in particular if on hyperconverge setup when it can be harmful
> by delaying sanlock I/O.
>
> https://github.com/oVirt/ovirt-site/blob/4a9b28aac48870343c5ea4d1e83a63c1...
> (Patch in discussion)
Oh, I had seen this page, thanks. Is "remote-dio=enabled" harmful as in things breaking, or just worse performance?
I think the issue is delayed writes to storage until you call fsync() on the node. The
kernel may try to flush too much data, which may cause delays in other processes.
Many years ago we had such issues that cause sanlock failures in renewing leases, which
can end in terminating vdsm. This is the reason we always use direct I/O when copying
disks. If you have hypervonvrged setup and use network.remote-dio you may have the
same problems.
Sahina worked on a bug that was related to network.remote-dio, I hope she can add more
details.
I was a bit reluctant to turn it on, but after seeing it was part of the virt group I thought it must have been safe.
I think it should be safe for general virt usage, but you may need to tweak some
host settings to avoid large delays when using fsync().
Perhaps some of the other options like "performance.strict-o-direct on" would solve my performance issues in a nicer way (I will test it out first thing on monday)
This is not likely to improve performance, but it is seems to be required if you use
network.remote-dio = off. Without it direct I/O does not behave in a predictable way
and may cause failures in qemu, vdsm and ovirt-imageio.
Nir