On Sun, Sep 15, 2019 at 8:37 PM Mikael Öhman <micketeer(a)gmail.com> wrote:
> What do you mean by "small block sizes"?
inside a VM, or directly on the mounted glusterfs;
dd if=/dev/zero of=tmpfile bs=1M count=100 oflag=direct
of course, a terrible way to write data, but also things like compiling
software inside one of the VMs was terrible slow, 5-10x slower than
hardware, consisting of almost only idling.
Uploading disk images never got above 30MB/s.
(and I did try all options i could find; using upload_disk.py on one of
the hosts, even through a unix socket or with -d option, tweaking buffer
size, all of which made no difference).
Adding an NFS volume and uploading to it I reach +200MB/s.
I tried tuning a few parameters on glusterfs but saw no improvements until
I got to network.remote-dio, which made everything listed above really fast.
> Note that network.remote-dio is not the recommended configuration
> for ovirt, in particular if on hyperconverge setup when it can be harmful
> by delaying sanlock I/O.
> (Patch in discussion)
Oh, I had seen this page, thanks. Is "remote-dio=enabled" harmful as in
things breaking, or just worse performance?
I think the issue is delayed writes to storage until you call fsync() on
the node. The
kernel may try to flush too much data, which may cause delays in other
Many years ago we had such issues that cause sanlock failures in renewing
can end in terminating vdsm. This is the reason we always use direct I/O
disks. If you have hypervonvrged setup and use network.remote-dio you may
Sahina worked on a bug that was related to network.remote-dio, I hope she
can add more
I was a bit reluctant to turn it on, but after seeing it was part of
virt group I thought it must have been safe.
I think it should be safe for general virt usage, but you may need to tweak
host settings to avoid large delays when using fsync().
Perhaps some of the other options like
would solve my performance issues in a nicer way (I will test it out first
thing on monday)
This is not likely to improve performance, but it is seems to be required
if you use
network.remote-dio = off. Without it direct I/O does not behave in a
and may cause failures in qemu, vdsm and ovirt-imageio.