Im in the middle of a priority issue right now, so cant take time out to rerun the bench, but...
Usually in that kind of situation, if you dont turn on sync-to-disk on every write, you get benchmarks that are artificially HIGH.
Forcing O_DIRECT slows throughput down.
Dont you think the results are bad enough already? :-}

----- Original Message -----
From: "Stefan Hajnoczi" <stefanha@redhat.com>
To: "Philip Brown" <pbrown@medata.com>
Cc: "Nir Soffer" <nsoffer@redhat.com>, "users" <users@ovirt.org>, "qemu-block" <qemu-block@nongnu.org>, "Paolo Bonzini" <pbonzini@redhat.com>, "Sergio Lopez Pascual" <slp@redhat.com>, "Mordechai Lehrer" <mlehrer@redhat.com>, "Kevin Wolf" <kwolf@redhat.com>
Sent: Thursday, July 23, 2020 6:09:39 AM
Subject: Re: [BULK]  Re: [ovirt-users] very very bad iscsi performance


Hi,
At first glance it appears that the filebench OLTP workload does not use
O_DIRECT, so this isn't a measurement of pure disk I/O performance:
https://github.com/filebench/filebench/blob/master/workloads/oltp.f

If you suspect that disk performance is the issue please run a benchmark
that bypasses the page cache using O_DIRECT.

The fio setting is direct=1.

Here is an example fio job for 70% read/30% write 4KB random I/O:

  [global]
  filename=/path/to/device
  runtime=120
  ioengine=libaio
  direct=1
  ramp_time=10            # start measuring after warm-up time

  [read]
  readwrite=randrw
  rwmixread=70
  rwmixwrite=30
  iodepth=64
  blocksize=4k

(Based on https://blog.vmsplice.net/2017/11/common-disk-benchmarking-mistakes.html)

Stefan