Check and tune the following: - I/O scheduler on the host (usually noop/none are good for writes, (mq-)deadline for reads) - CPU cstates - Tuned profile , there are some 'dirty' settings that will avoid I/O locks - MTU size and tcp offloading (some users report enabled is better, others feel disabled brings better performance) -I/O scheduler in VMs is best to be noop/none as we already reorder our I/O on Hypervisour level - with oVirt 4.4 Hugepages for VMs should be fixed, so you can try - Use high-performance VMs - Using libgfapi is better than FUSE, so if you are OK with the limitations that it imposes -> you can swith to libgfapi instead of FUSE (requires VM power off and power on to switch) Best Regards, Strahil Nikolov На 19 август 2020 г. 21:26:44 GMT+03:00, info--- via Users <users(a)ovirt.org&gt; написа: >Hello, > >I'm running a home setup with 3 nodes and 2 SATA SSDs per node. >As storage I'm running glusterfs and 40GBit/s links. > >Software Version:4.3.9.4-1.el7 > >I've a lot of I/O Wait on the nodes (20%) and on the VMs (50%). > >gluster volume top vmstore write-perf bs 2014 count 1024 | grep Through >Throughput 635.54 MBps time 0.0032 secs >Throughput 614.89 MBps time 0.0034 secs >Throughput 622.31 MBps time 0.0033 secs >Throughput 643.07 MBps time 0.0032 secs >Throughput 621.75 MBps time 0.0033 secs >Throughput 609.26 MBps time 0.0034 secs > >gluster volume top vmstore read-perf bs 2014 count 1024 | grep Through >Throughput 1274.62 MBps time 0.0016 secs >Throughput 1320.32 MBps time 0.0016 secs >Throughput 1203.93 MBps time 0.0017 secs >Throughput 1293.81 MBps time 0.0016 secs >Throughput 1213.14 MBps time 0.0017 secs >Throughput 1193.48 MBps time 0.0017 secs > >Volume Name: vmstore >Type: Distributed-Replicate >Volume ID: 195e2a05-9667-4b8b-b0b7-82294631de50 >Status: Started >Snapshot Count: 0 >Number of Bricks: 2 x 3 = 6 >Transport-type: tcp >Bricks: >Brick1: 10.9.9.101:/gluster_bricks/vmstore/vmstore >Brick2: 10.9.9.102:/gluster_bricks/vmstore/vmstore >Brick3: 10.9.9.103:/gluster_bricks/vmstore/vmstore >Brick4: 10.9.9.101:/gluster_bricks/S4CYNF0M219849L/S4CYNF0M219849L >Brick5: 10.9.9.102:/gluster_bricks/S4CYNF0M219836L/S4CYNF0M219836L >Brick6: 10.9.9.103:/gluster_bricks/S4CYNF0M219801Y/S4CYNF0M219801Y >Options Reconfigured: >performance.client-io-threads: on >nfs.disable: on >transport.address-family: inet >performance.strict-o-direct: on >performance.quick-read: off >performance.read-ahead: off >performance.io-cache: off >performance.low-prio-threads: 32 >network.remote-dio: off >cluster.eager-lock: enable >cluster.quorum-type: auto >cluster.server-quorum-type: server >cluster.data-self-heal-algorithm: full >cluster.locking-scheme: granular >cluster.shd-max-threads: 8 >cluster.shd-wait-qlength: 10000 >features.shard: on >user.cifs: off >cluster.choose-local: off >client.event-threads: 4 >server.event-threads: 4 >network.ping-timeout: 30 >storage.owner-uid: 36 >storage.owner-gid: 36 >cluster.granular-entry-heal: enable > >Please help me to analyse the root cause. > >Many thanks >Metz >_______________________________________________ >Users mailing list -- users(a)ovirt.org >To unsubscribe send an email to users-leave(a)ovirt.org >Privacy Statement: https://www.ovirt.org/privacy-policy.html >oVirt Code of Conduct: >https://www.ovirt.org/community/about/community-guidelines/ >List Archives: >https://lists.ovirt.org/archives/list/users@ovirt.org/message/TSACLDXFZXD5XV6WQ5GPJDBJQBNAUC7P/

Additional Info: I've running Nextcloud on a VM. - to sync (download) 1 file to the client with 300 MB is fast - to sync (download) 300 files to the client with in total 1 MB is very slow

Thank you. They are all Thin. I have something to do at the weekend and change them to Preallocated. Looks like stopping the VM. Downloading and uploading the disk to change the format.

Thank you. ping -M do -s 8972 another-gluster-node is working. The rest I'll check on the weekend.

Libgfapi bypasses the context switching from User space to kernel to user space (FUSE) , so it gets better performance. I can't find the previous communication ,so can you share your volume settings again ? Best Regards, Strahil Nikolov В неделя, 23 август 2020 г., 21:45:22 Гринуич+3, info--- via Users <users(a)ovirt.org&gt; написа: Setting cluster.choose-local to on helped a lot to improve the read performance. Write performance still bad. Am I right that this look then more like a glusterfs issue and not something what need to be changed on ovirt (libgfapi) or on VMs. Changing tcp offloading did not make any difference. - ethtool --offload enp1s0f0 rx off tx off - ethtool --offload enp1s0f0 rx on tx on MTU 9000 is fine and ping is working. - ping -M do -s 8972 another-gluster-node _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/23FIY4DLT5C...

On Wed, Aug 26, 2020 at 8:19 PM info--- via Users <users(a)ovirt.org&gt; wrote: Latest status I remember was this one reported by me at 4.3 RC2 time last year: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KHWQA72JZVG... and I think nothing changed in the mean time, neither for 4.3 nor for 4.4. But not directly tested. Someone reported 4x-5x improvements using libgfapi but initially there were some blockers related to snapshots and HA if I remember correctly and so disabled. My idea is that all previous problems in upstream qemu and libvirtd pieces are now solved since many months but developers didn't consider implementing them due to not so big improvements detected in their tests and/or sufficient time to dedicate to fix and/or other priorities. Some bugzilla referred, to dig into if interested: https://bugzilla.redhat.com/show_bug.cgi?id=1484227 https://bugzilla.redhat.com/show_bug.cgi?id=1465810 https://bugzilla.redhat.com/show_bug.cgi?id=1633642 HIH revamping the interesting topic. Gianluca

Thank you. Reboot of the engine and afterwards the backup server helped :-) Before the change -> 53 minutes for 10% and the restore broke at 98% ... - [2020-08-26:20:45:30] Restore Imaging: 1/1 - 10% Imaging in progress.... - [2020-08-26:20:41:22] Restore Imaging: 1/1 - 9% Imaging in progress... - [2020-08-26:20:37:33] Restore Imaging: 1/1 - 8% Imaging in progress. - [2020-08-26:20:32:04] Restore Imaging: 1/1 - 7% Imaging in progress... - [2020-08-26:20:25:38] Restore Imaging: 1/1 - 6% Imaging in progress... - [2020-08-26:20:21:34] Restore Imaging: 1/1 - 5% Imaging in progress... - [2020-08-26:20:18:57] Restore Imaging: 1/1 - 3% Imaging in progress. - [2020-08-26:20:03:15] Restore Imaging: 1/1 - 2% Imaging in progress.... - [2020-08-26:19:57:10] Restore Imaging: 1/1 - 1% Imaging in progress..... - [2020-08-26:19:53:20] Restore Imaging: 1/1 - 0% Imaging in progress........... - [2020-08-26:19:53:17] Restore Imaging: 1/1 - 0% Started Imaging Disk1.img.lzo to /dev/sdb After the change -> 10 minutes for 10% - [2020-08-27:22:08:59] Restore Imaging: 1/1 - 10% Imaging in progress... - [2020-08-27:22:08:02] Restore Imaging: 1/1 - 9% Imaging in progress... - [2020-08-27:22:07:05] Restore Imaging: 1/1 - 8% Imaging in progress... - [2020-08-27:22:05:34] Restore Imaging: 1/1 - 7% Imaging in progress... - [2020-08-27:22:04:43] Restore Imaging: 1/1 - 6% Imaging in progress...... - [2020-08-27:22:03:34] Restore Imaging: 1/1 - 5% Imaging in progress.......... - [2020-08-27:22:02:27] Restore Imaging: 1/1 - 4% Imaging in progress............. - [2020-08-27:22:01:39] Restore Imaging: 1/1 - 3% Imaging in progress........ - [2020-08-27:22:00:16] Restore Imaging: 1/1 - 2% Imaging in progress........... - [2020-08-27:21:59:24] Restore Imaging: 1/1 - 1% Imaging in progress........ - [2020-08-27:21:58:16] Restore Imaging: 1/1 - 0% Imaging in progress.......... - [2020-08-27:21:58:13] Restore Imaging: 1/1 - 0% Started Imaging Disk1.img.lzo to /dev/sdb Should I revert some of my previous changes? Reduce the write window size? - gluster volume set vmstore performance.read-ahead on - gluster volume set vmstore performance.stat-prefetch on - gluster volume set vmstore performance.write-behind-window-size 64MB - gluster volume set vmstore performance.flush-behind on - gluster volume set vmstore cluster.choose-local on Now I will reboot the other VMs and go on with moving from thin provisioned disk to preallocated.