Hello, I'm running a home setup with 3 nodes and 2 SATA SSDs per node. As storage I'm running glusterfs and 40GBit/s links. Software Version:4.3.9.4-1.el7 I've a lot of I/O Wait on the nodes (20%) and on the VMs (50%). gluster volume top vmstore write-perf bs 2014 count 1024 | grep Through Throughput 635.54 MBps time 0.0032 secs Throughput 614.89 MBps time 0.0034 secs Throughput 622.31 MBps time 0.0033 secs Throughput 643.07 MBps time 0.0032 secs Throughput 621.75 MBps time 0.0033 secs Throughput 609.26 MBps time 0.0034 secs gluster volume top vmstore read-perf bs 2014 count 1024 | grep Through Throughput 1274.62 MBps time 0.0016 secs Throughput 1320.32 MBps time 0.0016 secs Throughput 1203.93 MBps time 0.0017 secs Throughput 1293.81 MBps time 0.0016 secs Throughput 1213.14 MBps time 0.0017 secs Throughput 1193.48 MBps time 0.0017 secs Volume Name: vmstore Type: Distributed-Replicate Volume ID: 195e2a05-9667-4b8b-b0b7-82294631de50 Status: Started Snapshot Count: 0 Number of Bricks: 2 x 3 = 6 Transport-type: tcp Bricks: Brick1: 10.9.9.101:/gluster_bricks/vmstore/vmstore Brick2: 10.9.9.102:/gluster_bricks/vmstore/vmstore Brick3: 10.9.9.103:/gluster_bricks/vmstore/vmstore Brick4: 10.9.9.101:/gluster_bricks/S4CYNF0M219849L/S4CYNF0M219849L Brick5: 10.9.9.102:/gluster_bricks/S4CYNF0M219836L/S4CYNF0M219836L Brick6: 10.9.9.103:/gluster_bricks/S4CYNF0M219801Y/S4CYNF0M219801Y Options Reconfigured: performance.client-io-threads: on nfs.disable: on transport.address-family: inet performance.strict-o-direct: on performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.low-prio-threads: 32 network.remote-dio: off cluster.eager-lock: enable cluster.quorum-type: auto cluster.server-quorum-type: server cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-max-threads: 8 cluster.shd-wait-qlength: 10000 features.shard: on user.cifs: off cluster.choose-local: off client.event-threads: 4 server.event-threads: 4 network.ping-timeout: 30 storage.owner-uid: 36 storage.owner-gid: 36 cluster.granular-entry-heal: enable Please help me to analyse the root cause. Many thanks Metz
Check and tune the following: - I/O scheduler on the host (usually noop/none are good for writes, (mq-)deadline for reads) - CPU cstates - Tuned profile , there are some 'dirty' settings that will avoid I/O locks - MTU size and tcp offloading (some users report enabled is better, others feel disabled brings better performance) -I/O scheduler in VMs is best to be noop/none as we already reorder our I/O on Hypervisour level - with oVirt 4.4 Hugepages for VMs should be fixed, so you can try - Use high-performance VMs - Using libgfapi is better than FUSE, so if you are OK with the limitations that it imposes -> you can swith to libgfapi instead of FUSE (requires VM power off and power on to switch) Best Regards, Strahil Nikolov На 19 август 2020 г. 21:26:44 GMT+03:00, info--- via Users <users@ovirt.org> написа:
Hello,
I'm running a home setup with 3 nodes and 2 SATA SSDs per node. As storage I'm running glusterfs and 40GBit/s links.
Software Version:4.3.9.4-1.el7
I've a lot of I/O Wait on the nodes (20%) and on the VMs (50%).
gluster volume top vmstore write-perf bs 2014 count 1024 | grep Through Throughput 635.54 MBps time 0.0032 secs Throughput 614.89 MBps time 0.0034 secs Throughput 622.31 MBps time 0.0033 secs Throughput 643.07 MBps time 0.0032 secs Throughput 621.75 MBps time 0.0033 secs Throughput 609.26 MBps time 0.0034 secs
gluster volume top vmstore read-perf bs 2014 count 1024 | grep Through Throughput 1274.62 MBps time 0.0016 secs Throughput 1320.32 MBps time 0.0016 secs Throughput 1203.93 MBps time 0.0017 secs Throughput 1293.81 MBps time 0.0016 secs Throughput 1213.14 MBps time 0.0017 secs Throughput 1193.48 MBps time 0.0017 secs
Volume Name: vmstore Type: Distributed-Replicate Volume ID: 195e2a05-9667-4b8b-b0b7-82294631de50 Status: Started Snapshot Count: 0 Number of Bricks: 2 x 3 = 6 Transport-type: tcp Bricks: Brick1: 10.9.9.101:/gluster_bricks/vmstore/vmstore Brick2: 10.9.9.102:/gluster_bricks/vmstore/vmstore Brick3: 10.9.9.103:/gluster_bricks/vmstore/vmstore Brick4: 10.9.9.101:/gluster_bricks/S4CYNF0M219849L/S4CYNF0M219849L Brick5: 10.9.9.102:/gluster_bricks/S4CYNF0M219836L/S4CYNF0M219836L Brick6: 10.9.9.103:/gluster_bricks/S4CYNF0M219801Y/S4CYNF0M219801Y Options Reconfigured: performance.client-io-threads: on nfs.disable: on transport.address-family: inet performance.strict-o-direct: on performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.low-prio-threads: 32 network.remote-dio: off cluster.eager-lock: enable cluster.quorum-type: auto cluster.server-quorum-type: server cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-max-threads: 8 cluster.shd-wait-qlength: 10000 features.shard: on user.cifs: off cluster.choose-local: off client.event-threads: 4 server.event-threads: 4 network.ping-timeout: 30 storage.owner-uid: 36 storage.owner-gid: 36 cluster.granular-entry-heal: enable
Please help me to analyse the root cause.
Many thanks Metz _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TSACLDXFZXD5XV...
Thank you for the quick reply. - I/O scheduler hosts -> changed echo noop > /sys/block/sdb/queue/scheduler echo noop > /sys/block/sdc/queue/scheduler - CPU states -> can you explain this a bit more? cat /dev/cpu_dma_latency F hexdump -C /dev/cpu_dma_latency 00000000 46 00 00 00 |F...| 00000004 - Tuned profile -> can you explain this a bit more? - MTU was already set to 9000 -> can you explain a bit more tcp-offloading and how I change the settings? ethtool -k enp1s0f0 | grep offload tcp-segmentation-offload: on udp-fragmentation-offload: off [fixed] generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: on tx-vlan-offload: on l2-fwd-offload: off [fixed] hw-tc-offload: off rx-udp_tunnel-port-offload: on - I/O scheduler VM had already none cat /sys/block/vda/queue/scheduler none - oVirt 4.4 -> thanks. Was not aware that the new version is final. I will check and upgrade. - Use High Performance VMs -> I got following message -> So I need to do changes also on the VM? The following recommended settings for running the High Performance type with the optimal configuration were not detected. Please consider manually changing of the following before applying: CPU PINNING: VIRTUAL NUMA AND NUMA PINNING: HUGE PAGES: KERNEL SAME PAGE MERGING (KSM): - libgfapi -> I will execute on the engine following command and then it will change all VMs after reboot? engine-config -s LibgfApiSupported=true --cver4.2 - changed values (from your other threat): gluster volume set vmstore performance.read-ahead on gluster volume set vmstore performance.stat-prefetch on gluster volume set vmstore performance.write-behind-window-size 64MB gluster volume set vmstore performance.flush-behind on - already configured: performance.client-io-threads: on Best Regards Dirk
Additional Info: I've running Nextcloud on a VM. - to sync (download) 1 file to the client with 300 MB is fast - to sync (download) 300 files to the client with in total 1 MB is very slow
Are you using fully allocated VM disks ? На 20 август 2020 г. 0:53:40 GMT+03:00, info--- via Users <users@ovirt.org> написа:
Additional Info: I've running Nextcloud on a VM. - to sync (download) 1 file to the client with 300 MB is fast - to sync (download) 300 files to the client with in total 1 MB is very slow _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LZRN2NRC7Y57XQ...
Thank you. They are all Thin. I have something to do at the weekend and change them to Preallocated. Looks like stopping the VM. Downloading and uploading the disk to change the format.
На 19 август 2020 г. 22:39:22 GMT+03:00, info--- via Users <users@ovirt.org> написа:
Thank you for the quick reply.
- I/O scheduler hosts -> changed echo noop > /sys/block/sdb/queue/scheduler echo noop > /sys/block/sdc/queue/scheduler
On reboot it will be reverted. Test this way and if you notice improvement use udev rules. Keep in mind that multipath devices also have a scheduler.
- CPU states -> can you explain this a bit more?
On Intel CPUs you can use these kernel parameters (check them online as I type them by memory) processor.max_cstate=1 intel_idle.max_cstate=0
cat /dev/cpu_dma_latency F hexdump -C /dev/cpu_dma_latency 00000000 46 00 00 00 |F...| 00000004
- Tuned profile -> can you explain this a bit more?
Download ftp://ftp.redhat.com/redhat/linux/enterprise/7Server/en/RHS/SRPMS/redhat-storage-server-3.5.0.0-6.el7rhgs.src.rpm and extract the srpm. Inside it you will find 2 dirs: 'rhgs-random-io' &'rhgs-sequential-io'. Open the tuned.conf files inside them and check the 'dirty' sysctl tunings. One is for random I/O (usually the Vm workload is random I/O) and another for sequential. You can set the 'dirty' settings in your sysctl.conf or your custom tuned profile.
- MTU was already set to 9000 -> can you explain a bit more
Have you tested via : ping -M do -s 8972 another-gluster-node
tcp-offloading and how I change the settings?
You can google it. Usually ethtool is your best friend.
ethtool -k enp1s0f0 | grep offload tcp-segmentation-offload: on udp-fragmentation-offload: off [fixed] generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: on tx-vlan-offload: on l2-fwd-offload: off [fixed] hw-tc-offload: off rx-udp_tunnel-port-offload: on
- I/O scheduler VM had already none cat /sys/block/vda/queue/scheduler none
Consider using udev here too.
- oVirt 4.4 -> thanks. Was not aware that the new version is final. I will check and upgrade.
Upgrades are not so easy as you need to: A) Upgrade OS to EL8 B) Upgrade ovirt Test in advance before proceeding on prod !
- Use High Performance VMs -> I got following message -> So I need to do changes also on the VM? The following recommended settings for running the High Performance type with the optimal configuration were not detected. Please consider manually changing of the following before applying: CPU PINNING: VIRTUAL NUMA AND NUMA PINNING: HUGE PAGES: KERNEL SAME PAGE MERGING (KSM):
- libgfapi -> I will execute on the engine following command and then it will change all VMs after reboot? engine-config -s LibgfApiSupported=true --cver4.2
VM reboot won't reread the VM config. You will need an 'off' and 'on' action for each VM. I'm not sure about the command. I haven't used libgfapi for eons.
- changed values (from your other threat): gluster volume set vmstore performance.read-ahead on gluster volume set vmstore performance.stat-prefetch on gluster volume set vmstore performance.write-behind-window-size 64MB gluster volume set vmstore performance.flush-behind on
- already configured: performance.client-io-threads: on
It's easier to use: gluster volume set VOLUME group virt WARNING: This will enable sharding and SHARDING cannot be disabled in an easy way. NEVER EVER DISABLE SHARDING. You can check the 'virt' group of settings on your gluster nodes in /var/lib/glusterd/group.
Best Regards Dirk _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/VNE4O46SKA5KAL...
Thank you. ping -M do -s 8972 another-gluster-node is working. The rest I'll check on the weekend.
Setting cluster.choose-local to on helped a lot to improve the read performance. Write performance still bad. Am I right that this look then more like a glusterfs issue and not something what need to be changed on ovirt (libgfapi) or on VMs. Changing tcp offloading did not make any difference. - ethtool --offload enp1s0f0 rx off tx off - ethtool --offload enp1s0f0 rx on tx on MTU 9000 is fine and ping is working. - ping -M do -s 8972 another-gluster-node
Libgfapi bypasses the context switching from User space to kernel to user space (FUSE) , so it gets better performance. I can't find the previous communication ,so can you share your volume settings again ? Best Regards, Strahil Nikolov В неделя, 23 август 2020 г., 21:45:22 Гринуич+3, info--- via Users <users@ovirt.org> написа: Setting cluster.choose-local to on helped a lot to improve the read performance. Write performance still bad. Am I right that this look then more like a glusterfs issue and not something what need to be changed on ovirt (libgfapi) or on VMs. Changing tcp offloading did not make any difference. - ethtool --offload enp1s0f0 rx off tx off - ethtool --offload enp1s0f0 rx on tx on MTU 9000 is fine and ping is working. - ping -M do -s 8972 another-gluster-node _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/23FIY4DLT5CBZF...
I enabled libgfapi and powered off / on the VM. - engine-config --all - LibgfApiSupported: true version: 4.3 How can I see that this is active on the VM? The disk looks the same like before. - virsh dumpxml 15 <disk type='file' device='disk' snapshot='no'> <driver name='qemu' type='raw' cache='none' error_policy='stop' io='threads'/> <source file='/rhev/data-center/mnt/glusterSD/10.9.9.101:_vmstore/f2c621de-42bf-4dbf-920c-adf4506b786d/images/1e231e3e-d98c-491a-9236-907814d4837/c755aaa3-7d3d-4c0d-8184-c6aae37229ba'> <seclabel model='dac' relabel='no'/> </source> <backingStore/> <target dev='sdc' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='3'/> </disk> Here is the Volume setup: Volume Name: vmstore Type: Distributed-Replicate Volume ID: 195e2a05-9667-4b8b-b0b7-82294631de50 Status: Started Snapshot Count: 0 Number of Bricks: 2 x 3 = 6 Transport-type: tcp Bricks: Brick1: 10.9.9.101:/gluster_bricks/vmstore/vmstore Brick2: 10.9.9.102:/gluster_bricks/vmstore/vmstore Brick3: 10.9.9.103:/gluster_bricks/vmstore/vmstore Brick4: 10.9.9.101:/gluster_bricks/S4CYNF0M219849L/S4CYNF0M219849L Brick5: 10.9.9.102:/gluster_bricks/S4CYNF0M219836L/S4CYNF0M219836L Brick6: 10.9.9.103:/gluster_bricks/S4CYNF0M219801Y/S4CYNF0M219801Y Options Reconfigured: performance.write-behind-window-size: 64MB performance.flush-behind: on performance.stat-prefetch: on performance.client-io-threads: on nfs.disable: on transport.address-family: inet performance.strict-o-direct: on performance.quick-read: off performance.read-ahead: on performance.io-cache: off performance.low-prio-threads: 32 network.remote-dio: off cluster.eager-lock: enable cluster.quorum-type: auto cluster.server-quorum-type: server cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-max-threads: 8 cluster.shd-wait-qlength: 10000 features.shard: on user.cifs: off cluster.choose-local: on client.event-threads: 4 server.event-threads: 4 network.ping-timeout: 30 storage.owner-uid: 36 storage.owner-gid: 36 cluster.granular-entry-heal: enable Thank you for your support.
On Wed, Aug 26, 2020 at 8:19 PM info--- via Users <users@ovirt.org> wrote:
I enabled libgfapi and powered off / on the VM.
- engine-config --all - LibgfApiSupported: true version: 4.3
How can I see that this is active on the VM? The disk looks the same like before.
- virsh dumpxml 15 <disk type='file' device='disk' snapshot='no'> <driver name='qemu' type='raw' cache='none' error_policy='stop' io='threads'/> <source file='/rhev/data-center/mnt/glusterSD/10.9.9.101: _vmstore/f2c621de-42bf-4dbf-920c-adf4506b786d/images/1e231e3e-d98c-491a-9236-907814d4837/c755aaa3-7d3d-4c0d-8184-c6aae37229ba'> <seclabel model='dac' relabel='no'/> </source> <backingStore/> <target dev='sdc' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='3'/> </disk>
Latest status I remember was this one reported by me at 4.3 RC2 time last year: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KHWQA72JZVGLVX... and I think nothing changed in the mean time, neither for 4.3 nor for 4.4. But not directly tested. Someone reported 4x-5x improvements using libgfapi but initially there were some blockers related to snapshots and HA if I remember correctly and so disabled. My idea is that all previous problems in upstream qemu and libvirtd pieces are now solved since many months but developers didn't consider implementing them due to not so big improvements detected in their tests and/or sufficient time to dedicate to fix and/or other priorities. Some bugzilla referred, to dig into if interested: https://bugzilla.redhat.com/show_bug.cgi?id=1484227 https://bugzilla.redhat.com/show_bug.cgi?id=1465810 https://bugzilla.redhat.com/show_bug.cgi?id=1633642 HIH revamping the interesting topic. Gianluca
Looks like you’ve got a posix or nfs mount there? Is your gluster storage domain of type GlusterFS? And make sure you restarted the ovirt-engine after enabling LibfgApiSupported, before stopping and restarting the vm. An active libgf mount looks like: <disk type='network' device='disk' snapshot='no'> <driver name='qemu' type='raw' cache='none' error_policy='stop' io='native'/> <source protocol='gluster' name='gv1/a10b0cc4-b6ec-4b26-bef0-316e0e7bdcac/images/7e8b2be0-8206-4090-8fa4-c87d81fb5ea0/6b8c2ff9-15f3-47a9-af5d-bb7dea31b59b'> <host name='gstor0' port='24007'/> </source>
On Aug 26, 2020, at 1:12 PM, info--- via Users <users@ovirt.org> wrote:
I enabled libgfapi and powered off / on the VM.
- engine-config --all - LibgfApiSupported: true version: 4.3
How can I see that this is active on the VM? The disk looks the same like before.
- virsh dumpxml 15 <disk type='file' device='disk' snapshot='no'> <driver name='qemu' type='raw' cache='none' error_policy='stop' io='threads'/> <source file='/rhev/data-center/mnt/glusterSD/10.9.9.101:_vmstore/f2c621de-42bf-4dbf-920c-adf4506b786d/images/1e231e3e-d98c-491a-9236-907814d4837/c755aaa3-7d3d-4c0d-8184-c6aae37229ba'> <seclabel model='dac' relabel='no'/> </source> <backingStore/> <target dev='sdc' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='3'/> </disk>
Here is the Volume setup:
Volume Name: vmstore Type: Distributed-Replicate Volume ID: 195e2a05-9667-4b8b-b0b7-82294631de50 Status: Started Snapshot Count: 0 Number of Bricks: 2 x 3 = 6 Transport-type: tcp Bricks: Brick1: 10.9.9.101:/gluster_bricks/vmstore/vmstore Brick2: 10.9.9.102:/gluster_bricks/vmstore/vmstore Brick3: 10.9.9.103:/gluster_bricks/vmstore/vmstore Brick4: 10.9.9.101:/gluster_bricks/S4CYNF0M219849L/S4CYNF0M219849L Brick5: 10.9.9.102:/gluster_bricks/S4CYNF0M219836L/S4CYNF0M219836L Brick6: 10.9.9.103:/gluster_bricks/S4CYNF0M219801Y/S4CYNF0M219801Y Options Reconfigured: performance.write-behind-window-size: 64MB performance.flush-behind: on performance.stat-prefetch: on performance.client-io-threads: on nfs.disable: on transport.address-family: inet performance.strict-o-direct: on performance.quick-read: off performance.read-ahead: on performance.io-cache: off performance.low-prio-threads: 32 network.remote-dio: off cluster.eager-lock: enable cluster.quorum-type: auto cluster.server-quorum-type: server cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-max-threads: 8 cluster.shd-wait-qlength: 10000 features.shard: on user.cifs: off cluster.choose-local: on client.event-threads: 4 server.event-threads: 4 network.ping-timeout: 30 storage.owner-uid: 36 storage.owner-gid: 36 cluster.granular-entry-heal: enable
Thank you for your support. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z3JD7MQV2PIQZJ...
Thank you. Reboot of the engine and afterwards the backup server helped :-) Before the change -> 53 minutes for 10% and the restore broke at 98% ... - [2020-08-26:20:45:30] Restore Imaging: 1/1 - 10% Imaging in progress.... - [2020-08-26:20:41:22] Restore Imaging: 1/1 - 9% Imaging in progress... - [2020-08-26:20:37:33] Restore Imaging: 1/1 - 8% Imaging in progress. - [2020-08-26:20:32:04] Restore Imaging: 1/1 - 7% Imaging in progress... - [2020-08-26:20:25:38] Restore Imaging: 1/1 - 6% Imaging in progress... - [2020-08-26:20:21:34] Restore Imaging: 1/1 - 5% Imaging in progress... - [2020-08-26:20:18:57] Restore Imaging: 1/1 - 3% Imaging in progress. - [2020-08-26:20:03:15] Restore Imaging: 1/1 - 2% Imaging in progress.... - [2020-08-26:19:57:10] Restore Imaging: 1/1 - 1% Imaging in progress..... - [2020-08-26:19:53:20] Restore Imaging: 1/1 - 0% Imaging in progress........... - [2020-08-26:19:53:17] Restore Imaging: 1/1 - 0% Started Imaging Disk1.img.lzo to /dev/sdb After the change -> 10 minutes for 10% - [2020-08-27:22:08:59] Restore Imaging: 1/1 - 10% Imaging in progress... - [2020-08-27:22:08:02] Restore Imaging: 1/1 - 9% Imaging in progress... - [2020-08-27:22:07:05] Restore Imaging: 1/1 - 8% Imaging in progress... - [2020-08-27:22:05:34] Restore Imaging: 1/1 - 7% Imaging in progress... - [2020-08-27:22:04:43] Restore Imaging: 1/1 - 6% Imaging in progress...... - [2020-08-27:22:03:34] Restore Imaging: 1/1 - 5% Imaging in progress.......... - [2020-08-27:22:02:27] Restore Imaging: 1/1 - 4% Imaging in progress............. - [2020-08-27:22:01:39] Restore Imaging: 1/1 - 3% Imaging in progress........ - [2020-08-27:22:00:16] Restore Imaging: 1/1 - 2% Imaging in progress........... - [2020-08-27:21:59:24] Restore Imaging: 1/1 - 1% Imaging in progress........ - [2020-08-27:21:58:16] Restore Imaging: 1/1 - 0% Imaging in progress.......... - [2020-08-27:21:58:13] Restore Imaging: 1/1 - 0% Started Imaging Disk1.img.lzo to /dev/sdb Should I revert some of my previous changes? Reduce the write window size? - gluster volume set vmstore performance.read-ahead on - gluster volume set vmstore performance.stat-prefetch on - gluster volume set vmstore performance.write-behind-window-size 64MB - gluster volume set vmstore performance.flush-behind on - gluster volume set vmstore cluster.choose-local on Now I will reboot the other VMs and go on with moving from thin provisioned disk to preallocated.
See below:
On Aug 27, 2020, at 3:19 PM, info--- via Users <users@ovirt.org> wrote:
Thank you. Reboot of the engine and afterwards the backup server helped :-)
Good deal.
Should I revert some of my previous changes? Reduce the write window size? - gluster volume set vmstore performance.read-ahead on - gluster volume set vmstore performance.stat-prefetch on - gluster volume set vmstore performance.write-behind-window-size 64MB - gluster volume set vmstore performance.flush-behind on - gluster volume set vmstore cluster.choose-local on
I find these tend to be workload & system dependent, you’re best bet is to benchmark and test yourself on your system. I’m using these on my volumes: performance.stat-prefetch: on performance.read-ahead: off performance.write-behind-window-size: 64MB
participants (4)
-
Darrell Budic -
Gianluca Cecchi -
info＠dsdm.ch -
Strahil Nikolov