2:26 p.m.
Hello,
I'm running a home setup with 3 nodes and 2 SATA SSDs per node.
As storage I'm running glusterfs and 40GBit/s links.
Software Version:4.3.9.4-1.el7
I've a lot of I/O Wait on the nodes (20%) and on the VMs (50%).
gluster volume top vmstore write-perf bs 2014 count 1024 | grep Through
Throughput 635.54 MBps time 0.0032 secs
Throughput 614.89 MBps time 0.0034 secs
Throughput 622.31 MBps time 0.0033 secs
Throughput 643.07 MBps time 0.0032 secs
Throughput 621.75 MBps time 0.0033 secs
Throughput 609.26 MBps time 0.0034 secs
gluster volume top vmstore read-perf bs 2014 count 1024 | grep Through
Throughput 1274.62 MBps time 0.0016 secs
Throughput 1320.32 MBps time 0.0016 secs
Throughput 1203.93 MBps time 0.0017 secs
Throughput 1293.81 MBps time 0.0016 secs
Throughput 1213.14 MBps time 0.0017 secs
Throughput 1193.48 MBps time 0.0017 secs
Volume Name: vmstore
Type: Distributed-Replicate
Volume ID: 195e2a05-9667-4b8b-b0b7-82294631de50
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: 10.9.9.101:/gluster_bricks/vmstore/vmstore
Brick2: 10.9.9.102:/gluster_bricks/vmstore/vmstore
Brick3: 10.9.9.103:/gluster_bricks/vmstore/vmstore
Brick4: 10.9.9.101:/gluster_bricks/S4CYNF0M219849L/S4CYNF0M219849L
Brick5: 10.9.9.102:/gluster_bricks/S4CYNF0M219836L/S4CYNF0M219836L
Brick6: 10.9.9.103:/gluster_bricks/S4CYNF0M219801Y/S4CYNF0M219801Y
Options Reconfigured:
performance.client-io-threads: on
nfs.disable: on
transport.address-family: inet
performance.strict-o-direct: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.low-prio-threads: 32
network.remote-dio: off
cluster.eager-lock: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
features.shard: on
user.cifs: off
cluster.choose-local: off
client.event-threads: 4
server.event-threads: 4
network.ping-timeout: 30
storage.owner-uid: 36
storage.owner-gid: 36
cluster.granular-entry-heal: enable
Please help me to analyse the root cause.
Many thanks
Metz
2:59 p.m.
Check and tune the following:
- I/O scheduler on the host (usually noop/none are good for writes, (mq-)deadline
for reads)
- CPU cstates
- Tuned profile , there are some 'dirty' settings that will avoid I/O locks
- MTU size and tcp offloading (some users report enabled is better, others feel
disabled brings better performance)
-I/O scheduler in VMs is best to be noop/none as we already reorder our I/O on Hypervisour
level
- with oVirt 4.4 Hugepages for VMs should be fixed, so you can try
- Use high-performance VMs
- Using libgfapi is better than FUSE, so if you are OK with the limitations that it
imposes -> you can swith to libgfapi instead of FUSE (requires VM power off and power
on to switch)
Best Regards,
Strahil Nikolov
На 19 август 2020 г. 21:26:44 GMT+03:00, info--- via Users <users(a)ovirt.org>
написа:
>Hello,
>
>I'm running a home setup with 3 nodes and 2 SATA SSDs per node.
>As storage I'm running glusterfs and 40GBit/s links.
>
>Software Version:4.3.9.4-1.el7
>
>I've a lot of I/O Wait on the nodes (20%) and on the VMs (50%).
>
>gluster volume top vmstore write-perf bs 2014 count 1024 | grep Through
>Throughput 635.54 MBps time 0.0032 secs
>Throughput 614.89 MBps time 0.0034 secs
>Throughput 622.31 MBps time 0.0033 secs
>Throughput 643.07 MBps time 0.0032 secs
>Throughput 621.75 MBps time 0.0033 secs
>Throughput 609.26 MBps time 0.0034 secs
>
>gluster volume top vmstore read-perf bs 2014 count 1024 | grep Through
>Throughput 1274.62 MBps time 0.0016 secs
>Throughput 1320.32 MBps time 0.0016 secs
>Throughput 1203.93 MBps time 0.0017 secs
>Throughput 1293.81 MBps time 0.0016 secs
>Throughput 1213.14 MBps time 0.0017 secs
>Throughput 1193.48 MBps time 0.0017 secs
>
>Volume Name: vmstore
>Type: Distributed-Replicate
>Volume ID: 195e2a05-9667-4b8b-b0b7-82294631de50
>Status: Started
>Snapshot Count: 0
>Number of Bricks: 2 x 3 = 6
>Transport-type: tcp
>Bricks:
>Brick1: 10.9.9.101:/gluster_bricks/vmstore/vmstore
>Brick2: 10.9.9.102:/gluster_bricks/vmstore/vmstore
>Brick3: 10.9.9.103:/gluster_bricks/vmstore/vmstore
>Brick4: 10.9.9.101:/gluster_bricks/S4CYNF0M219849L/S4CYNF0M219849L
>Brick5: 10.9.9.102:/gluster_bricks/S4CYNF0M219836L/S4CYNF0M219836L
>Brick6: 10.9.9.103:/gluster_bricks/S4CYNF0M219801Y/S4CYNF0M219801Y
>Options Reconfigured:
>performance.client-io-threads: on
>nfs.disable: on
>transport.address-family: inet
>performance.strict-o-direct: on
>performance.quick-read: off
>performance.read-ahead: off
>performance.io-cache: off
>performance.low-prio-threads: 32
>network.remote-dio: off
>cluster.eager-lock: enable
>cluster.quorum-type: auto
>cluster.server-quorum-type: server
>cluster.data-self-heal-algorithm: full
>cluster.locking-scheme: granular
>cluster.shd-max-threads: 8
>cluster.shd-wait-qlength: 10000
>features.shard: on
>user.cifs: off
>cluster.choose-local: off
>client.event-threads: 4
>server.event-threads: 4
>network.ping-timeout: 30
>storage.owner-uid: 36
>storage.owner-gid: 36
>cluster.granular-entry-heal: enable
>
>Please help me to analyse the root cause.
>
>Many thanks
>Metz
>_______________________________________________
>Users mailing list -- users(a)ovirt.org
>To unsubscribe send an email to users-leave(a)ovirt.org
>Privacy Statement: https://www.ovirt.org/privacy-policy.html
>oVirt Code of Conduct:
>https://www.ovirt.org/community/about/community-guidelines/
>List Archives:
>https://lists.ovirt.org/archives/list/users@ovirt.org/message/TSACLDXFZXD5XV6WQ5GPJDBJQBNAUC7P/
3:39 p.m.
Thank you for the quick reply.
- I/O scheduler hosts -> changed
echo noop > /sys/block/sdb/queue/scheduler
echo noop > /sys/block/sdc/queue/scheduler
- CPU states -> can you explain this a bit more?
cat /dev/cpu_dma_latency
F
hexdump -C /dev/cpu_dma_latency
00000000 46 00 00 00 |F...|
00000004
- Tuned profile -> can you explain this a bit more?
- MTU was already set to 9000 -> can you explain a bit more tcp-offloading and how I
change the settings?
ethtool -k enp1s0f0 | grep offload
tcp-segmentation-offload: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
l2-fwd-offload: off [fixed]
hw-tc-offload: off
rx-udp_tunnel-port-offload: on
- I/O scheduler VM had already none
cat /sys/block/vda/queue/scheduler
none
- oVirt 4.4 -> thanks. Was not aware that the new version is final. I will check and
upgrade.
- Use High Performance VMs -> I got following message -> So I need to do changes
also on the VM?
The following recommended settings for running the High Performance type with the optimal
configuration were not detected. Please consider manually changing of the following before
applying:
CPU PINNING:
VIRTUAL NUMA AND NUMA PINNING:
HUGE PAGES:
KERNEL SAME PAGE MERGING (KSM):
- libgfapi -> I will execute on the engine following command and then it will change
all VMs after reboot?
engine-config -s LibgfApiSupported=true --cver4.2
- changed values (from your other threat):
gluster volume set vmstore performance.read-ahead on
gluster volume set vmstore performance.stat-prefetch on
gluster volume set vmstore performance.write-behind-window-size 64MB
gluster volume set vmstore performance.flush-behind on
- already configured:
performance.client-io-threads: on
Best Regards
Dirk
5:53 p.m.
Additional Info: I've running Nextcloud on a VM.
- to sync (download) 1 file to the client with 300 MB is fast
- to sync (download) 300 files to the client with in total 1 MB is very slow
7:23 a.m.
На 19 август 2020 г. 22:39:22 GMT+03:00, info--- via Users <users(a)ovirt.org>
написа:
Thank you for the quick reply.
- I/O scheduler hosts -> changed
echo noop > /sys/block/sdb/queue/scheduler
echo noop > /sys/block/sdc/queue/scheduler
On reboot it will be reverted. Test this way and if you notice improvement use udev rules.
Keep in mind that multipath devices also have a scheduler.
- CPU states -> can you explain this a bit more?
On Intel CPUs you can use these kernel parameters (check them online as I type them by
memory)
processor.max_cstate=1 intel_idle.max_cstate=0
cat /dev/cpu_dma_latency
F
hexdump -C /dev/cpu_dma_latency
00000000 46 00 00 00 |F...|
00000004
- Tuned profile -> can you explain this a bit more?
Download
ftp://ftp.redhat.com/redhat/linux/enterprise/7Server/en/RHS/SRPMS/redhat-...
and extract the srpm.
Inside it you will find 2 dirs: 'rhgs-random-io'
&'rhgs-sequential-io'.
Open the tuned.conf files inside them and check the 'dirty' sysctl tunings. One
is for random I/O (usually the Vm workload is random I/O) and another for sequential.
You can set the 'dirty' settings in your sysctl.conf or your custom tuned
profile.
- MTU was already set to 9000 -> can you explain a bit more
Have you tested via :
ping -M do -s 8972 another-gluster-node
tcp-offloading and how I change the settings?
You can google it. Usually ethtool is your best friend.
ethtool -k enp1s0f0 | grep offload
tcp-segmentation-offload: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
l2-fwd-offload: off [fixed]
hw-tc-offload: off
rx-udp_tunnel-port-offload: on
- I/O scheduler VM had already none
cat /sys/block/vda/queue/scheduler
none
Consider using udev here too.
- oVirt 4.4 -> thanks. Was not aware that the new version is final.
I
will check and upgrade.
Upgrades are not so easy as you need to:
A) Upgrade OS to EL8
B) Upgrade ovirt
Test in advance before proceeding on prod !
- Use High Performance VMs -> I got following message -> So I
need to
do changes also on the VM?
The following recommended settings for running the High Performance
type with the optimal configuration were not detected. Please consider
manually changing of the following before applying:
CPU PINNING:
VIRTUAL NUMA AND NUMA PINNING:
HUGE PAGES:
KERNEL SAME PAGE MERGING (KSM):
- libgfapi -> I will execute on the engine following command and then
it will change all VMs after reboot?
engine-config -s LibgfApiSupported=true --cver4.2
VM reboot won't reread the VM config. You will need an 'off' and 'on'
action for each VM. I'm not sure about the command. I haven't used libgfapi for
eons.
- changed values (from your other threat):
gluster volume set vmstore performance.read-ahead on
gluster volume set vmstore performance.stat-prefetch on
gluster volume set vmstore performance.write-behind-window-size 64MB
gluster volume set vmstore performance.flush-behind on
- already configured:
performance.client-io-threads: on
It's easier to use:
gluster volume set VOLUME group virt
WARNING: This will enable sharding and SHARDING cannot be disabled in an easy way. NEVER
EVER DISABLE SHARDING.
You can check the 'virt' group of settings on your gluster nodes in
/var/lib/glusterd/group.
>Best Regards
>Dirk
>_______________________________________________
>Users mailing list -- users(a)ovirt.org
>To unsubscribe send an email to users-leave(a)ovirt.org
>Privacy Statement: https://www.ovirt.org/privacy-policy.html
>oVirt Code of Conduct:
>https://www.ovirt.org/community/about/community-guidelines/
>List Archives:
>https://lists.ovirt.org/archives/list/users@ovirt.org/message/VNE4O46SKA5KALKNW77MQ6UL3V2MJWWI/
7:24 a.m.
Are you using fully allocated VM disks ?
На 20 август 2020 г. 0:53:40 GMT+03:00, info--- via Users <users(a)ovirt.org> написа:
>Additional Info: I've running Nextcloud on a VM.
>- to sync (download) 1 file to the client with 300 MB is fast
>- to sync (download) 300 files to the client with in total 1 MB is very
>slow
>_______________________________________________
>Users mailing list -- users(a)ovirt.org
>To unsubscribe send an email to users-leave(a)ovirt.org
>Privacy Statement: https://www.ovirt.org/privacy-policy.html
>oVirt Code of Conduct:
>https://www.ovirt.org/community/about/community-guidelines/
>List Archives:
>https://lists.ovirt.org/archives/list/users@ovirt.org/message/LZRN2NRC7Y57XQ6GV2OJT4NY7Q74N6OE/
4:27 p.m.
Thank you. They are all Thin. I have something to do at the weekend and change them to
Preallocated. Looks like stopping the VM. Downloading and uploading the disk to change the
format.
4:28 p.m.
Thank you.
ping -M do -s 8972 another-gluster-node is working.
The rest I'll check on the weekend.
2:43 p.m.
Setting cluster.choose-local to on helped a lot to improve the read performance. Write
performance still bad.
Am I right that this look then more like a glusterfs issue and not something what need to
be changed on ovirt (libgfapi) or on VMs.
Changing tcp offloading did not make any difference.
- ethtool --offload enp1s0f0 rx off tx off
- ethtool --offload enp1s0f0 rx on tx on
MTU 9000 is fine and ping is working.
- ping -M do -s 8972 another-gluster-node
3:06 p.m.
Libgfapi bypasses the context switching from User space to kernel to user space (FUSE) ,
so it gets better performance.
I can't find the previous communication ,so can you share your volume settings again
?
Best Regards,
Strahil Nikolov
В неделя, 23 август 2020 г., 21:45:22 Гринуич+3, info--- via Users <users(a)ovirt.org>
написа:
Setting cluster.choose-local to on helped a lot to improve the read performance. Write
performance still bad.
Am I right that this look then more like a glusterfs issue and not something what need to
be changed on ovirt (libgfapi) or on VMs.
Changing tcp offloading did not make any difference.
- ethtool --offload enp1s0f0 rx off tx off
- ethtool --offload enp1s0f0 rx on tx on
MTU 9000 is fine and ping is working.
- ping -M do -s 8972 another-gluster-node
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/23FIY4DLT5C...
2:12 p.m.
I enabled libgfapi and powered off / on the VM.
- engine-config --all
- LibgfApiSupported: true version: 4.3
How can I see that this is active on the VM? The disk looks the same like before.
- virsh dumpxml 15
<disk type='file' device='disk' snapshot='no'>
<driver name='qemu' type='raw' cache='none'
error_policy='stop' io='threads'/>
<source
file='/rhev/data-center/mnt/glusterSD/10.9.9.101:_vmstore/f2c621de-42bf-4dbf-920c-adf4506b786d/images/1e231e3e-d98c-491a-9236-907814d4837/c755aaa3-7d3d-4c0d-8184-c6aae37229ba'>
<seclabel model='dac' relabel='no'/>
</source>
<backingStore/>
<target dev='sdc' bus='scsi'/>
<address type='drive' controller='0' bus='0'
target='0' unit='3'/>
</disk>
Here is the Volume setup:
Volume Name: vmstore
Type: Distributed-Replicate
Volume ID: 195e2a05-9667-4b8b-b0b7-82294631de50
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: 10.9.9.101:/gluster_bricks/vmstore/vmstore
Brick2: 10.9.9.102:/gluster_bricks/vmstore/vmstore
Brick3: 10.9.9.103:/gluster_bricks/vmstore/vmstore
Brick4: 10.9.9.101:/gluster_bricks/S4CYNF0M219849L/S4CYNF0M219849L
Brick5: 10.9.9.102:/gluster_bricks/S4CYNF0M219836L/S4CYNF0M219836L
Brick6: 10.9.9.103:/gluster_bricks/S4CYNF0M219801Y/S4CYNF0M219801Y
Options Reconfigured:
performance.write-behind-window-size: 64MB
performance.flush-behind: on
performance.stat-prefetch: on
performance.client-io-threads: on
nfs.disable: on
transport.address-family: inet
performance.strict-o-direct: on
performance.quick-read: off
performance.read-ahead: on
performance.io-cache: off
performance.low-prio-threads: 32
network.remote-dio: off
cluster.eager-lock: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
features.shard: on
user.cifs: off
cluster.choose-local: on
client.event-threads: 4
server.event-threads: 4
network.ping-timeout: 30
storage.owner-uid: 36
storage.owner-gid: 36
cluster.granular-entry-heal: enable
Thank you for your support.
4:46 p.m.
On Wed, Aug 26, 2020 at 8:19 PM info--- via Users <users(a)ovirt.org> wrote:
I enabled libgfapi and powered off / on the VM.
- engine-config --all
- LibgfApiSupported: true version: 4.3
How can I see that this is active on the VM? The disk looks the same like
before.
- virsh dumpxml 15
<disk type='file' device='disk' snapshot='no'>
<driver name='qemu' type='raw' cache='none'
error_policy='stop'
io='threads'/>
<source file='/rhev/data-center/mnt/glusterSD/10.9.9.101:
_vmstore/f2c621de-42bf-4dbf-920c-adf4506b786d/images/1e231e3e-d98c-491a-9236-907814d4837/c755aaa3-7d3d-4c0d-8184-c6aae37229ba'>
<seclabel model='dac' relabel='no'/>
</source>
<backingStore/>
<target dev='sdc' bus='scsi'/>
<address type='drive' controller='0' bus='0'
target='0' unit='3'/>
</disk>
Latest status I remember was this one reported by me at 4.3 RC2 time last
year:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KHWQA72JZVG...
and I think nothing changed in the mean time, neither for 4.3 nor for 4.4.
But not directly tested.
Someone reported 4x-5x improvements using libgfapi but initially there were
some blockers related to snapshots and HA if I remember correctly and so
disabled.
My idea is that all previous problems in upstream qemu and libvirtd pieces
are now solved since many months but developers didn't consider
implementing them due to not so big improvements detected in their tests
and/or sufficient time to dedicate to fix and/or other priorities.
Some bugzilla referred, to dig into if interested:
https://bugzilla.redhat.com/show_bug.cgi?id=1484227
https://bugzilla.redhat.com/show_bug.cgi?id=1465810
https://bugzilla.redhat.com/show_bug.cgi?id=1633642
HIH revamping the interesting topic.
Gianluca
11:55 a.m.
Looks like you’ve got a posix or nfs mount there? Is your gluster storage domain of type
GlusterFS? And make sure you restarted the ovirt-engine after enabling LibfgApiSupported,
before stopping and restarting the vm.
An active libgf mount looks like:
<disk type='network' device='disk' snapshot='no'>
<driver name='qemu' type='raw' cache='none'
error_policy='stop' io='native'/>
<source protocol='gluster'
name='gv1/a10b0cc4-b6ec-4b26-bef0-316e0e7bdcac/images/7e8b2be0-8206-4090-8fa4-c87d81fb5ea0/6b8c2ff9-15f3-47a9-af5d-bb7dea31b59b'>
<host name='gstor0' port='24007'/>
</source>
On Aug 26, 2020, at 1:12 PM, info--- via Users
<users(a)ovirt.org> wrote:
I enabled libgfapi and powered off / on the VM.
- engine-config --all
- LibgfApiSupported: true version: 4.3
How can I see that this is active on the VM? The disk looks the same like before.
- virsh dumpxml 15
<disk type='file' device='disk' snapshot='no'>
<driver name='qemu' type='raw' cache='none'
error_policy='stop' io='threads'/>
<source
file='/rhev/data-center/mnt/glusterSD/10.9.9.101:_vmstore/f2c621de-42bf-4dbf-920c-adf4506b786d/images/1e231e3e-d98c-491a-9236-907814d4837/c755aaa3-7d3d-4c0d-8184-c6aae37229ba'>
<seclabel model='dac' relabel='no'/>
</source>
<backingStore/>
<target dev='sdc' bus='scsi'/>
<address type='drive' controller='0' bus='0'
target='0' unit='3'/>
</disk>
Here is the Volume setup:
Volume Name: vmstore
Type: Distributed-Replicate
Volume ID: 195e2a05-9667-4b8b-b0b7-82294631de50
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: 10.9.9.101:/gluster_bricks/vmstore/vmstore
Brick2: 10.9.9.102:/gluster_bricks/vmstore/vmstore
Brick3: 10.9.9.103:/gluster_bricks/vmstore/vmstore
Brick4: 10.9.9.101:/gluster_bricks/S4CYNF0M219849L/S4CYNF0M219849L
Brick5: 10.9.9.102:/gluster_bricks/S4CYNF0M219836L/S4CYNF0M219836L
Brick6: 10.9.9.103:/gluster_bricks/S4CYNF0M219801Y/S4CYNF0M219801Y
Options Reconfigured:
performance.write-behind-window-size: 64MB
performance.flush-behind: on
performance.stat-prefetch: on
performance.client-io-threads: on
nfs.disable: on
transport.address-family: inet
performance.strict-o-direct: on
performance.quick-read: off
performance.read-ahead: on
performance.io-cache: off
performance.low-prio-threads: 32
network.remote-dio: off
cluster.eager-lock: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
features.shard: on
user.cifs: off
cluster.choose-local: on
client.event-threads: 4
server.event-threads: 4
network.ping-timeout: 30
storage.owner-uid: 36
storage.owner-gid: 36
cluster.granular-entry-heal: enable
Thank you for your support.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z3JD7MQV2PI...
4:19 p.m.
Thank you. Reboot of the engine and afterwards the backup server helped :-)
Before the change -> 53 minutes for 10% and the restore broke at 98% ...
- [2020-08-26:20:45:30] Restore Imaging: 1/1 - 10% Imaging in progress....
- [2020-08-26:20:41:22] Restore Imaging: 1/1 - 9% Imaging in progress...
- [2020-08-26:20:37:33] Restore Imaging: 1/1 - 8% Imaging in progress.
- [2020-08-26:20:32:04] Restore Imaging: 1/1 - 7% Imaging in progress...
- [2020-08-26:20:25:38] Restore Imaging: 1/1 - 6% Imaging in progress...
- [2020-08-26:20:21:34] Restore Imaging: 1/1 - 5% Imaging in progress...
- [2020-08-26:20:18:57] Restore Imaging: 1/1 - 3% Imaging in progress.
- [2020-08-26:20:03:15] Restore Imaging: 1/1 - 2% Imaging in progress....
- [2020-08-26:19:57:10] Restore Imaging: 1/1 - 1% Imaging in progress.....
- [2020-08-26:19:53:20] Restore Imaging: 1/1 - 0% Imaging in progress...........
- [2020-08-26:19:53:17] Restore Imaging: 1/1 - 0% Started Imaging Disk1.img.lzo to
/dev/sdb
After the change -> 10 minutes for 10%
- [2020-08-27:22:08:59] Restore Imaging: 1/1 - 10% Imaging in progress...
- [2020-08-27:22:08:02] Restore Imaging: 1/1 - 9% Imaging in progress...
- [2020-08-27:22:07:05] Restore Imaging: 1/1 - 8% Imaging in progress...
- [2020-08-27:22:05:34] Restore Imaging: 1/1 - 7% Imaging in progress...
- [2020-08-27:22:04:43] Restore Imaging: 1/1 - 6% Imaging in progress......
- [2020-08-27:22:03:34] Restore Imaging: 1/1 - 5% Imaging in progress..........
- [2020-08-27:22:02:27] Restore Imaging: 1/1 - 4% Imaging in progress.............
- [2020-08-27:22:01:39] Restore Imaging: 1/1 - 3% Imaging in progress........
- [2020-08-27:22:00:16] Restore Imaging: 1/1 - 2% Imaging in progress...........
- [2020-08-27:21:59:24] Restore Imaging: 1/1 - 1% Imaging in progress........
- [2020-08-27:21:58:16] Restore Imaging: 1/1 - 0% Imaging in progress..........
- [2020-08-27:21:58:13] Restore Imaging: 1/1 - 0% Started Imaging Disk1.img.lzo to
/dev/sdb
Should I revert some of my previous changes? Reduce the write window size?
- gluster volume set vmstore performance.read-ahead on
- gluster volume set vmstore performance.stat-prefetch on
- gluster volume set vmstore performance.write-behind-window-size 64MB
- gluster volume set vmstore performance.flush-behind on
- gluster volume set vmstore cluster.choose-local on
Now I will reboot the other VMs and go on with moving from thin provisioned disk to
preallocated.
6:01 p.m.
See below:
On Aug 27, 2020, at 3:19 PM, info--- via Users
<users(a)ovirt.org> wrote:
Thank you. Reboot of the engine and afterwards the backup server helped :-)
Good deal.
Should I revert some of my previous changes? Reduce the write window
size?
- gluster volume set vmstore performance.read-ahead on
- gluster volume set vmstore performance.stat-prefetch on
- gluster volume set vmstore performance.write-behind-window-size 64MB
- gluster volume set vmstore performance.flush-behind on
- gluster volume set vmstore cluster.choose-local on
I find these tend to be workload & system dependent, you’re best bet is to benchmark
and test yourself on your system.
I’m using these on my volumes:
performance.stat-prefetch: on
performance.read-ahead: off
performance.write-behind-window-size: 64MB
1356
days inactive
1365
days old
14 comments
4 participants
participants (4)
-
Darrell Budic -
Gianluca Cecchi -
info＠dsdm.ch -
Strahil Nikolov