[ovirt-users] Re: very very bad iscsi performance

On Tue, Jul 21, 2020 at 2:20 AM Philip Brown <pbrown(a)medata.com&gt; wrote: What we see in your logs: You are using: - thin disk - qcow2 image on logical volume: - virtio-scsi <disk type='block' device='disk' snapshot='no'> <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='native'/> <source dev='/rhev/data-center/mnt/blockSD/fddf9b00-6c80-4be2-85f1-4b3852d93786/images/47af0207-8b51-4a59-a93e-fddf9ed56d44/743550ef-7670-4556-8d7f-4d6fcfd5eb70'> <seclabel model='dac' relabel='no'/> </source> <backingStore/> <target dev='sda' bus='scsi'/> <serial>47af0207-8b51-4a59-a93e-fddf9ed56d44</serial> <boot order='1'/> <alias name='ua-47af0207-8b51-4a59-a93e-fddf9ed56d44'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> -object iothread,id=iothread1 \ -device virtio-scsi-pci,iothread=iothread1,id=ua-a50f193d-fa74-419d-bf03-f5a2677acd2a,bus=pci.0,addr=0x5 \ -drive file=/rhev/data-center/mnt/blockSD/87cecd83-d6c8-4313-9fad-12ea32768703/images/47af0207-8b51-4a59-a93e-fddf9ed56d44/743550ef-7670-4556-8d7f-4d6fcfd5eb70,format=qcow2,if=none,id=drive-ua-47af0207-8b51-4a59-a93e-fddf9ed56d44,serial=47af0207-8b51-4a59-a93e-fddf9ed56d44,werror=stop,rerror=stop,cache=none,aio=native \ This is the most flexible option oVirt has, but not the default. Known issue with such a disk is possible pausing of the VM when the disk becomes full, if oVirt cannot extend the underlying logical volume fast enough. It can be mitigated by using larger chunks in vdsm. We recommend these settings if you are going to use VMs with heavy I/O with thin disks: # cat /etc/vdsm/vdsm.conf.d/99-local.conf [irs] # Together with volume_utilization_chunk_mb, set the minimal free # space before a thin provisioned block volume is extended. Use lower # values to extend earlier. # default value: # volume_utilization_percent = 50 volume_utilization_percent = 25 # Size of extension chunk in megabytes, and together with # volume_utilization_percent, set the free space limit. Use higher # values to extend in bigger chunks. # default value: # volume_utilization_chunk_mb = 1024 volume_utilization_chunk_mb = 4096 With this configuration, when free space on the disk is 1 GiB, oVirt will extend the disk by 4 GiB. So your disk may be up to 5 GiB larger than the used space, but if the VM is writing data very fast, the chance of pausing is reduced. If you want to reduce the chance of pausing your database in the most busy times to zero, using a preallocated disk is the way. In oVirt 4.4. you can check this option when creating a disk: [x] Enable Incremental Backup With: Allocation Policy: [Preallocated] You will get a preallocated disk in the specified size, using qcow2 format. This gives you both the option to use incremental backup, faster disk operations in oVirt (since qemu-img does not need to read the entire disk), and avoids the pausing issue. It may also defeat thin provisioning, but if your backend storage supports thin provisioning anyway it does not matter. To get best performance for database use case preallocated volume should be better. Please try to benchmark: - raw preallocated disk - using virtio instead of virtio-scsi If your database can use multiple disks, you may get better performance by adding multiple disks and use one iothread per disk. See also interesting talk about storage performance from 2017: https://events19.lfasiallc.com/wp-content/uploads/2017/11/Storage-Perform...