[ovirt-users] Re: [BULK] Re: very very bad iscsi performance

Thank you for the analysis. I have some further comments: First off, filebench pre-writes the files before doing oltp benchmarks, so I dont think the thin provisioning is at play here. I will double check this, but if you dont hear otherwise, please presume that is the case :) Secondly, I am surprised at your recommendation to use virtio instead of virtio-scsi. since the writeup for virtio-scsi claims it has equivalent performance in general, and adds better scaling https://www.ovirt.org/develop/release-management/features/storage/virtio-... As far as your suggestion for using multiple disks for scaling higher: We are using an SSD. Isnt the whole advantage of using SSD drives, that you can get the IOP/s performance of 10 drives, out of a single drive? We certainly get that using it natively, outside of a VM. SO it would be nice to see performance approaching that within an ovirt VM. ----- Original Message ----- From: "Nir Soffer" <nsoffer(a)redhat.com&gt; To: "Philip Brown" <pbrown(a)medata.com&gt; Cc: "users" <users(a)ovirt.org&gt;, "qemu-block" <qemu-block(a)nongnu.org&gt;, "Stefan Hajnoczi" <stefanha(a)redhat.com&gt;, "Paolo Bonzini" <pbonzini(a)redhat.com&gt;, "Sergio Lopez Pascual" <slp(a)redhat.com&gt;, "Mordechai Lehrer" <mlehrer(a)redhat.com&gt;, "Kevin Wolf" <kwolf(a)redhat.com&gt; Sent: Tuesday, July 21, 2020 4:23:36 AM Subject: [BULK] Re: [ovirt-users] very very bad iscsi performance On Tue, Jul 21, 2020 at 2:20 AM Philip Brown <pbrown(a)medata.com&gt; wrote: What we see in your logs: You are using: - thin disk - qcow2 image on logical volume: - virtio-scsi <disk type='block' device='disk' snapshot='no'> <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='native'/> <source dev='/rhev/data-center/mnt/blockSD/fddf9b00-6c80-4be2-85f1-4b3852d93786/images/47af0207-8b51-4a59-a93e-fddf9ed56d44/743550ef-7670-4556-8d7f-4d6fcfd5eb70'> <seclabel model='dac' relabel='no'/> </source> <backingStore/> <target dev='sda' bus='scsi'/> <serial>47af0207-8b51-4a59-a93e-fddf9ed56d44</serial> <boot order='1'/> <alias name='ua-47af0207-8b51-4a59-a93e-fddf9ed56d44'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> -object iothread,id=iothread1 \ -device virtio-scsi-pci,iothread=iothread1,id=ua-a50f193d-fa74-419d-bf03-f5a2677acd2a,bus=pci.0,addr=0x5 \ -drive file=/rhev/data-center/mnt/blockSD/87cecd83-d6c8-4313-9fad-12ea32768703/images/47af0207-8b51-4a59-a93e-fddf9ed56d44/743550ef-7670-4556-8d7f-4d6fcfd5eb70,format=qcow2,if=none,id=drive-ua-47af0207-8b51-4a59-a93e-fddf9ed56d44,serial=47af0207-8b51-4a59-a93e-fddf9ed56d44,werror=stop,rerror=stop,cache=none,aio=native \ This is the most flexible option oVirt has, but not the default. Known issue with such a disk is possible pausing of the VM when the disk becomes full, if oVirt cannot extend the underlying logical volume fast enough. It can be mitigated by using larger chunks in vdsm. We recommend these settings if you are going to use VMs with heavy I/O with thin disks: # cat /etc/vdsm/vdsm.conf.d/99-local.conf [irs] # Together with volume_utilization_chunk_mb, set the minimal free # space before a thin provisioned block volume is extended. Use lower # values to extend earlier. # default value: # volume_utilization_percent = 50 volume_utilization_percent = 25 # Size of extension chunk in megabytes, and together with # volume_utilization_percent, set the free space limit. Use higher # values to extend in bigger chunks. # default value: # volume_utilization_chunk_mb = 1024 volume_utilization_chunk_mb = 4096 With this configuration, when free space on the disk is 1 GiB, oVirt will extend the disk by 4 GiB. So your disk may be up to 5 GiB larger than the used space, but if the VM is writing data very fast, the chance of pausing is reduced. If you want to reduce the chance of pausing your database in the most busy times to zero, using a preallocated disk is the way. In oVirt 4.4. you can check this option when creating a disk: [x] Enable Incremental Backup With: Allocation Policy: [Preallocated] You will get a preallocated disk in the specified size, using qcow2 format. This gives you both the option to use incremental backup, faster disk operations in oVirt (since qemu-img does not need to read the entire disk), and avoids the pausing issue. It may also defeat thin provisioning, but if your backend storage supports thin provisioning anyway it does not matter. To get best performance for database use case preallocated volume should be better. Please try to benchmark: - raw preallocated disk - using virtio instead of virtio-scsi If your database can use multiple disks, you may get better performance by adding multiple disks and use one iothread per disk. See also interesting talk about storage performance from 2017: https://events19.lfasiallc.com/wp-content/uploads/2017/11/Storage-Perform...