On Thu, Oct 24, 2019 at 12:53 PM Vojtech Juranek <vjuranek@redhat.com> wrote:
Hi, I'm doing some manual storage tests on RHEL 8.1 host (kernel version 4.18.0-107.el8.x86_64) and run into following issue: when I try to move an image to block SD and RHEL 8.1 host is SPM, qemu-img gets stuck. There are errors regarding blk_cloned_rq_check_limits on RHEL host and also max_write_same_len on iscsi target (see bellow). iscsi target runs CentOS 7.6.
Before running qemu-img command, everything works fine (e.g. cat a file from iscsi target), after running qemu-img, all IO on iscsi target is very slow.
When I try to run qemu-img command manually, it's very slow (1GB image take several minutes), but finishes, when it's run from vdsm, it seems to be hung forever.
qemu-img is using BLKZEROOUT to write zeroes quickly, and this uses SCSI WRITE_SAME in the kernel.
In similar cases I found it could be fixed by adjusting various values in /sys/block/#DEVICE/queue/, but in this case, AFAICT all values are correct (same as on CentOS 7.7 host where everything work). Or it was identified as a bug in the kernel and was suggested to upgrade to newer kernel.
Do you know any workaround, how to make it working? Or does it sound to you like a bug in kernel which should be reported?
The issue is that the client kernel see wrong value of max_write_same_len. For some reason it thinks that the value is 65535 sectors (32MiB) but the value defined on the server is only 4096 sectors (2 MiB). When qemu-img issue WRITE_SAME with wrong payload size, the server reject the request and then the client treat this as a fatal error. You can fix this by configuring these attributes on the server side: $ targetcli /backstores/fileio/my-disk set attribute \ emulate_tpu=1 \ emulate_tpws=1 \ max_write_same_len=65335 - emulate_tpu enables discard - emulate_tpws enables write_same - max_write_same_len matches what the client see You can test the settings using blkdiscard like this: 1. Set max_write_same_len attribute on a backstore 2. login to the iscsi server 3. zero the LUN with that you modified on the server blkdiscard -z /dev/mapper/xxxyyy I think this is a kernel bug, but not sure if this on the initiator or target side. Nir Nir