- Devel - oVirt List Archives

qemu-img process stuck
by Vojtech Juranek 01 Nov '19

01 Nov '19

Hi, I'm doing some manual storage tests on RHEL 8.1 host (kernel version 4.18.0-107.el8.x86_64) and run into following issue: when I try to move an image to block SD and RHEL 8.1 host is SPM, qemu-img gets stuck. There are errors regarding blk_cloned_rq_check_limits on RHEL host and also max_write_same_len on iscsi target (see bellow). iscsi target runs CentOS 7.6. Before running qemu-img command, everything works fine (e.g. cat a file from iscsi target), after running qemu-img, all IO on iscsi target is very slow. When I try to run qemu-img command manually, it's very slow (1GB image take several minutes), but finishes, when it's run from vdsm, it seems to be hung forever. In similar cases I found it could be fixed by adjusting various values in /sys/block/#DEVICE/queue/, but in this case, AFAICT all values are correct (same as on CentOS 7.7 host where everything work). Or it was identified as a bug in the kernel and was suggested to upgrade to newer kernel. Do you know any workaround, how to make it working? Or does it sound to you like a bug in kernel which should be reported? Thanks Vojta Example of qemu-img command: [root@localhost ~]#/usr/bin/qemu-img convert -p -t none -T none -f raw /rhev/data-center/mnt/blockSD/ceba76d5-9d5d-4b04-9226-dffc7e77d4be/images/9aafce3d-53a6-41d7-b0d5-0e1db8aa59fc/7b7c7800-4837-4c3f-967c-c942b9c54e1b -O raw -W /rhev/data-center/mnt/blockSD/a17d7695-50fc-412e-abc7-008ebb6cdf23/images/9aafce3d-53a6-41d7-b0d5-0e1db8aa59fc/7b7c7800-4837-4c3f-967c-c942b9c54e1b (100.00/100%) Errors on RHEL 8.1 host: [ 356.658973] print_req_error: critical target error, dev dm-8, sector 31203328 flags 9 [ 356.658982] print_req_error: critical target error, dev dm-8, sector 31268863 flags 9 [ 356.658984] print_req_error: critical target error, dev dm-8, sector 31334398 flags 9 [ 356.658986] print_req_error: critical target error, dev dm-8, sector 31399933 flags 9 [ 356.658987] print_req_error: critical target error, dev dm-8, sector 31465468 flags 9 [ 356.658989] print_req_error: critical target error, dev dm-8, sector 31531003 flags 9 [ 356.658999] print_req_error: critical target error, dev dm-8, sector 31596538 flags 9 [ 356.659001] print_req_error: critical target error, dev dm-8, sector 31662073 flags 9 [ 356.659008] blk_cloned_rq_check_limits: over max size limit. [ 356.659044] device-mapper: multipath: Failing path 8:80. [ 356.669966] print_req_error: critical target error, dev dm-8, sector 31727608 flags 9 [ 356.669987] print_req_error: critical target error, dev dm-8, sector 31793143 flags 9 [ 358.923614] device-mapper: multipath: Reinstating path 8:80. [ 358.933888] sd 6:0:0:4: alua: port group 00 state A non-preferred supports TOlUSNA [ 358.933980] blk_cloned_rq_check_limits: over max size limit. [ 358.934008] device-mapper: multipath: Failing path 8:80. [ 364.927647] device-mapper: multipath: Reinstating path 8:80. [ 364.956799] sd 6:0:0:4: alua: port group 00 state A non-preferred supports TOlUSNA [ 364.956890] blk_cloned_rq_check_limits: over max size limit. [...] [ 475.001255] blk_cloned_rq_check_limits: over max size limit. [ 475.001271] device-mapper: multipath: Failing path 8:80. [ 479.995729] device-mapper: multipath: Reinstating path 8:80. [ 480.006078] sd 6:0:0:4: alua: port group 00 state A non-preferred supports TOlUSNA [ 480.006114] blk_cloned_rq_check_limits: over max size limit. [ 480.006134] device-mapper: multipath: Failing path 8:80. [ 484.997257] device-mapper: multipath: Reinstating path 8:80. [ 485.006868] sd 6:0:0:4: alua: port group 00 state A non-preferred supports TOlUSNA [ 485.006908] blk_cloned_rq_check_limits: over max size limit. [ 485.006960] device-mapper: multipath: Failing path 8:80. [ 489.998641] device-mapper: multipath: Reinstating path 8:80. [ 490.010288] sd 6:0:0:4: alua: port group 00 state A non-preferred supports TOlUSNA [ 490.010430] blk_cloned_rq_check_limits: over max size limit. [ 490.010455] device-mapper: multipath: Failing path 8:80. [ 492.922671] INFO: task qemu-img:9509 blocked for more than 120 seconds. [ 492.922675] Tainted: G --------- -t - 4.18.0-107.el8.x86_64 #1 [ 492.922676] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 492.922677] qemu-img D 0 9509 7416 0x00000080 [ 492.922679] Call Trace: [ 492.922703] ? __schedule+0x253/0x830 [ 492.922704] schedule+0x28/0x70 [ 492.922705] schedule_timeout+0x26d/0x390 [ 492.922711] ? blk_flush_plug_list+0xd7/0x100 [ 492.922713] io_schedule_timeout+0x19/0x40 [ 492.922714] wait_for_completion_io+0x11f/0x190 [ 492.922717] ? wake_up_q+0x70/0x70 [ 492.922719] submit_bio_wait+0x5b/0x80 [ 492.922728] blkdev_issue_zeroout+0x142/0x220 [ 492.922737] blkdev_fallocate+0x11e/0x190 [ 492.922745] vfs_fallocate+0x13f/0x280 [ 492.922753] ksys_fallocate+0x3c/0x80 [ 492.922754] __x64_sys_fallocate+0x1a/0x20 [ 492.922756] do_syscall_64+0x5b/0x1b0 [ 492.922764] entry_SYSCALL_64_after_hwframe+0x65/0xca [ 492.922767] RIP: 0033:0x7f9e2f8f1e85 [ 492.922771] Code: Bad RIP value. [ 492.922771] RSP: 002b:00007f9e2d79ba20 EFLAGS: 00000293 ORIG_RAX: 000000000000011d [ 492.922772] RAX: ffffffffffffffda RBX: 000000000000000a RCX: 00007f9e2f8f1e85 [ 492.922773] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 000000000000000a [ 492.922773] RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000001 [ 492.922774] R10: 0000000040000000 R11: 0000000000000293 R12: 0000000000000000 [ 492.922774] R13: 0000000040000000 R14: 000055b21fe81390 R15: 00007f9e2d79bbc0 Errors on iscsi target: [ 703.199374] WRITE_SAME sectors: 65535 exceeds max_write_same_len: 4096 [ 703.200836] WRITE_SAME sectors: 65535 exceeds max_write_same_len: 4096 [ 703.202089] WRITE_SAME sectors: 65535 exceeds max_write_same_len: 4096 [ 703.203289] WRITE_SAME sectors: 65535 exceeds max_write_same_len: 4096 [ 703.204479] WRITE_SAME sectors: 65535 exceeds max_write_same_len: 4096 [ 703.205819] WRITE_SAME sectors: 65535 exceeds max_write_same_len: 4096 [ 703.206878] WRITE_SAME sectors: 65535 exceeds max_write_same_len: 4096 [ 703.208020] WRITE_SAME sectors: 65535 exceeds max_write_same_len: 4096 [ 703.209084] WRITE_SAME sectors: 65535 exceeds max_write_same_len: 4096 [ 703.210357] WRITE_SAME sectors: 65535 exceeds max_write_same_len: 4096 [ 703.211439] WRITE_SAME sectors: 65535 exceeds max_write_same_len: 4096 [ 703.212584] WRITE_SAME sectors: 65535 exceeds max_write_same_len: 4096 [ 703.213756] WRITE_SAME sectors: 65535 exceeds max_write_same_len: 4096 [ 703.214813] WRITE_SAME sectors: 65535 exceeds max_write_same_len: 4096

2 2

Creating bonds between Network Interface Cards with SDK
by Jamie Holohan 31 Oct '19

31 Oct '19

How can I bond one network interface to another using the SDK. I am try to perform the task in java but I'm unsure in what context I can find the utilities that will do this task. The task can be performed in the OVM UI after a host has been initialised. By navigating from the side bar Compute -> Hosts -> (Select Host) -> (Host tabs) Network Interfaces -> From here, click Setup Host Networks. In the pop up you can select a network interface and drag it to another to create a bond between them. How can I perform this step using the SDK. I am attempting to perform this in the context of a nicservice, found like this: systemService.hostsService().hostService(hostObject).nicservice(NICObject)).get().send().nic(); but I can't find any methods associated with the retrieved NIC or nicservice that seem relevant to my task.

2 1

Planned restart of production services
by Evgheni Dereveanchin 30 Oct '19

30 Oct '19

Hi everyone, I'll be restarting several production systems today to apply security updates. The following services may be unreachable for short periods of time: - resources.ovirt.org - software repositories - jenkins.ovirt.org - CI master No new builds will be started during this period. I will announce you once maintenance is complete. -- Regards, Evgheni Dereveanchin

1 1

vdsm has been tagged (v4.30.35)
by Petr Horacek 30 Oct '19

30 Oct '19

1 0

ovirt-engine has been tagged (ovirt-engine-4.3.7.1)
by Tal Nisan 30 Oct '19

30 Oct '19

1 0

CentOS builds fail because cbs.centos.org/repos/virt7-kvm-common-testing/{x86_64/ppc64le} not found
by Nir Soffer 30 Oct '19

30 Oct '19

We have failing imageio builds, seems that the centos virt7-kvm-common-testing is broken. ppc64le: https://cbs.centos.org/repos/virt7-kvm-common-testing/ppc64le/os/repodata/r…: [Errno 14] HTTPS Error 404 - Not Found https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/1611/artif… x86_64: https://cbs.centos.org/repos/virt7-kvm-common-testing/x86_64/os/repodata/re…: [Errno 14] HTTPS Error 404 - Not Found https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/1611/artif… Patch: https://gerrit.ovirt.org/c/101788/ Sandro, can you check this? Nr

2 1

[vdsm] Standard patch check broken by Black formatter for network stuff
by Vojtech Juranek 29 Oct '19

29 Oct '19

Hi, standard patch check tests for vdsm are broken due to Black formatter check failure (see e.g. [1]): Oh no! 10:16:24 6 files would be reformatted, 164 files would be left unchanged. 10:16:24 ERROR: InvocationError for command /home/jenkins/workspace/ vdsm_standard-check-patch/vdsm/.tox/black/bin/black -l 79 -S --check ./lib/ vdsm/network/ ./tests/network (exited with code 1) Can someone take a look and fix it? I did a quick check and wasn't able to find any recent commit which should cause this failure:-( Thanks Vojta [1] https://jenkins.ovirt.org/job/vdsm_standard-check-patch/13396

3 2

OST: Failing migrations on el8
by Milan Zamazal 29 Oct '19

29 Oct '19

Hi, I looked at the failing migrations in OST on el8, when running basic-suite-master with https://gerrit.ovirt.org/#/c/103888/31. The migration fails even before started, when Vdsm tries to talk to the remote Vdsm and can't reach it. Indeed, there seems to be a networking problem between the hosts: [root@lago-basic-suite-master-host-1 ~]# ping -c 1 lago-basic-suite-master-host-0 PING lago-basic-suite-master-host-0(lago-basic-suite-master-host-0.lago.local (fd8f:1391:3a82:201::c0a8:c902)) 56 data bytes From lago-basic-suite-master-host-1 (fd8f:1391:3a82:200::c0a8:c899): icmp_seq=1 Destination unreachable: Address unreachable Regards, Milan

5 5

Errors while building ovirt-engine master on fedora 29
by Kaustav Majumder 29 Oct '19

29 Oct '19

Hi, I am currently facing this error while building ovirt-engine master (commit da774d861d5a593666932f6e678f5383222ed8d5) on fedora 29. *[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project ovirt-checkstyle-extension: Fatal error compiling: invalid flag: --release -> [Help 1]* full build log here: https://pastebin.com/rJA5PHWu -- Thanks, Kaustav Majumder

3 6

Automatic snapshots
by Joost Evertse 29 Oct '19

29 Oct '19

Hello everyone, We run a large number of virtual desktops on RHEV and we use the snapshot mechanism occasionally. Would it be possible to create automatic snapshots when VM’s are started and when they are stopped? My gut feeling says this would be a bad idea, because taking snapshots does not guarantee anything, but I have the task of exploring the option. If needed, I would change source code to create something and contribute it. Currently I have taken a look at - the vdsm hook mechanism, but that seems to be aimed at the VM lifecycle, not operations like starting and stopping an instance. - the ovirt-agent hooks , but they also seem to be useful when migrating a VM The goal of the snapshot is to have a backup-backup, just handy when dealing with malfunctions inside a VM workstation, it does not have to be perfect. Maybe I can create scripts to enable functionality in a different (existing) place, but I am a bit reluctant to create unnecessary scripting because that creates admin overhead. Maybe from an architectural point of view automatic snapshots is just a plain wrong idea, not sure. What are your thoughts about this? With kind regards, Joost Evertse

1 0