- Devel - oVirt List Archives

Re: [virt-devel] Timeout failures in the OST
by Nir Soffer 02 Nov '20

02 Nov '20

On Mon, Nov 2, 2020 at 2:27 PM Benny Zlotnik <bzlotnik(a)redhat.com> wrote: Issues like this belong to oVirt devel mailing list. > looks like live merge failed[1]: > 2020-11-01 10:31:49,903+0100 ERROR (periodic/0) [virt.vm] (vmId='fcabfd2e-2937-4419-9b25-78fdd2b9c7c2') Unable to get watermarks for drive vdb: invalid argument: invalid path /rhev/data-center/mnt/blockSD/97b6175e-b6a9-419b-bd54-7c1e38c1bf71/images/fbb11a06-b8ef-4078-9530-978e7ca8ea0b/a911ad89-e461-4db4-88bf-a5d6590608b5 not assigned to domain (vm:1213) This may be a bug in vdsm live merge flow, tring to monitor a volume after the volume was already removed, or it may be libvirt/qemu bug. > ... > 2020-11-01 10:31:53,138+0100 ERROR (jsonrpc/1) [virt.vm] (vmId='fcabfd2e-2937-4419-9b25-78fdd2b9c7c2') merge: libvirt does not support volume chain monitoring. Unable to perform live merge. drive: vdb, alias: ua-fbb11a06-b8ef-4078-9530-978e7ca8ea0b, chains: {} (vm:5411) > > libvirt logs report: > ... > 2020-11-01 09:30:50.021+0000: 40137: error : virProcessRunInFork:1161 : internal error: child reported (status=125): > 2020-11-01 09:30:50.025+0000: 40137: error : virProcessRunInFork:1161 : internal error: child reported (status=125): internal error: child reported (status=125): > 2020-11-01 09:30:50.025+0000: 40137: warning : qemuDomainSnapshotDiskUpdateSource:15582 : Unable to move disk metadata on vm vm0 > 2020-11-01 09:31:45.539+0000: 40134: error : qemuMonitorJSONCheckError:412 : internal error: unable to execute QEMU command 'blockdev-del': Node libvirt-6-format is in use > 2020-11-01 09:31:45.539+0000: 40134: error : qemuMonitorJSONCheckError:412 : internal error: unable to execute QEMU command 'blockdev-del': Block device libvirt-6-storage is in use > 2020-11-01 09:31:45.540+0000: 40134: error : qemuMonitorJSONCheckError:412 : internal error: unable to execute QEMU command 'blockdev-del': Node 'libvirt-7-format' is busy: node is used as backing hd of 'libvirt-6-format' > 2020-11-01 09:31:45.541+0000: 40134: error : qemuMonitorJSONCheckError:412 : internal error: unable to execute QEMU command 'blockdev-del': Block device libvirt-7-storage is in use > 2020-11-01 09:31:45.900+0000: 40133: error : qemuDomainGetBlockInfo:12272 : invalid argument: invalid path /rhev/data-center/mnt/blockSD/97b6175e-b6a9-419b-bd54-7c1e38c1bf71/images/fbb11a06-b8ef-4078-9530-978e7ca8ea0b/a911ad89-e461-4db4-88bf-a5d6590608b5 not assigned to domain These smell like libvirt/qemu bug. Is this reproducible with RHEL 8.3? > Looks like the issue previously discussed in "[rhev-devel] Live storage migration instability in OST" two months ago has resurfaced > > > > Another issue seems to be the removal of the source disk: > 2020-11-01 10:31:48,056+0100 ERROR (tasks/3) [storage.StorageDomainManifest] removed image dir: /rhev/data-center/mnt/192.168.202.2:_exports_nfs_share1/3ca0e492-45f2-4383-b149-439043408bce/images/_remove_me_fbb11a06-b8ef-4078-9530-978e7ca8ea0b can't be removed (fileSD:258) > 2020-11-01 10:31:48,056+0100 ERROR (tasks/3) [storage.TaskManager.Task] (Task='70db80c2-076a-4ba1-a65d-821e6b5fe52c') Unexpected error (task:880) > Traceback (most recent call last): > File "/usr/lib/python3.6/site-packages/vdsm/storage/fileSD.py", line 251, in purgeImage > self.oop.os.rmdir(toDelDir) > File "/usr/lib/python3.6/site-packages/vdsm/storage/outOfProcess.py", line 238, in rmdir > self._iop.rmdir(path) > File "/usr/lib/python3.6/site-packages/ioprocess/__init__.py", line 550, in rmdir > return self._sendCommand("rmdir", {"path": path}, self.timeout) > File "/usr/lib/python3.6/site-packages/ioprocess/__init__.py", line 479, in _sendCommand > raise OSError(errcode, errstr) > OSError: [Errno 39] Directory not empty > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 887, in _run > return fn(*args, **kargs) > File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 350, in run > return self.cmd(*self.argslist, **self.argsdict) > File "/usr/lib/python3.6/site-packages/vdsm/storage/securable.py", line 79, in wrapper > return method(self, *args, **kwargs) > File "/usr/lib/python3.6/site-packages/vdsm/storage/sp.py", line 1947, in purgeImage > domain.purgeImage(sdUUID, imgUUID, volsByImg, discard) > File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 855, in purgeImage > self._manifest.purgeImage(sdUUID, imgUUID, volsImgs, discard) > File "/usr/lib/python3.6/site-packages/vdsm/storage/fileSD.py", line 259, in purgeImage > raise se.ImageDeleteError("%s %s" % (imgUUID, str(e))) > vdsm.storage.exception.ImageDeleteError: Could not remove all image's volumes: ('fbb11a06-b8ef-4078-9530-978e7ca8ea0b [Errno 39] Directory not empty',) > > But it's unclear what the leftover is Maybe be leftover from previous failed LSM. I think we need a better error message here, it should list the files in the non-empty directory. > [1] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/12782… > > > On Mon, Nov 2, 2020 at 1:44 PM Steven Rosenberg <srosenbe(a)redhat.com> wrote: >> >> Dear Benny, >> >> Thank you for your response. >> >> Here is the timeout engine log from one of the ps 45 failures [1]. >> >> It seems like this timeout is related to the engine failing and the ost scripts are not designed to detect the fail, thus timing out: >> >> 2020-11-01 10:31:55,175+01 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-67) [live_storage_migration] Command id: '60b3f7fc-93db-48b2-82a1-8a93c47e18e1 failed child command status for step 'MERGE_STATUS' >> >> >> >> >> [1] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/12796… >> >> With Best Regards. >> >> Steven. >> >> >> >> On Mon, Nov 2, 2020 at 12:51 PM Benny Zlotnik <bzlotnik(a)redhat.com> wrote: >>> >>> Can you link to the relevant engine/vdsm logs? >>> The timeout in the tests indicates that the desired state wasn't >>> reached so test logs don't provide the information about what exactly >>> happened >>> >>> On Sun, Nov 1, 2020 at 5:10 PM Steven Rosenberg <srosenbe(a)redhat.com> wrote: >>> > >>> > Dear virt-devel, >>> > >>> > We are currently experiencing many timeout failures in various patch sets for a gerrit issue 11395 [1]. >>> > >>> > The timeouts occur intermittently and seem to be unrelated to the changes which are only in the 004 module [2] and should have only affected VM1 / Disk1. >>> > >>> > We could use some advice on addressing these issues as well as a review of the patch to ensure we can move this patch forward. The patch sets and relevant timeouts are as follows: >>> > >>> > PS 40: >>> > >>> > test_live_storage_migration – test 004 [3] >>> > >>> > PS 41: >>> > >>> > on test_verify_engine_backup – test 002 [4] >>> > >>> > PS 43: >>> > >>> > on test_virtual_machines - test 100 [5] >>> > >>> > PS 45: >>> > >>> > on test_live_storage_migration – 004 [6] >>> > >>> > >>> > >>> > >>> > [1] https://gerrit.ovirt.org/#/c/111395/ >>> > [2] basic-suite-master/test-scenarios/004_basic_sanity.py >>> > [3] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/12628… >>> > [4] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/12659… >>> > [5] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/12744… >>> > [6] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/12782… >>> > >>> > With Best Regards. >>> > >>> > Steven >>> > >>> > >>>

1 0

Fwd: Timeout failures in the OST
by Steven Rosenberg 02 Nov '20

02 Nov '20

---------- Forwarded message --------- From: Steven Rosenberg <srosenbe(a)redhat.com> Date: Sun, Nov 1, 2020 at 5:09 PM Subject: Timeout failures in the OST To: virt-devel <virt-devel(a)redhat.com> Dear ovirt Personnel, We are currently experiencing many timeout failures in various patch sets for a gerrit issue 11395 [1]. The timeouts occur intermittently and seem to be unrelated to the changes which are only in the 004 module [2] and should have only affected VM1 / Disk1. We could use some advice on addressing these issues as well as a review of the patch to ensure we can move this patch forward. The patch sets and relevant timeouts are as follows: PS 40: test_live_storage_migration – test 004 [3] PS 41: on test_verify_engine_backup – test 002 [4] PS 43: on test_virtual_machines - test 100 [5] PS 45: on test_live_storage_migration – 004 [6] [1] https://gerrit.ovirt.org/#/c/111395/ [2] basic-suite-master/test-scenarios/004_basic_sanity.py [3] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/12628… [4] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/12659… [5] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/12744… [6] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/12782… With Best Regards. Steven

1 0

ovirt-engine has been tagged (ovirt-engine-4.4.3.9)
by Tal Nisan 27 Oct '20

27 Oct '20

1 0

vdsm fails on Travis
by Vojtech Juranek 22 Oct '20

22 Oct '20

Hi, some network tests fail on Travis, see e.g. [1]. Could someone from network team take a look? My guess is it's caused by commit 3ec1ad9a9f38 [2], but I'm not sure. Thanks Vojta [1] https://travis-ci.org/github/oVirt/vdsm/jobs/737665066 [2] https://github.com/oVirt/vdsm/commit/3ec1ad9a9f3879bf1c1b4aef2cbed4c3eda169…

3 3

Vdsm: ovirt-4.4.3 branch created
by Milan Zamazal 22 Oct '20

22 Oct '20

Hi, ovirt-4.4.3 branch has been created in Vdsm. Any patches after Vdsm 4.40.35 that should go to 4.4.3 (blockers only) must be backported. You can merge 4.4.4+ patches to master now. Regards, Milan

1 0

ovirt-engine 4.4.3 was branched
by Tal Nisan 22 Oct '20

22 Oct '20

Hi everyone, As you probably know, yesterday we've tagged and built the 4.4.3 RC build, to maintain stability and to allow work on 4.4.4 we have branched 4.4.3 so: For your 4.4.4 work - just push to master For 4.4.3 work (blockers/exceptions only!) - push to master and once it's merged backport to the ovirt-engine-4.4.3.z branch Thanks!

1 0

ovirt-engine has been tagged (ovirt-engine-4.4.3.8)
by Tal Nisan 21 Oct '20

21 Oct '20

1 0

callback hanging making commands executing serial
by Liran Rotenberg 20 Oct '20

20 Oct '20

Hi all, Investigating a bug about serial execution <https://bugzilla.redhat.com/1885286> of the same command more than once shows a problem with our callbacks. The problem is felt when we have synchronized operation (such as running ansible runner service) that takes a bit time to be executed. The problem is, when running multiple commands and one has synchronized long operation within a child command, it will hang out all the commands running on the engine using callbacks. The good example is given in the bug, since export OVA command using ansible(sync) and the pack_ova script takes time to finish. This is not really noticeable with short synchronized commands, but it does execute serial for them as well instead of parallel. Running export to OVA command and then other commands with callbacks will get them hanging if the export command callback started and it reached to performNextOperation. Some technical details: We have one thread in CommandCallbacksPoller[1] that runs, collects the current command on the engine and processes them. Once we have the above scenario (let's say 2 commands), the first one will go into callback.doPolling in invokeCallbackMethodsImpl. [2] In that case it will go to ChildCommandsCallbackBase::doPolling, eventually to childCommandsExecutingEnded. [3] While there are more actions to perform we will do performNextOperation [4], which calls executeNextOperation (in the bug case [5]). When the next operation is long and synchronized, this will block the CommandCallbacksPoller thread and only when it finishes the thread is released and the callbacks continue working. Any idea how to solve this issue? Regards, Liran [1] - https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/b… [2] - https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/b… [3] - https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/b… [4] - https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/b… [5] - https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/b…

4 5

oVirt Survey Autumn 2020 results
by Sandro Bonazzola 19 Oct '20

19 Oct '20

Thanks to the 79 participants to oVirt Survey Autumn 2020! The survey is now closed and results are publicly available at https://docs.google.com/forms/d/1eMYaCoysPkYt2iHZW2RRO1HmSqsD0phrwSTVYFlqLy… We'll analyze collected data in order to improve oVirt thanks to your feedback. Thanks, -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.* * <https://www.redhat.com/it/forums/emea/italy-track>*

1 0

Re: [CQ]: 111471,2 (ovirt-engine) failed "ovirt-master" system tests
by Yedidyah Bar David 15 Oct '20

15 Oct '20

On Wed, Oct 14, 2020 at 7:06 PM oVirt Jenkins <jenkins(a)ovirt.org> wrote: > > Change 111471,2 (ovirt-engine) is probably the reason behind recent system test > failures in the "ovirt-master" change queue and needs to be fixed. > > This change had been removed from the testing queue. Artifacts build from this > change will not be released until it is fixed. > > For further details about the change see: > https://gerrit.ovirt.org/#/c/111471/2 This fails compilation. Liran already commented there with details, posting this for visibility. Is anyone handling this? Should we revert? Thanks and best regards, -- Didi

4 8