- Devel - oVirt List Archives

global_setup.sh fail: "docker.service: Failed at step EXEC spawning /usr/bin/dockerd-current: Permission denied"
by Nir Soffer 07 Dec '20

07 Dec '20

Seems that all builds fail now because the global setup script fails to setup the slave. Some examples: - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25522/ - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25520/ - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25519/ - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25516/ - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25515/ - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25514/ - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25513/ - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25511/ - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25510/ - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25509/ - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25508/ Log from the failed step: https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… [2020-12-07T18:09:18.284Z] Dec 07 15:11:08 vm0148.workers-phx.ovirt.org dockerd-current[1342]: time="2020-12-07T15:11:08.426348650Z" level=warning msg="d10f349413a9ce2b5f6cb33d7d721d133b323aba1bf00021ce4d714059b00813 cleanup: failed to unmount secrets: invalid argument" [2020-12-07T18:09:18.284Z] Dec 07 15:11:39 vm0148.workers-phx.ovirt.org dockerd-current[1342]: time="2020-12-07T15:11:39.112783512Z" level=warning msg="55ff0f77d37781237a199a8ba86bc544932f10b1018a3e614bd967b8b5956fd5 cleanup: failed to unmount secrets: invalid argument" [2020-12-07T18:09:18.284Z] Dec 07 15:11:40 vm0148.workers-phx.ovirt.org dockerd-current[1342]: time="2020-12-07T15:11:40.654819711Z" level=info msg="Layer sha256:9d23de0c6d4682039041ca8bb2825278a71c411a22799aeb7b47739514a731d0 cleaned up" [2020-12-07T18:09:18.284Z] Dec 07 15:11:41 vm0148.workers-phx.ovirt.org dockerd-current[1342]: time="2020-12-07T15:11:41.290807003Z" level=info msg="Layer sha256:9d23de0c6d4682039041ca8bb2825278a71c411a22799aeb7b47739514a731d0 cleaned up" [2020-12-07T18:09:18.284Z] Dec 07 15:11:45 vm0148.workers-phx.ovirt.org systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT [2020-12-07T18:09:18.284Z] -- Subject: Unit process exited [2020-12-07T18:09:18.284Z] -- Defined-By: systemd [2020-12-07T18:09:18.284Z] -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel [2020-12-07T18:09:18.284Z] -- [2020-12-07T18:09:18.284Z] -- An ExecStart= process belonging to unit docker.service has exited. [2020-12-07T18:09:18.284Z] -- [2020-12-07T18:09:18.284Z] -- The process' exit code is 'exited' and its exit status is 2. [2020-12-07T18:09:18.284Z] Dec 07 15:11:45 vm0148.workers-phx.ovirt.org systemd[1]: docker.service: Failed with result 'exit-code'. [2020-12-07T18:09:18.284Z] -- Subject: Unit failed [2020-12-07T18:09:18.284Z] -- Defined-By: systemd [2020-12-07T18:09:18.284Z] -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel [2020-12-07T18:09:18.284Z] -- [2020-12-07T18:09:18.284Z] -- The unit docker.service has entered the 'failed' state with result 'exit-code'. [2020-12-07T18:09:18.284Z] Dec 07 15:12:30 vm0148.workers-phx.ovirt.org systemd[1]: Starting Docker Application Container Engine... [2020-12-07T18:09:18.284Z] -- Subject: A start job for unit docker.service has begun execution [2020-12-07T18:09:18.284Z] -- Defined-By: systemd [2020-12-07T18:09:18.284Z] -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel [2020-12-07T18:09:18.284Z] -- [2020-12-07T18:09:18.284Z] -- A start job for unit docker.service has begun execution. [2020-12-07T18:09:18.284Z] -- [2020-12-07T18:09:18.284Z] -- The job identifier is 109048. [2020-12-07T18:09:18.284Z] Dec 07 15:12:30 vm0148.workers-phx.ovirt.org systemd[4524]: docker.service: Failed to execute command: Permission denied [2020-12-07T18:09:18.284Z] Dec 07 15:12:30 vm0148.workers-phx.ovirt.org systemd[4524]: docker.service: Failed at step EXEC spawning /usr/bin/dockerd-current: Permission denied [2020-12-07T18:09:18.284Z] -- Subject: Process /usr/bin/dockerd-current could not be executed [2020-12-07T18:09:18.284Z] -- Defined-By: systemd [2020-12-07T18:09:18.284Z] -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel [2020-12-07T18:09:18.284Z] -- [2020-12-07T18:09:18.284Z] -- The process /usr/bin/dockerd-current could not be executed and failed. [2020-12-07T18:09:18.284Z] -- [2020-12-07T18:09:18.284Z] -- The error number returned by this process is ERRNO. [2020-12-07T18:09:18.284Z] Dec 07 15:12:30 vm0148.workers-phx.ovirt.org systemd[1]: docker.service: Main process exited, code=exited, status=203/EXEC [2020-12-07T18:09:18.284Z] -- Subject: Unit process exited [2020-12-07T18:09:18.284Z] -- Defined-By: systemd [2020-12-07T18:09:18.284Z] -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel [2020-12-07T18:09:18.284Z] -- [2020-12-07T18:09:18.284Z] -- An ExecStart= process belonging to unit docker.service has exited. [2020-12-07T18:09:18.284Z] -- [2020-12-07T18:09:18.284Z] -- The process' exit code is 'exited' and its exit status is 203. ...

1 0

ovirt-system-tests: engine fqdn, dns domain name, etc
by Yedidyah Bar David 07 Dec '20

07 Dec '20

Hi all, Current [1] is broken - it misses the domain name. After spending several hours on this last few days, I decided to give up for now and just fix by appending hard-coded 'lago.local'. I am testing locally, and if it passes, I'll push to gerrit. Going forward, I'd like to have as few as possible guesses and hard-coded strings that must match: - Engine fqdn should be hard-coded, arbitrary (but not "engine", as explained), ideally in a single place, propagated from there as needed - storage server (used in HE suites) should also be hard-coded and not guessed from the engine machine name like now - Same for domain name ("lago.local") - Since it's hard to read lago init file and parse it to get stuff from it without actually using lago (and even when using it, e.g. I failed to get dns domain name), we should decide on these things before starting lago and inject them into the lagoinitfile (like we do with other stuff via env). - Also, and to better (IMO) match what I think should happen in reality (for real setups), it should all be done on DNS level (dnsmasq of libvirt, controlled by lago, AFAIU), and we should stop playing with /etc/hosts. If there is agreement, I might decide to work on this. Or we can at least open a bug/ticket somewhere :-). For some of my thoughts/attempts along the way, you might want to see [2][3][4]. All are WIP, none work, probably none even is worth the time of a real review. Thoughts/opinions/ideas are most welcome! Thanks and best regards, [1] https://gerrit.ovirt.org/c/ovirt-system-tests/+/112488 [2] https://github.com/didib/ovirt-system-tests/tree/ostengine [3] https://github.com/didib/ovirt-system-tests/tree/fqdn-from-env [4] https://github.com/didib/ovirt-system-tests/tree/engine-fqdn-env -- Didi

1 0

planned Jenkins restart
by Evgheni Dereveanchin 02 Dec '20

02 Dec '20

Hi everyone, I'll be performing a planned Jenkins restart within the next hour. No new CI jobs will be scheduled during this maintenance period. I will inform you once it is back online. -- Regards, Evgheni Dereveanchin

2 2

Storage OST Failure
by Steven Rosenberg 02 Dec '20

02 Dec '20

Dear Personnel, It seems that the OST is failing due to a storage issue [1]. The failure is on the test_verify_and_remove_cloned_vm test which is not changed on the patch [2] . The engine log prints the error: 2020-12-01 13:07:48,297+01 ERROR [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-1) [] An error occurred while fetching unregistered disks from Storage Domain id '26d55c27-bf61-4f01-b5a3-d9a6a01c3e8a'2020-12-01 13:07:48,298+01 DEBUG [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-1) [] Skipping format update for domain '26d55c27-bf61-4f01-b5a3-d9a6a01c3e8a' (type 'ImportExport') and returns the following error: 2020-12-01 13:17:46,784+01 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-2) [] Operation Failed: [Cannot remove VM. The VM is performing an operation on a Snapshot. Please wait for the operation to finish, and try again.] Maybe there is a recent change causing a regression here? [1] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/14137/ [2] https://github.com/oVirt/ovirt-system-tests/blob/master/basic-suite-master/… With Best Regards. Steven.

5 5

ovirt-engine has been tagged (ovirt-engine-4.4.4.3)
by Tal Nisan 02 Dec '20

02 Dec '20

1 0

[OST] Error updating ost images - packages with conflicts/broken deps
by Nir Soffer 30 Nov '20

30 Nov '20

Trying dnf update on ost system fails now with: $ sudo dnf update ... Error: Problem 1: package ost-images-el8-host-deps-installed-1-202011270303.x86_64 requires ost-images-el8-upgrade = 1-202011270303, but none of the providers can be installed - cannot install both ost-images-el8-upgrade-1-202011290315.x86_64 and ost-images-el8-upgrade-1-202011270303.x86_64 - cannot install the best update candidate for package ost-images-el8-upgrade-1-202011270303.x86_64 - cannot install the best update candidate for package ost-images-el8-host-deps-installed-1-202011270303.x86_64 Problem 2: package ost-images-el8-engine-deps-installed-1-202011270303.x86_64 requires ost-images-el8-upgrade = 1-202011270303, but none of the providers can be installed - cannot install both ost-images-el8-upgrade-1-202011290315.x86_64 and ost-images-el8-upgrade-1-202011270303.x86_64 - package ost-images-el8-engine-installed-1-202011290315.x86_64 requires ost-images-el8-upgrade = 1-202011290315, but none of the providers can be installed - cannot install the best update candidate for package ost-images-el8-engine-installed-1-202011270303.x86_64 - cannot install the best update candidate for package ost-images-el8-engine-deps-installed-1-202011270303.x86_64 Problem 3: problem with installed package ost-images-el8-host-deps-installed-1-202011270303.x86_64 - package ost-images-el8-host-deps-installed-1-202011270303.x86_64 requires ost-images-el8-upgrade = 1-202011270303, but none of the providers can be installed - cannot install both ost-images-el8-upgrade-1-202011290315.x86_64 and ost-images-el8-upgrade-1-202011270303.x86_64 - package ost-images-el8-host-installed-1-202011290315.x86_64 requires ost-images-el8-upgrade = 1-202011290315, but none of the providers can be installed - cannot install the best update candidate for package ost-images-el8-host-installed-1-202011270303.x86_64 (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) $ sudo dnf update --nobest works, but complains about: Skipping packages with conflicts: (add '--best --allowerasing' to command line to force their upgrade): ost-images-el8-upgrade x86_64 1-202011290315 ost-images 15 M Skipping packages with broken dependencies: ost-images-el8-engine-installed x86_64 1-202011290315 ost-images 1.0 G ost-images-el8-host-installed x86_64 1-202011290315 ost-images 518 M I have not tried to run ost with the updated packages yet. Nir

2 3

Planned gerrit.ovirt.org outage 24.11.2020 from 21:00 UTC
by Evgheni Dereveanchin 30 Nov '20

30 Nov '20

Hi everyone, I will be performing planned upgrades on gerrit.ovirt.org tonight. The following services will be unreachable for up to several hours: * gerrit.ovirt.org - code review service * jenkins.ovirt.org - continuous integration service Further announcements will be sent out once the outage window opens and closes. -- Regards, Evgheni Dereveanchin

3 7

Breaking change introduced in vdsm-jsonrpc-java-1.6.0
by Artur Socha 30 Nov '20

30 Nov '20

Hi, We have recently introduced the breaking change in vdsm-jsonrpc-java [1]. Reactive streams package has been replaced with java.util.concurrent.Flow (shipped with JDK). This change is released with vdsm-jsonrpc-java-1.6.0. This change had to be reflected in ovirt-engine as well (otherwise it would not compile). I was handled by this patch [2]. Unfortunately these two changes must be in sync. Please update vdsm-jsonrpc-java to 1.6.0 if you are already working on ovirt engine revision that includes [2]. [1] https://gerrit.ovirt.org/#/c/vdsm-jsonrpc-java/+/109916/ [2] https://gerrit.ovirt.org/#/c/ovirt-engine/+/112347/ Please accept my apologies for not letting you know earlier, ideally before these patches got merged. -- Artur Socha Senior Software Engineer, RHV Red Hat

1 0

Health check endpoint in the engine hanging forever on CheckDBConnection
by Marcin Sobczyk 27 Nov '20

27 Nov '20

Hi, I'm trying out (I think) the latest build of ovirt-engine in OST [1] and the basic suite fails when we do engine reconfiguration and then restart the service [2]. After restarting we wait on the health check endpoint status here [3]. This however ends with a timeout. I tried running manually: curl -D - http://engine/ovirt-engine/services/health and that command also hangs forever. In the engine log the last related entry seem to be: 2020-11-26 16:20:15,761+01 DEBUG [org.ovirt.engine.core.utils.servlet.LocaleFilter] (default task-7) [] Incoming locale 'en-US'. Filter determined locale to be 'en-US' 2020-11-26 16:20:15,761+01 DEBUG [org.ovirt.engine.core.services.HealthStatus] (default task-7) [] Health Status servlet: entry 2020-11-26 16:20:15,761+01 DEBUG [org.ovirt.engine.core.services.HealthStatus] (default task-7) [] Calling CheckDBConnection query Has anyone else also encountered that? Regards, Marcin [1] ovirt-engine-4.4.4.3-0.0.master.20201126133903.gitc2c805a2662.el8.noarch [2] https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0e9… [3] https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0e9…

4 5

ovirt-engine has been tagged (ovirt-engine-4.4.4.2)
by Tal Nisan 25 Nov '20

25 Nov '20

1 0