- Devel - oVirt List Archives

[OST failure] Basic suite for 4.2 failing with "Error: Package: ovirt-imageio-daemon"
by Ales Musil 05 Sep '18

05 Sep '18

Hello, the OST CI is failing for patch [1] with: + yum -y install ovirt-host Error: Package: ovirt-imageio-daemon-1.4.2-0.201807122005.gitae0c56d.el7.noarch (alocalsync) Requires: ovirt-imageio-common = 1.4.2 Available: ovirt-imageio-common-1.4.2-0.201807122005.gitae0c56d.el7.noarch (alocalsync) ovirt-imageio-common = 1.4.2-0.201807122005.gitae0c56d.el7 Installing: ovirt-imageio-common-1.4.4-0.el7.x86_64 (alocalsync) ovirt-imageio-common = 1.4.4-0.el7 Full log [2]. Any idea what could be possibly wrong? As the patch shouldn't be related to imageio rpm. Thank you. Best regards, Ales Musil [1] https://gerrit.ovirt.org/c/93978/ [2] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/1791/… -- ALES MUSIL Associate Software Engineer - rhv network Red Hat EMEA <https://www.redhat.com/> amusil(a)redhat.com IM: amusil <https://red.ht/sig>

3 4

Fwd: [rhev-devel] Error Message in HE master suite: "Operation Failed". Fault detail is "[Cannot add Host. You are using an unmanaged hosted engine VM ....
by Ehud Yonasi 04 Sep '18

04 Sep '18

Hey, I see failures on hosted-engine and when i was looking at the build-artifacts jobs of he-setup: http://jenkins.ovirt.org/job/ovirt-hosted-engine-setup_master_build-artifac… I can see a lot of triggering of old patches, so as far as i understand you restored that repository. Is the error below is the root cause of that recovery? Would appreciate a quick answer because we need to merge a patch to fix CQ failures because of that. Thanks in advance! Ehud Yonasi. ---------- Forwarded message --------- From: Yaniv Kaul <ykaul(a)redhat.com> Date: Tue, Sep 4, 2018 at 10:03 AM Subject: Re: [rhev-devel] Error Message in HE master suite: "Operation Failed". Fault detail is "[Cannot add Host. You are using an unmanaged hosted engine VM .... To: Galit Rosenthal <grosenth(a)redhat.com> Cc: rhev-devel <rhev-devel(a)redhat.com> Well, looking at engine.log[1], there's NPE around: ERROR [org.ovirt.engine.core.vdsbroker.monitoring.FullListAdapter] (EE-ManagedThreadFactory-engineScheduled-Thread-74) [57b53db9] Failed during parsing configuration of VM b9c1bf34-c42f-4999-97d4-8d4895826abd (<domain type='kvm' id='1'> <name>HostedEngine</name> ... </domain> ), error is: {}: java.lang.NullPointerException at org.ovirt.engine.core.vdsbroker.libvirt.VmConverter.extractCoreGeneralInfo(VmConverter.java:47) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.libvirt.VmConverter.convert(VmConverter.java:36) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.FullListAdapter.extractCoreInfo(FullListAdapter.java:86) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.FullListAdapter.lambda$dumpXmls$0(FullListAdapter.java:54) [vdsbroker.jar:] at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) [rt.jar:1.8.0_181] at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) [rt.jar:1.8.0_181] Y. [1] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… On Tue, Sep 4, 2018 at 9:21 AM, Galit Rosenthal <grosenth(a)redhat.com> wrote: > > Hi, > > This is an example of a he basic suite master that fails on : > > Error Message: > > *00:26:36* Error while running thread > > *00:26:36* Traceback (most recent call last): > > *00:26:36* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue > > *00:26:36* queue.put({'return': func()}) > > *00:26:36* File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-master/ovirt-system-tests/he-basic-suite-master/test-scenarios/002_bootstrap.py", line 379, in _add_he_host_4 > > *00:26:36* deploy_hosted_engine=True, > > *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 11530, in add > > *00:26:36* return self._internal_add(host, headers, query, wait) > > *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add > > *00:26:36* return future.wait() if wait else future > > *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait > > *00:26:36* return self._code(response) > > *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback > > *00:26:36* self._check_fault(response) > > *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault > > *00:26:36* self._raise_error(response, body) > > *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error > > *00:26:36* raise error > > *00:26:36* Error: Fault reason is "Operation Failed". Fault detail is "[Cannot add Host. You are using an unmanaged hosted engine VM. Please add the first storage domain in order to start the hosted engine import process.]". HTTP response code is 409. > > > > https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… > > Who can help me with this issue? > > Thanks, > Galit > > -- > > GALIT ROSENTHAL > > SOFTWARE ENGINEER > > Red Hat > > <https://www.redhat.com/> > > galit(a)gmail.com T: 972-9-7692230 > <https://red.ht/sig> >

2 6

Failing OST Suites
by Anton Marchukov 04 Sep '18

04 Sep '18

Hello All. This is a reminder to corresponding maintainers to check the OST suits [1], there are failing ones such as: ovirt-system-tests_ansible-suite-master ovirt-system-tests_compat-3.6-suite-master ovirt-system-tests_compat-4.0-suite-master ovirt-system-tests_compat-4.1-suite-master ovirt-system-tests_compat-4.2-suite-master ovirt-system-tests_hc-basic-suite-4.2 ovirt-system-tests_he-node-ng-suite-master If everything is not needed we might also consider disabling/deleting. Thanks. [1] https://jenkins.ovirt.org/view/oVirt%20system%20tests/

3 4

[ OST Failure Report ] [ oVirt master (ovirt-ansible-engine-setup) ] [ Aug 31 ] [ 001_upgrade_engine.test_initialize_engine ]
by Anton Marchukov 31 Aug '18

31 Aug '18

Hello All. We have a failure in OST master. The link to job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/9255/ and to the output: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/9255/testRepo… It seems like schema fails to apply during upgrade: File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/db/schema.py", line 437, in _misc raise RuntimeError(_('Engine schema refresh failed')) RuntimeError: Engine schema refresh failed I think the patch referred by the job is not the cause as ovirt-ansible-engine-setup has only release bumps recently as I see. I will reply here if I find a probably cause. -- Anton Marchukov Team Lead - Release Management - RHV DevOps - Red Hat

2 2

ovirt-guest-agent_master_build-artifacts-fc28-x86_64 Fails to Build
by Anton Marchukov 31 Aug '18

31 Aug '18

Hello All. Looks like ovirt-guest-agent_master_build-artifacts-fc28-x86_64 nolonger works, as I see it fails to build with the following error: *10:53:41* 0009:fixme:module:find_dll_file skipping L"C:\\windows\\system32\\dbghelp.dll" because of wrong architecture*10:53:42* 004a:err:msiexec:custom_action_server Failed to create custom action server pipe: 2 than it fails to recognize that and the whole job fails on a time out: *16:41:14* Build timed out (after 360 minutes). Marking the build as failed. As a result, OST for guest-agent also fails. I tried to rerun the job and got the same problem. The failing job is at [1] and I also opened JIRA issue [2] to track it. [1] http://jenkins.ovirt.org/job/ovirt-guest-agent_master_build-artifacts-fc28-… [2] https://ovirt-jira.atlassian.net/browse/OVIRT-2466 -- Anton Marchukov Team Lead - Release Management - RHV DevOps - Red Hat

3 2

s390x (mainframe) CI node is currently not available
by Barak Korren 30 Aug '18

30 Aug '18

Hi all, Unfortunately, the s390x build host we've been using had gone off-line a couple of days ago, and the ETA for having it back up is unknown. The implication of having the host down is that projects that build on s390x have their builds be stuck forever waiting for the s390x node to become available. Sine the s390x builds are missing, builds for other platforms, while being successful do not get passed to the change queue and as a result are not being released to the nightly repos. As a work around for the current situation, we need to disable the s390x builds. Here is a patch that does this for vdsm: https://gerrit.ovirt.org/c/93316/ For V2 projects the change should be done in the projects own repo, for example here is how we did it for the CI jobs of the 'jenkins' repo itself: https://gerrit.ovirt.org/c/93288/ -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

2 1

Failing Build On vdsm_master_check-patch
by Shani Leviim 29 Aug '18

29 Aug '18

Hello All, I got into the following failure during the CI build for a patch I did: 14:33:56 ====================================================================== 14:33:56 FAIL: test_sourceroute_add_remove_and_read (network.sourceroute_test.TestSourceRoute) 14:33:56 ---------------------------------------------------------------------- 14:33:56 Traceback (most recent call last): 14:33:56 File "/home/jenkins/workspace/vdsm_master_check-patch-fc28-x86_64/vdsm/tests/testValidation.py", line 193, in wrapper 14:33:56 return f(*args, **kwargs) 14:33:56 File "/home/jenkins/workspace/vdsm_master_check-patch-fc28-x86_64/vdsm/tests/network/sourceroute_test.py", line 80, in test_sourceroute_add_remove_and_read 14:33:56 self.assertEqual(2, len(routes), routes) 14:33:56 AssertionError: 2 != 0 My patch is [1] and its build is [2] [1] https://gerrit.ovirt.org/#/c/92416/ [2] http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc28-x86_64/747/ Can you please assist? Thanks! *Regards,* *Shani Leviim*

3 4

OST compatibility check failing on backup snapshot removal
by Milan Zamazal 29 Aug '18

29 Aug '18

Hi, when basic-suite-master OST is run with cluster level < 4.2, verify_backup_snapshot_removed test fails. Indeed, the snapshot removal fails, with error such as: [c784a2c3-a700-41f7-80bd-2d1e25a712c5] Failed to live merge. Top volume 15b82854-c9dd-44aa-83c9-aebe273ac6a4 is still in qemu chain [15b82854-c9dd-44aa-83c9-aebe273ac6a4, 3c21269c-e71b-4e96-977e-a9168bbba9bf] See e.g. https://jenkins.ovirt.org/job/ovirt-system-tests_compat-3.6-suite-master/52… Does anybody have an idea what might be wrong? Thanks, Milan

3 6

Re: Image Transfer mechanism queries/API support
by Ranjit DSouza 28 Aug '18

28 Aug '18

Hello oVirt Development team Not sure you got a chance to see this email, resending it. Thanks Ranjit From: Ranjit DSouza Sent: Monday, July 9, 2018 1:16 PM To: 'Nir Soffer' <nsoffer(a)redhat.com> Cc: devel <devel(a)ovirt.org>; DL-VTAS-ENG-NBU-EverestFalcons <DL-VTAS-ENG-NBU-EverestFalcons(a)veritas.com>; Navin Tah <Navin.Tah(a)veritas.com>; Sudhakar Paulzagade <Sudhakar.Paulzagade(a)veritas.com>; Pavan Chavva; Yaniv Lavi (Dary) <ylavi(a)redhat.com>; Nisan, Tal <tnisan(a)redhat.com>; Daniel Erez <derez(a)redhat.com> Subject: RE: [EXTERNAL] Re: [ovirt-devel] Image Transfer mechanism queries/API support Hi Nir Thanks for getting back! We have few follow up questions: 1. On the new ‘Allocated extents API’: Can you share the release timeline for 4.2.z? From the link below it seems like it will be available on 7/30/2018. https://www.ovirt.org/develop/release-management/releases/4.2.z/release-man… However, we thought we would double check on this. 2. If this will be available in 4.2.z, does It mean, we can assume it will be backported to 4.3 also? 3. When we downloaded the snapshot disk using Image Transfer API, the resulting format of the disk is “raw”. However, for upload, we must upload a qcow2 disk (to enable further snapshots). It means, we need to convert it first using qemu-img convert. Or is there a way we directly ask via API for a qcow2 instead? Portion of the response of “GET /storagedomains/{storagedomain:id}/disksnapshots” "format" : "raw", "shareable" : "false", "sparse" : "true", "status" : "ok", "snapshot" : { "id" : "4756036e-92aa-4ebb-ae4b-052a30cd5109" }, "actual_size" : "1345228800", "content_type" : "data", "propagate_errors" : "false", "provisioned_size" : "21474836480", "storage_type" : "image", "total_size" : "0", "wipe_after_delete" : "false", 4. When downloading a snapshot in chunks, Is there any recommended chunk size? For our study we used 512 KB. Checked the documentation too. 5. Regarding your ask on ‘Can you file RFE for this, explaining the use case and the current performance’. Since you do not recommend direct access to NFS storage, we will consider it only if we see significant performance degradation using http download. Also, if time permits, we may check if there is a significant performance benefit using Unix socket, over http download (via Rest API). But as it stands now, getting the allocated extents support (soon), will alleviate most of our performance concerns. Thanks Ranjit From: Nir Soffer [mailto:nsoffer@redhat.com] Sent: Tuesday, July 3, 2018 4:29 PM To: Ranjit DSouza <Ranjit.DSouza(a)veritas.com<mailto:Ranjit.DSouza@veritas.com>> Cc: devel <devel(a)ovirt.org<mailto:devel@ovirt.org>>; DL-VTAS-ENG-NBU-EverestFalcons <DL-VTAS-ENG-NBU-EverestFalcons(a)veritas.com<mailto:DL-VTAS-ENG-NBU-EverestFalcons@veritas.com>>; Navin Tah <Navin.Tah(a)veritas.com<mailto:Navin.Tah@veritas.com>>; Sudhakar Paulzagade <Sudhakar.Paulzagade(a)veritas.com<mailto:Sudhakar.Paulzagade@veritas.com>>; Pavan Chavva; Yaniv Lavi (Dary) <ylavi(a)redhat.com<mailto:ylavi@redhat.com>>; Nisan, Tal <tnisan(a)redhat.com<mailto:tnisan@redhat.com>>; Daniel Erez <derez(a)redhat.com<mailto:derez@redhat.com>> Subject: [EXTERNAL] Re: [ovirt-devel] Image Transfer mechanism queries/API support On Tue, Jul 3, 2018 at 11:32 AM Ranjit DSouza <Ranjit.DSouza(a)veritas.com<mailto:Ranjit.DSouza@veritas.com>> wrote: ... We had a conversation with Pavan Chavva for supporting RHV. He had suggested to contact you with queries related to oVirt APIs we plan to use. We have following queries: 1. While downloading a snapshot disk, can we identify allocated extents and download only those using oVirt API? We are able to download the disk using the Image Transfer API mechanism. However, this method downloads the entire disk including the non-allocated extents, which is a performance overhead. If this functionality does not exist at this point will it be available in near future? Hi Ranjit, There is no way to do this in current 4.2, but we plan to introduce in in 4.2.z. The API will be something like: GET /images/xxx-yyy/map ... [{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 0}, { "start": 65536, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": 65536}, { "start": 1048576, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 1048576}, { "start": 1114112, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": 1114112}, ... { "start": 5465571328, "length": 22675456, "depth": 0, "zero": false, "data": true, "offset": 5465571328}, { "start": 5488246784, "length": 954138624, "depth": 0, "zero": true, "data": false, "offset": 5488246784}, { "start": 6442385408, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 6442385408}] This is basically what you get using qemu-img map. You can test play this with this using: virt-builder Fedora-27 -o /var/tmp/fedora-27.img qemu-img map -f raw --output json /var/tmp/fedora-27.img This is the first data segment: { "start": 0, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 0} This is a hole between the first data segment and the second: { "start": 65536, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": 65536} This is the second data segment: { "start": 1048576, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": 1048576} Based on this output, you will be able to get the allocated parts of the image using: Request: GET /image/xxx-yyy HTTP/1.1 Range: bytes=0-65535 Response: HTTP/1.1 206 Partial Content Content-Range: bytes 0-65535/6442450944 <data of first segment> Request: GET /image/xxx-yyy HTTP/1.1 Range: bytes=1048576-1114111 Response: HTTP/1.1 206 Partial Content Content-Range: bytes 1048576-1114111/6442450944 <data of second segment> And so on. If you create a sparse file on your backup media, and download and write the data segments at the correct offset, you will get the a sparse version of the image as on the server side. This will work for raw or qcow2 images on NFS >= 4.2, or for qcow2 images on block storage. For older NFS versions, or raw images on block storage, we can solve the issue by reading the entire image and detecting zeroes - which is quite expensive, so I'm not sure we will implement this, maybe it will be done later. We have experimental patch using special sparse format, that can support this use case, downloading entire image in one pass. This commit message explain the format: https://gerrit.ovirt.org/#/c/85413/12//COMMIT_MSG For more info on using random I/O APIs, see: http://ovirt.github.io/ovirt-imageio/random-io (available since 4.2.3) For example code uploading sparse images see: https://github.com/oVirt/ovirt-imageio/blob/master/examples/upload For best performance, you should run your application on a oVirt host, using unix socket to communicate with imageio. See: http://ovirt.github.io/ovirt-imageio/unix-socket (will be available in 4.2.5) All this will work only for the non-active layer in a qcow2 chain. We are working now on incremental backup which will allow the same for the active layer with a running vm. This is expected in 4.3, and we may have a tech preview at some point in 4.2.z. Incremantal backup will use the similar API, allowing detection of dirty parts of an image, so you can download only the data that was changed since the last backup. Please watch and comment on the feature page: https://ovirt.org/develop/release-management/features/storage/incremental-b… We are also considering exposing images using NBD. This will allow downloading and uploading images using qemu-img from any host. This work depends on TLS-PSK support in qemu-img and qemu-nbd. You can follow this work here: https://lists.nongnu.org/archive/html/qemu-devel/2018-06/threads.html#08491 2. Is there an alternate method to transfer a snapshot to and from RHV storage? Are there other methods such as NFS share where we can download snapshot image to and from RHV storage? We don't support direct access to storage by 3rd party. You should use imageio API. Can you file RFE for this, explaning the use case and the current performance issues you experience? Nir

4 4

vdsm has been tagged (v4.20.39)
by Milan Zamazal 28 Aug '18

28 Aug '18

Vdsm 4.20.39 for oVirt 4.2.6 rc3

1 0