July 2020 - Devel - oVirt List Archives

[VDSM] Started getting virNetTLSContextCheckCertTimes:154 : The server certificate /etc/pki/vdsm/certs/vdsmcert.pem has expired
by Edward Haas 11 Jan '24

11 Jan '24

Hi, I have a VM which acts as an oVirt host with VDSM installed. I use it for testing without connecting it to Engine. Recently VDSM fails to come up and I see this error: Sep 26 17:58:18 localhost libvirtd: 2018-09-26 14:58:18.978+0000: 12176: error : virNetTLSContextCheckCertTimes:154 : The server certificate /etc/pki/vdsm/certs/vdsmcert.pem has expired Any hint on how to generate or fetch a new certificate without the help of Engine? Thanks, Edy.

3 5

Error: Adding new Host to ovirt-engine
by Ahmad Khiet 14 Aug '23

14 Aug '23

Hi, Can't add new host to ovirt engine, because the following error: 2019-06-12 12:23:09,664 p=4134 u=engine | TASK [ovirt-host-deploy-facts : Set facts] ************************************* 2019-06-12 12:23:09,684 p=4134 u=engine | ok: [10.35.1.17] => { "ansible_facts": { "ansible_python_interpreter": "/usr/bin/python2", "host_deploy_vdsm_version": "4.40.0" }, "changed": false } 2019-06-12 12:23:09,697 p=4134 u=engine | TASK [ovirt-provider-ovn-driver : Install ovs] ********************************* 2019-06-12 12:23:09,726 p=4134 u=engine | fatal: [10.35.1.17]: FAILED! => {} MSG: The conditional check 'cluster_switch == "ovs" or (ovn_central is defined and ovn_central | ipaddr and ovn_engine_cluster_version is version_compare('4.2', '>='))' failed. The error was: The ipaddr filter requires python's netaddr be installed on the ansible controller The error appears to be in '/home/engine/apps/engine/share/ovirt-engine/playbooks/roles/ovirt-provider-ovn-driver/tasks/configure.yml': line 3, column 5, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - block: - name: Install ovs ^ here 2019-06-12 12:23:09,728 p=4134 u=engine | PLAY RECAP ********************************************************************* 2019-06-12 12:23:09,728 p=4134 u=engine | 10.35.1.17 : ok=3 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 whats missing!? Thanks -- Ahmad Khiet Red Hat <https://www.redhat.com/> akhiet(a)redhat.com M: +972-54-6225629 <https://red.ht/sig>

7 10

Purging inactive maintainers from vdsm-master-maintainers
by Nir Soffer 21 Jun '21

21 Jun '21

I want to remove inactive contributors from vdsm-master-maintainers. I suggest the simple rule of 2 years of inactivity for removing from this group, based on git log. See the list below for current status: https://gerrit.ovirt.org/#/admin/groups/106,members Thoughts? Nir

6 11

Error Java SDK Issue??
by Geschwentner, Patrick 02 Apr '21

02 Apr '21

Dear Ladies and Gentlemen! I am currently working with the java-sdk and I encountered a problem. If I would like to retrieve the disk details, I get the following error: Disk currDisk = ovirtConnection.followLink(diskAttachment.disk()); The Error is occurring in this line: [cid:image001.png@01D44537.AF127FD0] The getResponst looks quiet ok. (I inspected: [cid:image002.png@01D44537.AF127FD0] and it looks ok). Error: wrong number of arguments The code is quiet similar to what you published on github (https://github.com/oVirt/ovirt-engine-sdk-java/blob/master/sdk/src/test/jav… ). Can you confirm the defect? Best regards Patrick

6 11

CentOS Stream support
by Michal Skrivanek 01 Feb '21

01 Feb '21

Hi all, we would like to ask about interest in community about oVirt moving to CentOS Stream. There were some requests before but it’s hard to see how many people would really like to see that. With CentOS releases lagging behind RHEL for months it’s interesting to consider moving to CentOS Stream as it is much more up to date and allows us to fix bugs faster, with less workarounds and overhead for maintaining old code. E.g. our current integration tests do not really pass on CentOS 8.1 and we can’t really do much about that other than wait for more up to date packages. It would also bring us closer to make oVirt run smoothly on RHEL as that is also much closer to Stream than it is to outdated CentOS. So..would you like us to support CentOS Stream? We don’t really have capacity to run 3 different platforms, would you still want oVirt to support CentOS Stream if it means “less support” for regular CentOS? There are some concerns about Stream being a bit less stable, do you share those concerns? Thank you for your comments, michal

6 12

device compatibility interface for live migration with assigned devices
by Yan Zhao 15 Sep '20

15 Sep '20

hi folks, we are defining a device migration compatibility interface that helps upper layer stack like openstack/ovirt/libvirt to check if two devices are live migration compatible. The "devices" here could be MDEVs, physical devices, or hybrid of the two. e.g. we could use it to check whether - a src MDEV can migrate to a target MDEV, - a src VF in SRIOV can migrate to a target VF in SRIOV, - a src MDEV can migration to a target VF in SRIOV. (e.g. SIOV/SRIOV backward compatibility case) The upper layer stack could use this interface as the last step to check if one device is able to migrate to another device before triggering a real live migration procedure. we are not sure if this interface is of value or help to you. please don't hesitate to drop your valuable comments. (1) interface definition The interface is defined in below way: __ userspace /\ \ / \write / read \ ________/__________ ___\|/_____________ | migration_version | | migration_version |-->check migration --------------------- --------------------- compatibility device A device B a device attribute named migration_version is defined under each device's sysfs node. e.g. (/sys/bus/pci/devices/0000\:00\:02.0/$mdev_UUID/migration_version). userspace tools read the migration_version as a string from the source device, and write it to the migration_version sysfs attribute in the target device. The userspace should treat ANY of below conditions as two devices not compatible: - any one of the two devices does not have a migration_version attribute - error when reading from migration_version attribute of one device - error when writing migration_version string of one device to migration_version attribute of the other device The string read from migration_version attribute is defined by device vendor driver and is completely opaque to the userspace. for a Intel vGPU, string format can be defined like "parent device PCI ID" + "version of gvt driver" + "mdev type" + "aggregator count". for an NVMe VF connecting to a remote storage. it could be "PCI ID" + "driver version" + "configured remote storage URL" for a QAT VF, it may be "PCI ID" + "driver version" + "supported encryption set". (to avoid namespace confliction from each vendor, we may prefix a driver name to each migration_version string. e.g. i915-v1-8086-591d-i915-GVTg_V5_8-1) (2) backgrounds The reason we hope the migration_version string is opaque to the userspace is that it is hard to generalize standard comparing fields and comparing methods for different devices from different vendors. Though userspace now could still do a simple string compare to check if two devices are compatible, and result should also be right, it's still too limited as it excludes the possible candidate whose migration_version string fails to be equal. e.g. an MDEV with mdev_type_1, aggregator count 3 is probably compatible with another MDEV with mdev_type_3, aggregator count 1, even their migration_version strings are not equal. (assumed mdev_type_3 is of 3 times equal resources of mdev_type_1). besides that, driver version + configured resources are all elements demanding to take into account. So, we hope leaving the freedom to vendor driver and let it make the final decision in a simple reading from source side and writing for test in the target side way. we then think the device compatibility issues for live migration with assigned devices can be divided into two steps: a. management tools filter out possible migration target devices. Tags could be created according to info from product specification. we think openstack/ovirt may have vendor proprietary components to create those customized tags for each product from each vendor. e.g. for Intel vGPU, with a vGPU(a MDEV device) in source side, the tags to search target vGPU are like: a tag for compatible parent PCI IDs, a tag for a range of gvt driver versions, a tag for a range of mdev type + aggregator count for NVMe VF, the tags to search target VF may be like: a tag for compatible PCI IDs, a tag for a range of driver versions, a tag for URL of configured remote storage. b. with the output from step a, openstack/ovirt/libvirt could use our proposed device migration compatibility interface to make sure the two devices are indeed live migration compatible before launching the real live migration process to start stream copying, src device stopping and target device resuming. It is supposed that this step would not bring any performance penalty as -in kernel it's just a simple string decoding and comparing -in openstack/ovirt, it could be done by extending current function check_can_live_migrate_destination, along side claiming target resources.[1] [1] https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/libvi… Thanks Yan

14 117

[ARM64] Possiblity to support oVirt on ARM64
by Zhenyu Zheng 11 Sep '20

11 Sep '20

Hi oVirt, We are currently trying to make oVirt work on ARM64 platform, since I'm quite new to oVirt community, I'm wondering what is the current status about ARM64 support in the oVirt upstream, as I saw the oVirt Wikipedia page mentioned there is an ongoing efforts to support ARM platform. We have a small team here and we are willing to also help to make this work. Thanks alot, Zhenyu Zheng

6 8

OST for 4.3 is broken on build stage
by Amit Bawer 26 Aug '20

26 Aug '20

Hi Lookup for el7 repo fails: [2020-06-22T07:46:48.800Z] ERROR: Command failed: [2020-06-22T07:46:48.800Z] # /usr/bin/yum --installroot /var/lib/mock/epel-7-ppc64le-60da95f193baa5a745076a0c6b674c73-bootstrap-6968/root/ --releasever 7 install yum yum-utils --setopt=tsflags=nocontexts [2020-06-22T07:46:48.800Z] Failed to set locale, defaulting to C [2020-06-22T07:46:48.800Z] http://cbs.centos.org/repos/virt7-ovirt-common-candidate/ppc64le/os/repodat…: [Errno 14] HTTPS Error 404 - Not Found [2020-06-22T07:46:48.800Z] Trying other mirror. [2020-06-22T07:46:48.800Z] To address this issue please refer to the below wiki article [2020-06-22T07:46:48.800Z] Taken from https://jenkins.ovirt.org/job/vdsm_standard-check-patch/21966/consoleText Thanks

2 1

implementing hotplugCd/hotunplugCd in vdsm
by Fedor Gavrilov 25 Aug '20

25 Aug '20

Hey, So in an attempt to fix change CD functionality we discovered a few other potential issues and what Nir suggested was to implement two [somewhat] new functions in VDSM: hotplug and hotunplug for CDs similar to how it works for normal disks now. Existing changeCD function will be left as is for backwards compatibility. As I found out, engine already calculates iface and index before invoking VDSM functions, so we will just pass these along with PDIV to the VDSM. Suggested flow is, let me quote: >So the complete change CD flow should be: > >1. get the previous drivespec from vm metadata >2. prepare new drivespec >3. add new drivespec to vm metadata >4. attach a new device to vm >5. teardown the previous drivespec >6. remove previous drivespec from vm metadata > >When the vm is stopped, it must do: > >1. get drive spec from vm metadata >2. teardown drivespec > >During attach, there are interesting races: >- what happens if vdsm crashes after step 2? who will teardown the volume? > maybe you need to add the new drivespec to the metadata first, >before preparing it. >- what happens if attach failed? who will remove the new drive from >the metadata? Now, what makes hotplugDisk/hotunplugDisk different? From what I understand, the flow is same there, so what difference is there as far as VDSM is concerned? If none, this means if I more or less copy that code, changing minor details and data accordingly for CDs, this should work, shouldn't it? Thanks! Fedor

4 11

Improving VM behavior in case of IO errors
by Shubha Kulkarni 16 Aug '20

16 Aug '20

Hello, In OVirt, we have a property propagate_error at the disk level that decides in case of an error, how this error be propagated to the VM. This value is maintained in the database table with the default value set as Off. The default setting(Off) results in a policy that ends up pausing the VM rather than propagating the errors to VM. There is no provision in the UI currently to configure this property for disk (images or luns). So there is no easy way to set this value. Further, even if the value is manually set to "On" in db, it gets overwriiten by UI everytime some other property is updated as described here - https://bugzilla.redhat.com/show_bug.cgi?id=1669367 Setting the value to "Off" is not ideal for multipath devices where a single path failure causes vm to pause. It puts serious restrictions for the DR situation and unlike VMWare * Hyper-V, oVirt is not able to support the DR functionality - https://bugzilla.redhat.com/show_bug.cgi?id=1314160 While we wait for RFE, the proposal here is to revise the out of the box behavior for LUNs. For LUNs, we should propagate the errors to VM rather than directly stopping those. This will allow us to handle short-term multipath outages and improve availability. This is a simple change in behavior but will have good positive impact. I would like to seek feedback about this to make sure that everyone is ok with the proposal. Thanks, Shubha

3 11

[VDSM] Trouble merging patch, related to jenkins whitelist or new RHEL CI?
by Nir Soffer 03 Aug '20

03 Aug '20

I could not merge: https://gerrit.ovirt.org/c/109402/ Although it was approved and verified, and got +1 from Continuous Integration. There is a new "RHEL-Continuous-Integration", which does not run with this change, even when I trigger the tests manually with "ci test". I tried to add +1 for "RHEL-Continuous-Integration" but this is not possible, I see: RHEL-Continuous-Integration You don't have permission to edit this label. So finally I downloaded the patch and pushed it manually. I think this patch will fix the problem: https://gerrit.ovirt.org/#/c/110576/ But I need CI experts to review this. Since we have 50 projects that need this, this probably should be fixed elsewhere, and inherited by all projects. I know that Germano was not able to trigger tests because he was missing in the jenkins whitelist, but this was fixed last week. Germano, maybe just to check that everything works for you, you can post some trivial patch? Nir

4 9

please i'm facing a with making el8
by Amit Chaudhary 30 Jul '20

30 Jul '20

ovirt.zip <https://drive.google.com/file/d/1I1Kcbnmi3IIE1taUsWEe2uQaWFWvERyE/view?usp=…> i have attached my ovirt zip. the issue im facing is that the kickstart is not generated for el8 and im not able to figure out how to use .j2 file that is being generated. i've manage to frankenstein a kickstart file that is in the zip with all the settings but its also giving error while compressing (dracut module 'livenet' cannot be found or installed) while if i search installed modules of dracut i can see livenet is installed. see see my zip it have logs of last compilation also. sir/mam it would be a great help for my college final year project project on hci and my research paper. please sir/mam please help me Amit Chaudhary +919650596355 -- *CONFIDENTIALITY NOTICE* This communication (including any attachment/s) is intended only for the use of the addressee(s) and contains information that is PRIVILEGED AND CONFIDENTIAL. Unauthorized reading, dissemination, distribution, or copying of this communication is prohibited. * DISCLAIMER* Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the University. WARNING:- This communication, including any attachments, may not be free of viruses, interceptions or interference, and may not be compatible with your systems. You should carry out your own virus checks before opening any attachment to this e-mail. The sender of this e-mail and Galgotias University shall not be liable for any damage that you may sustain as a result of viruses, incompleteness of this message, a delay in receipt of this message or computer problems experienced.

2 1

OST switching back to CirrOS
by Michal Skrivanek 30 Jul '20

30 Jul '20

Hi, FYI we’ve just switched back from CentOS to CirrOS VMs in OST basic suite, saving 900MB of data transferred all the time (and copied back and forth during test) as well as lower memory requirements (down to 128MB from 384MB, times 3 VMs). For a good measure a memory hot unplug test was added too. Let me know if you see anything unusual, other than faster run times. Thanks, michal

2 2

Error while executing action New NFS Storage Domain: Unexpected exception
by Dana Elfassy 30 Jul '20

30 Jul '20

Hi, When I'm trying to add a storage domain via the webadmin I'm getting the error message that's on the subject. in the vdsm.log I see this error: 2020-07-29 04:53:48,045-0400 ERROR (check/loop) [storage.Monitor] Error checking path /rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata (monitor:525) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 523, in _pathChecked delay = result.delay() File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line 391, in delay raise exception.MiscFileReadException(self.path, self.rc, self.err) vdsm.storage.exception.MiscFileReadException: Internal file read failure: ('/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata', 1, bytearray(b"/usr/bin/dd: failed to open \'/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata\': No such file or directory\n")) when looking at the directory I do see that metadata was created under this directory, with the following content: [delfassy@delfassy mnt]$ cd new_nfs/cddff7bb-bb0a-4fc8-8ba4-023be91d822d/dom_md/ [delfassy@delfassy dom_md]$ less metadata CLASS=Data DESCRIPTION=spider IOOPTIMEOUTSEC=10 LEASERETRIES=3 LEASETIMESEC=60 LOCKPOLICY= LOCKRENEWALINTERVALSEC=5 POOL_UUID= REMOTE_PATH=vserver-spider.eng.lab.tlv.redhat.com:/pub/delfassy/new_nfs ROLE=Regular SDUUID=cddff7bb-bb0a-4fc8-8ba4-023be91d822d TYPE=NFS VERSION=4 _SHA_CKSUM=f4953016c94f205127b00a5b10f0e7d5d9cf24f7 The permissions on the directory are: drwxrwxrwx. 3 vdsm kvm 4096 Jul 29 11:49 new_mnt Are there any other configurations I'm missing? Any idea how this issue can be resolved? Thanks, Dana

2 3

Re: Issue with ovirt 4.4 after doing some incremental backups.
by Eyal Shenitzky 29 Jul '20

29 Jul '20

According to the vdsm.log it seems that the host reports checkpoint 'ef0dfe55-c08c-4d9e-ad32-d6b6d5cbdac6' as defined checkpoint for the VM - 7-28 16:24:06,488+0200 INFO (jsonrpc/3) [api.virt] FINISH list_checkpoints return={'result': ['cd078706-84c0-4370-a6ec-654ccd6a21aa', '906fe1fd-4e81-473c-9582-de693985f462', 'd20de661-fcc1-43ff-86a3-6294a0a94e1b', '805b447e-d033-4e b1-905a-d287ca50c85a', '9ae68264-a0c9-4f75-9ffa-20c40323abe9', '8664da63-6d92-44b8-a690-04e17afb483a', 'fb6fa799-3b6f-44fa-a60e-c66ce8c76e35', 'f9f53ee7-2a8c-4dfc-a669-2dd788361eae', '7a0f37db-02bc-49c4-bfea-a03d3474f1f8', '95d8d5c7-55fc- 4c39-8a72-fc3aaec72c39', 'aa71a6e9-174b-4d8d-a2da-1a12d5e2ab2e', 'ac833f22-c8fa-4925-becc-dcf4dac81d3f', '7fc9afd1-076f-4985-b412-4ea029fd2af7', 'ae299a18-c4b0-46c4-81e1-f9583e6954f6', 'cee97ef9-6e7c-4518-91b1-dfa3b467f5c4', '51b3bf23-3cc 3-4a16-b6d9-942c6e4f7573', '0f376950-a25e-477a-93b8-d1d0ffbff716', 'ef0dfe55-c08c-4d9e-ad32-d6b6d5cbdac6'], 'status': {'code': 0, 'message': 'Done'}} from=::ffff:10.40.1.66,56474, flow_id=32ba9d9d-32c2-4652-ac01-895e92123323, vmId=116aa6e b-31a1-43db-9b1e-ad6e32fb9260 (api:54) It means as Nir said that the host assumes the corresponding bitmap is also exists in QEMU. The logs only contain a short period of time so I cannot tell if some bitmap destructive operation occurs on the VM (Live merge, cold snapshot creation etc..) Peter, is there any other log that may assist? On Wed, 29 Jul 2020 at 11:28, Łukasz Kołaciński <l.kolacinski(a)storware.eu> wrote: > Hello, > I am sending logs from vdsm, maybe they will help. I cannot find useful > libvirt logs and file libvirt.log doesn't exists. > > Best Regards, > > Łukasz Kołaciński > > Junior Java Developer > > e-mail: l.kolacinski(a)storware.eu > <m.helbert(a)storware.eu> > > > > > *[image: STORWARE]* <http://www.storware.eu/> > > > > *ul. Leszno 8/44 01-192 Warszawa www.storware.eu > <https://www.storware.eu/>* > > *[image: facebook]* <https://www.facebook.com/storware> > > *[image: twitter]* <https://twitter.com/storware> > > *[image: linkedin]* <https://www.linkedin.com/company/storware> > > *[image: Storware_Stopka_09]* > <https://www.youtube.com/channel/UCKvLitYPyAplBctXibFWrkw> > > > > *Storware Spółka z o.o. nr wpisu do ewidencji KRS dla M.St. Warszawa > 000510131* *, NIP 5213672602.** Wiadomość ta jest przeznaczona jedynie > dla osoby lub podmiotu, który jest jej adresatem i może zawierać poufne > i/lub uprzywilejowane informacje. Zakazane jest jakiekolwiek przeglądanie, > przesyłanie, rozpowszechnianie lub inne wykorzystanie tych informacji lub > podjęcie jakichkolwiek działań odnośnie tych informacji przez osoby lub > podmioty inne niż zamierzony adresat. Jeżeli Państwo otrzymali przez > pomyłkę tę informację prosimy o poinformowanie o tym nadawcy i usunięcie > tej wiadomości z wszelkich komputerów. **This message is intended only > for the person or entity to which it is addressed and may contain > confidential and/or privileged material. Any review, retransmission, > dissemination or other use of, or taking of any action in reliance upon, > this information by persons or entities other than the intended recipient > is prohibited. If you have received this message in error, please contact > the sender and remove the material from all of your computer systems.* > > ------------------------------ > *Od:* Nir Soffer <nsoffer(a)redhat.com> > *Wysłane:* środa, 29 lipca 2020 00:31 > *Do:* Łukasz Kołaciński <l.kolacinski(a)storware.eu> > *DW:* Eyal Shenitzky <eshenitz(a)redhat.com>; Mateusz Maziarz < > m.maziarz(a)storware.eu>; Marcin Kubacki <m.kubacki(a)storware.eu>; Peter > Krempa <pkrempa(a)redhat.com>; devel <devel(a)ovirt.org> > *Temat:* Re: Issue with ovirt 4.4 after doing some incremental backups. > > On Tue, Jul 28, 2020 at 4:05 PM Łukasz Kołaciński < > l.kolacinski(a)storware.eu> wrote: > > Hello, > > > Hi Łukasz > > I moved the discussion to devel(a)ovirt.org since it is more appropriate > for this issue. > > After doing a few vm backups, something breaks and I am unable to perform > any operations. I cannot do incremental backups and even full backups > doesn't work. I have this issue third time. I don't know how to fix this so > I am currently making new vms for testing purposes > > VDSM ovirt44-h2.storware.local command StartVmBackupVDS failed: Backup > Error: {'vm_id': '116aa6eb-31a1-43db-9b1e-ad6e32fb9260', 'backup': > <vdsm.virt.backup.BackupConfig object at 0x7f42602bba20>, 'reason': "Error > starting backup: internal error: unable to execute QEMU command > 'transaction': Dirty bitmap 'ef0dfe55-c08c-4d9e-ad32-d6b6d5cbdac6' not > found"} > > This means that libvirt cannot find the dirty bitmap when starting the > backup. > > When we start a backup, we get the list of checkpoints from libvirt and we > redefine all checkpoints. We assume that all redefined checkpoints have > a bitmap in qemu at the time of the redefine. > > Peter, is this assumption correct? > > If libvirt and engine agree on the existing checkpoints, we start the > backup. > In this case one of the bitmaps was missing, so the backup failed. > > We know about some flows that may cause loss of the bitmaps: > - copying disks (bitmaps are not copied yet) > - live storage migration (it copy the disks) > - deleting snapshots > - live migration may cause this, not tested yet > - unclean shutdown of the vm > - storage is not accessible when vm is terminated > > Did you do any of these operations on the tested vm? > > Some of the issues can never be fixed, like unclean shutdown or storage > issue when qemu try to write the bitmaps to disk. So your backup > application > must be able to recover from this error. > > However oVirt does not provide a useful error that enables recovery. We > have > a special error when full backup is required, but it seem that this error > is > not returned in this case, and instead we return internal error. > > Also since you cannot do a full backup after this error, I guess that > engine > did not delete the checkpoint with the missing bitmap. This is not > surprising > since the error returned from vdsm is a generic error (BackupError), so > engine > cannot tell what is the reason for the failure. > > Did you check the backup events? What was the backup completion event? > > See this example how to get backup events: > > https://github.com/oVirt/ovirt-engine-sdk/blob/4a143351fcd3cdb0df8c50888931… > > https://github.com/oVirt/ovirt-engine-sdk/blob/4a143351fcd3cdb0df8c50888931… > > Eyal, looking at the API docs: > http://ovirt.github.io/ovirt-engine-api-model/master/#types/event > > Event code is an integer. This is not usable for detecting errors since > the value > is not part of the API. We need an enum like: > > http://ovirt.github.io/ovirt-engine-api-model/4.4/#types/image_transfer_pha… > > Peter, do we have a specific error code in libvirt about missing bitmap? > we need > this to pass useful error to engine, and engine needs this error to pass > useful > error to the user. > > The error seen here is generated by: > > flags = libvirt.VIR_DOMAIN_BACKUP_BEGIN_REUSE_EXTERNAL > try: > dom.backupBegin(backup_xml, checkpoint_xml, flags=flags) > except libvirt.libvirtError as e: > raise exception.BackupError( > reason="Error starting backup: {}".format(e), > vm_id=vm.id, > backup=backup_cfg) > > Looking in the documentation: > https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBackupBegin > > There are no specified errors so we cannot detect the reason for the > failure > and return meaningful error to our caller. > > Peter, how do you suggest to recover from internal errors? how can we > tell if this temporary error that can succeed in the next attempt, or an > error > that requires starting from full backup? > > Nir > -- Regards, Eyal Shenitzky

2 1

Re: Issue with ovirt 4.4 after doing some incremental backups.
by Eyal Shenitzky 29 Jul '20

29 Jul '20

Libvirt logs need to be activated on the host in order to collect them, please try: - systemctl stop vdsmd - systemctl stop libvirtd - Under /etc/libvirt/libvirtd.conf add: - log_filters="1:qemu 1:libvirt 4:object 4:json 4:event 1:util" - log_outputs="1:file:/var/log/libvirt/libvirtd.log" - systemctl start libvirtd - systemctl start vdsmd On Wed, 29 Jul 2020 at 11:28, Łukasz Kołaciński <l.kolacinski(a)storware.eu> wrote: > Hello, > I am sending logs from vdsm, maybe they will help. I cannot find useful > libvirt logs and file libvirt.log doesn't exists. > > Best Regards, > > Łukasz Kołaciński > > Junior Java Developer > > e-mail: l.kolacinski(a)storware.eu > <m.helbert(a)storware.eu> > > > > > *[image: STORWARE]* <http://www.storware.eu/> > > > > *ul. Leszno 8/44 01-192 Warszawa www.storware.eu > <https://www.storware.eu/>* > > *[image: facebook]* <https://www.facebook.com/storware> > > *[image: twitter]* <https://twitter.com/storware> > > *[image: linkedin]* <https://www.linkedin.com/company/storware> > > *[image: Storware_Stopka_09]* > <https://www.youtube.com/channel/UCKvLitYPyAplBctXibFWrkw> > > > > *Storware Spółka z o.o. nr wpisu do ewidencji KRS dla M.St. Warszawa > 000510131* *, NIP 5213672602.** Wiadomość ta jest przeznaczona jedynie > dla osoby lub podmiotu, który jest jej adresatem i może zawierać poufne > i/lub uprzywilejowane informacje. Zakazane jest jakiekolwiek przeglądanie, > przesyłanie, rozpowszechnianie lub inne wykorzystanie tych informacji lub > podjęcie jakichkolwiek działań odnośnie tych informacji przez osoby lub > podmioty inne niż zamierzony adresat. Jeżeli Państwo otrzymali przez > pomyłkę tę informację prosimy o poinformowanie o tym nadawcy i usunięcie > tej wiadomości z wszelkich komputerów. **This message is intended only > for the person or entity to which it is addressed and may contain > confidential and/or privileged material. Any review, retransmission, > dissemination or other use of, or taking of any action in reliance upon, > this information by persons or entities other than the intended recipient > is prohibited. If you have received this message in error, please contact > the sender and remove the material from all of your computer systems.* > > ------------------------------ > *Od:* Nir Soffer <nsoffer(a)redhat.com> > *Wysłane:* środa, 29 lipca 2020 00:31 > *Do:* Łukasz Kołaciński <l.kolacinski(a)storware.eu> > *DW:* Eyal Shenitzky <eshenitz(a)redhat.com>; Mateusz Maziarz < > m.maziarz(a)storware.eu>; Marcin Kubacki <m.kubacki(a)storware.eu>; Peter > Krempa <pkrempa(a)redhat.com>; devel <devel(a)ovirt.org> > *Temat:* Re: Issue with ovirt 4.4 after doing some incremental backups. > > On Tue, Jul 28, 2020 at 4:05 PM Łukasz Kołaciński < > l.kolacinski(a)storware.eu> wrote: > > Hello, > > > Hi Łukasz > > I moved the discussion to devel(a)ovirt.org since it is more appropriate > for this issue. > > After doing a few vm backups, something breaks and I am unable to perform > any operations. I cannot do incremental backups and even full backups > doesn't work. I have this issue third time. I don't know how to fix this so > I am currently making new vms for testing purposes > > VDSM ovirt44-h2.storware.local command StartVmBackupVDS failed: Backup > Error: {'vm_id': '116aa6eb-31a1-43db-9b1e-ad6e32fb9260', 'backup': > <vdsm.virt.backup.BackupConfig object at 0x7f42602bba20>, 'reason': "Error > starting backup: internal error: unable to execute QEMU command > 'transaction': Dirty bitmap 'ef0dfe55-c08c-4d9e-ad32-d6b6d5cbdac6' not > found"} > > This means that libvirt cannot find the dirty bitmap when starting the > backup. > > When we start a backup, we get the list of checkpoints from libvirt and we > redefine all checkpoints. We assume that all redefined checkpoints have > a bitmap in qemu at the time of the redefine. > > Peter, is this assumption correct? > > If libvirt and engine agree on the existing checkpoints, we start the > backup. > In this case one of the bitmaps was missing, so the backup failed. > > We know about some flows that may cause loss of the bitmaps: > - copying disks (bitmaps are not copied yet) > - live storage migration (it copy the disks) > - deleting snapshots > - live migration may cause this, not tested yet > - unclean shutdown of the vm > - storage is not accessible when vm is terminated > > Did you do any of these operations on the tested vm? > > Some of the issues can never be fixed, like unclean shutdown or storage > issue when qemu try to write the bitmaps to disk. So your backup > application > must be able to recover from this error. > > However oVirt does not provide a useful error that enables recovery. We > have > a special error when full backup is required, but it seem that this error > is > not returned in this case, and instead we return internal error. > > Also since you cannot do a full backup after this error, I guess that > engine > did not delete the checkpoint with the missing bitmap. This is not > surprising > since the error returned from vdsm is a generic error (BackupError), so > engine > cannot tell what is the reason for the failure. > > Did you check the backup events? What was the backup completion event? > > See this example how to get backup events: > > https://github.com/oVirt/ovirt-engine-sdk/blob/4a143351fcd3cdb0df8c50888931… > > https://github.com/oVirt/ovirt-engine-sdk/blob/4a143351fcd3cdb0df8c50888931… > > Eyal, looking at the API docs: > http://ovirt.github.io/ovirt-engine-api-model/master/#types/event > > Event code is an integer. This is not usable for detecting errors since > the value > is not part of the API. We need an enum like: > > http://ovirt.github.io/ovirt-engine-api-model/4.4/#types/image_transfer_pha… > > Peter, do we have a specific error code in libvirt about missing bitmap? > we need > this to pass useful error to engine, and engine needs this error to pass > useful > error to the user. > > The error seen here is generated by: > > flags = libvirt.VIR_DOMAIN_BACKUP_BEGIN_REUSE_EXTERNAL > try: > dom.backupBegin(backup_xml, checkpoint_xml, flags=flags) > except libvirt.libvirtError as e: > raise exception.BackupError( > reason="Error starting backup: {}".format(e), > vm_id=vm.id, > backup=backup_cfg) > > Looking in the documentation: > https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBackupBegin > > There are no specified errors so we cannot detect the reason for the > failure > and return meaningful error to our caller. > > Peter, how do you suggest to recover from internal errors? how can we > tell if this temporary error that can succeed in the next attempt, or an > error > that requires starting from full backup? > > Nir > -- Regards, Eyal Shenitzky

2 1

Re: Issue with ovirt 4.4 after doing some incremental backups.
by Nir Soffer 29 Jul '20

29 Jul '20

On Tue, Jul 28, 2020 at 4:05 PM Łukasz Kołaciński <l.kolacinski(a)storware.eu> wrote: > Hello, > Hi Łukasz I moved the discussion to devel(a)ovirt.org since it is more appropriate for this issue. After doing a few vm backups, something breaks and I am unable to perform > any operations. I cannot do incremental backups and even full backups > doesn't work. I have this issue third time. I don't know how to fix this so > I am currently making new vms for testing purposes > > VDSM ovirt44-h2.storware.local command StartVmBackupVDS failed: Backup > Error: {'vm_id': '116aa6eb-31a1-43db-9b1e-ad6e32fb9260', 'backup': > <vdsm.virt.backup.BackupConfig object at 0x7f42602bba20>, 'reason': "Error > starting backup: internal error: unable to execute QEMU command > 'transaction': Dirty bitmap 'ef0dfe55-c08c-4d9e-ad32-d6b6d5cbdac6' not > found"} > > This means that libvirt cannot find the dirty bitmap when starting the backup. When we start a backup, we get the list of checkpoints from libvirt and we redefine all checkpoints. We assume that all redefined checkpoints have a bitmap in qemu at the time of the redefine. Peter, is this assumption correct? If libvirt and engine agree on the existing checkpoints, we start the backup. In this case one of the bitmaps was missing, so the backup failed. We know about some flows that may cause loss of the bitmaps: - copying disks (bitmaps are not copied yet) - live storage migration (it copy the disks) - deleting snapshots - live migration may cause this, not tested yet - unclean shutdown of the vm - storage is not accessible when vm is terminated Did you do any of these operations on the tested vm? Some of the issues can never be fixed, like unclean shutdown or storage issue when qemu try to write the bitmaps to disk. So your backup application must be able to recover from this error. However oVirt does not provide a useful error that enables recovery. We have a special error when full backup is required, but it seem that this error is not returned in this case, and instead we return internal error. Also since you cannot do a full backup after this error, I guess that engine did not delete the checkpoint with the missing bitmap. This is not surprising since the error returned from vdsm is a generic error (BackupError), so engine cannot tell what is the reason for the failure. Did you check the backup events? What was the backup completion event? See this example how to get backup events: https://github.com/oVirt/ovirt-engine-sdk/blob/4a143351fcd3cdb0df8c50888931… https://github.com/oVirt/ovirt-engine-sdk/blob/4a143351fcd3cdb0df8c50888931… Eyal, looking at the API docs: http://ovirt.github.io/ovirt-engine-api-model/master/#types/event Event code is an integer. This is not usable for detecting errors since the value is not part of the API. We need an enum like: http://ovirt.github.io/ovirt-engine-api-model/4.4/#types/image_transfer_pha… Peter, do we have a specific error code in libvirt about missing bitmap? we need this to pass useful error to engine, and engine needs this error to pass useful error to the user. The error seen here is generated by: flags = libvirt.VIR_DOMAIN_BACKUP_BEGIN_REUSE_EXTERNAL try: dom.backupBegin(backup_xml, checkpoint_xml, flags=flags) except libvirt.libvirtError as e: raise exception.BackupError( reason="Error starting backup: {}".format(e), vm_id=vm.id, backup=backup_cfg) Looking in the documentation: https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBackupBegin There are no specified errors so we cannot detect the reason for the failure and return meaningful error to our caller. Peter, how do you suggest to recover from internal errors? how can we tell if this temporary error that can succeed in the next attempt, or an error that requires starting from full backup? Nir

3 2

ovirt-engine has been tagged (ovirt-engine-4.4.2)
by Tal Nisan 29 Jul '20

29 Jul '20

3 2

ovirt-engine has been tagged (ovirt-engine-4.3.11.3)
by Tal Nisan 28 Jul '20

28 Jul '20

1 0

Engine on Fedora 32 (help needed)
by Pavel Bar 27 Jul '20

27 Jul '20

Hi, I upgraded the OS on my laptop from Fedora 30 to Fedora 32 and now have issues with running an engine - some libraries are missing. I know that the new Fedora releases are not supported. I couldn't downgrade back to Fedora 30 and I don't want to totally reinstall the OS (either to an old Fedora 30 or to a new CentOS 8.2). Issues that I encountered: 1) Some repositories do not exist for Fedora 32. Copr repo for ovirt-katello owned by sbonazzo 383 B/s | 341 B 00:00 Errors during downloading metadata for repository '*sbonazzo-ovirt-katello* ': - Status code: 404 for https://copr-be.cloud.fedoraproject.org/results/sbonazzo/ovirt-katello/fedo… (IP: 34.227.76.72) Error: Failed to download metadata for repo 'sbonazzo-ovirt-katello': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried Copr repo for ovirt-cockpit-sso owned by mlibra 380 B/s | 341 B 00:00 Errors during downloading metadata for repository ' *mlibra-ovirt-cockpit-sso*': - Status code: 404 for https://copr-be.cloud.fedoraproject.org/results/mlibra/ovirt-cockpit-sso/fe… (IP: 34.227.76.72) Error: Failed to download metadata for repo 'mlibra-ovirt-cockpit-sso': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried Copr repo for ovsdbapp owned by mdbarroso 339 B/s | 341 B 00:01 Errors during downloading metadata for repository '*mdbarroso-ovsdbapp*': - Status code: 404 for https://copr-be.cloud.fedoraproject.org/results/mdbarroso/ovsdbapp/fedora-3… (IP: 34.227.76.72) Error: Failed to download metadata for repo 'mdbarroso-ovsdbapp': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried Ignoring repositories: *sbonazzo-ovirt-katello*, *mlibra-ovirt-cockpit-sso*, *mdbarroso-ovsdbapp* Workaround (at least stopped the errors): I updated the " /etc/yum.repos.d/ovirt-master-dependencies.repo" and changed the " $releasever" into the hardcoded "30" for the 3 repositories above. 2) Unable to install "python3-otopi" and probably some other libraries. Workaround that didn't fully work: I updated " /etc/yum.repos.d/ovirt-master-snapshot.repo" and changed the "$releasever" into the hardcoded "30" for both "ovirt-master-snapshot" & " ovirt-master-snapshot-static". Now it finds "python3-otopi", but complains about a missing "python(abi)". The latter is actually installed, but I guess there is a collision between fc30 vs. fc32 :( pbar@laptop /etc/yum.repos.d $ sudo dnf install -y python3-otopi Latest oVirt master nightly snapshot 1.8 kB/s | 3.0 kB 00:01 Latest oVirt master additional nightly snapshot 1.8 kB/s | 3.0 kB 00:01 Error: Problem: conflicting requests - nothing provides python(abi) = 3.7 needed by python3-otopi-1.9.3-0.0.master.20200607140659.git775c2c5.*fc30*.noarch (try to add '--skip-broken' to skip uninstallable packages) pbar@laptop /etc/yum.repos.d $ sudo dnf install -y "python(abi)" Last metadata expiration check: 0:00:19 ago on Mon 27 Jul 2020 03:13:34 PM IDT. Package python27-2.7.18-2.*fc32*.x86_64 is already installed. Package python3-3.8.3-2.*fc32*.x86_64 is already installed. Dependencies resolved. Nothing to do. Complete! Any suggestions that do not require a clean new OS install? Thank you in advance! Pavel

3 4

Upgrade GWT to 2.9.0
by Radoslaw Szwajkowski 27 Jul '20

27 Jul '20

Hi all, we plan to bump the GWT version to 2.9.0 (patch [1]). Key changes: 1. support for building with Java 11 2. removing support for classic dev mode ( use super dev mode instead) 3. accumulated improvements and bug fixes (from versions 2.8.1, 2.8.2, 2.9.0) Please review and check if your areas are impacted. For details see BZ [2] best regards, radek [1] https://gerrit.ovirt.org/110359/ [2] https://bugzilla.redhat.com/1860309

2 2

vdsm.storage.exception.UnknownTask: Task id unknown (was: [oVirt Jenkins] ovirt-system-tests_he-basic-suite-master - Build # 1641 - Still Failing!)
by Yedidyah Bar David 21 Jul '20

21 Jul '20

On Wed, Jun 17, 2020 at 6:28 AM <jenkins(a)jenkins.phx.ovirt.org> wrote: > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1641/ This one failed while trying to create the disk image for the hosted-egnine VM: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1641… : 2020-06-16 23:03:20,527-0400 INFO ansible task start {'status': 'OK', 'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', 'ansible_task': 'ovirt.hosted_engine_setup : Add HE disks'} ... 2020-06-16 23:14:12,702-0400 DEBUG var changed: host "localhost" var "add_disks" type "<class 'dict'>" value: "{ ... "msg": "Timeout exceed while waiting on result state of the entity." https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1641… : 2020-06-16 23:03:22,612-04 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (default task-1) [16c24599-0048-44eb-a410-d39b7ce98712] CommandMultiAsyncTasks::attachTask: Attaching task '6b2a7648-748c-430b-94b6-5e3f719df2ac' to command 'fa81759d-c57a-4237-81e0-beb210faa64d'. 2020-06-16 23:03:22,659-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (default task-1) [16c24599-0048-44eb-a410-d39b7ce98712] Adding task '6b2a7648-748c-430b-94b6-5e3f719df2ac' (Parent Command 'AddImageFromScratch', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet.. 2020-06-16 23:03:22,699-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (default task-1) [16c24599-0048-44eb-a410-d39b7ce98712] BaseAsyncTask::startPollingTask: Starting to poll task '6b2a7648-748c-430b-94b6-5e3f719df2ac'. ... 2020-06-16 23:03:25,835-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] SPMAsyncTask::PollTask: Polling task '6b2a7648-748c-430b-94b6-5e3f719df2ac' (Parent Command 'AddImageFromScratch', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'success'. 2020-06-16 23:03:25,863-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] BaseAsyncTask::onTaskEndSuccess: Task '6b2a7648-748c-430b-94b6-5e3f719df2ac' (Parent Command 'AddImageFromScratch', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. But then: 2020-06-16 23:03:25,897-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-29) [16c24599-0048-44eb-a410-d39b7ce98712] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'AddImageFromScratch' succeeded, clearing tasks. 2020-06-16 23:03:25,897-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-29) [16c24599-0048-44eb-a410-d39b7ce98712] SPMAsyncTask::ClearAsyncTask: Attempting to clear task '6b2a7648-748c-430b-94b6-5e3f719df2ac' 2020-06-16 23:03:25,899-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-29) [16c24599-0048-44eb-a410-d39b7ce98712] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='3bcde3b4-b044-11ea-bbb6-5452c0a8c863', ignoreFailoverLimit='false', taskId='6b2a7648-748c-430b-94b6-5e3f719df2ac'}), log id: 481c2d3d 2020-06-16 23:03:25,900-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-29) [16c24599-0048-44eb-a410-d39b7ce98712] START, HSMClearTaskVDSCommand(HostName = lago-he-basic-suite-master-host-0, HSMTaskGuidBaseVDSCommandParameters:{hostId='85ecc51c-f2cb-46a1-9452-fd487399d8dd', taskId='6b2a7648-748c-430b-94b6-5e3f719df2ac'}), log id: 17360b3d ... 2020-06-16 23:03:26,054-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-29) [16c24599-0048-44eb-a410-d39b7ce98712] BaseAsyncTask::removeTaskFromDB: Removed task '6b2a7648-748c-430b-94b6-5e3f719df2ac' from DataBase But then: 2020-06-16 23:03:26,315-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-55) [7fe7b467] Command 'UploadStreamVDSCommand(HostName = lago-he-basic-suite-master-host-0, UploadStreamVDSCommandParameters:{hostId='85ecc51c-f2cb-46a1-9452-fd487399d8dd'})' execution failed: javax.net.ssl.SSLPeerUnverifiedException: Certificate for <lago-he-basic-suite-master-host-0.lago.local> doesn't match any of the subject alternative names: [lago-he-basic-suite-master-host-0.lago.local] 2020-06-16 23:03:26,315-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-55) [7fe7b467] FINISH, UploadStreamVDSCommand, return: , log id: 7e3a3e80 2020-06-16 23:03:26,316-04 ERROR [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-55) [7fe7b467] Command 'org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: javax.net.ssl.SSLPeerUnverifiedException: Certificate for <lago-he-basic-suite-master-host-0.lago.local> doesn't match any of the subject alternative names: [lago-he-basic-suite-master-host-0.lago.local] (Failed with error VDS_NETWORK_ERROR and code 5022) Any idea why? Anything changed in how we check the certificate? Perhaps related to upgrade to CentOS 8.2? And, how come it failed only this late? Don't we check the certificate earlier? Anyway, this left the host in "not responding" state, so: 2020-06-16 23:03:29,994-04 ERROR [org.ovirt.engine.core.bll.storage.disk.AddDiskCommandCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-79) [16c24599-0048-44eb-a410-d39b7ce98712] Failed to get volume info: org.ovirt.engine.core.common.errors.EngineException: EngineException: No host was found to perform the operation (Failed with error RESOURCE_MANAGER_VDS_NOT_FOUND and code 5004) And perhaps due to an unrelated issue, also: 2020-06-16 23:03:31,177-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMRevertTaskVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [16c24599-0048-44eb-a410-d39b7ce98712] Trying to revert unknown task '6b2a7648-748c-430b-94b6-5e3f719df2ac' I looked a bit also at: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1641… and see there some relevant stuff, but nothing I can spot about the root cause (e.g. the word "cert" does not appear there). Can anyone please have a look? Thanks. > Build Number: 1641 > Build Status: Still Failing > Triggered By: Started by timer > > ------------------------------------- > Changes Since Last Success: > ------------------------------------- > Changes for Build #1633 > [Marcin Sobczyk] ost-images: Drop rebasing of qcows > > [Ehud Yonasi] mock: fix yum repos injection. > > [Ehud Yonasi] onboard ost-images to stdci. > > > Changes for Build #1634 > [Marcin Sobczyk] ost-images: Drop rebasing of qcows > > > Changes for Build #1635 > [Marcin Sobczyk] ost-images: Drop rebasing of qcows > > > Changes for Build #1636 > [Marcin Sobczyk] ost-images: Drop rebasing of qcows > > > Changes for Build #1637 > [Marcin Sobczyk] ost-images: Drop rebasing of qcows > > > Changes for Build #1638 > [Marcin Sobczyk] ost-images: Drop rebasing of qcows > > [Ehud Yonasi] stdci_runner: update templates node to ost-images. > > > Changes for Build #1639 > [Marcin Sobczyk] ost-images: Drop rebasing of qcows > > > Changes for Build #1640 > [Yedidyah Bar David] Allow engine 20 minutes to come up after VM restart > > > Changes for Build #1641 > [Michal Skrivanek] test live storage migration again > > [Ehud Yonasi] poll: add ost-images to nightly. > > > > > ----------------- > Failed Tests: > ----------------- > No tests ran. -- Didi

5 12

Check OVF_STORE volume status task failures
by Artem Hrechanychenko 21 Jul '20

21 Jul '20

Hi all, maybe I miss some information, but I'm still have troubles with he installation using ost CI. Is that already fixed ? https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/10415/ https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/10415… 2020-07-20 06:36:04,455-0400 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:111 TASK [ovirt.hosted_engine_setup : Check OVF_STORE volume status] 2020-07-20 06:40:22,815-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:105 {'results': [{'cmd': ['vdsm-client', 'Volume', 'getInfo', 'storagepoolID=03a466fe-ca73-11ea-b77e-5452c0a8c863', 'storagedomainID=6685cca5-f0e1-4831-acdf-6f7b50596142', 'imageID=4d2f7009-5b79-4b44-b0ef-e152bc51649f', 'volumeID=7b953d0e-662d-4e72-9fdc-823ea867262b'], 'stdout': '{\n "apparentsize": "134217728",\n "capacity": "134217728",\n "children": [],\n "ctime": "1595241259",\n "description": "{\\"Updated\\":false,\\"Last Updated\\":null,\\"Storage Domains\\":[{\\"uuid\\":\\"6685cca5-f0e1-4831-acdf-6f7b50596142\\"}],\\"Disk Description\\":\\"OVF_STORE\\"}",\n "disktype": "OVFS",\n "domain": "6685cca5-f0e1-4831-acdf-6f7b50596142",\n "format": "RAW",\n "generation": 0,\n "image": "4d2f7009-5b79-4b44-b0ef-e152bc51649f",\n "lease": {\n "offset": 0,\n "owners": [],\n "path": "/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/6685cca5-f0e1-4831-acdf-6f7b50596142/images/4d2f7009-5b79-4b44-b0ef-e152bc51649f/7b953d0e-662d-4e72-9fdc-823ea867262b.lease",\n "version": null\n },\n "legality": "LEGAL",\n "mtime": "0",\n "parent": "00000000-0000-0000-0000-000000000000",\n "pool": "",\n "status": "OK",\n "truesize": "134217728",\n "type": "PREALLOCATED",\n "uuid": "7b953d0e-662d-4e72-9fdc-823ea867262b",\n "voltype": "LEAF"\n}', 'stderr': '', 'rc': 0, 'start': '2020-07-20 06:38:13.456845', 'end': '2020-07-20 06:38:13.897280', 'delta': '0:00:00.440435', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'vdsm-client Volume getInfo storagepoolID=03a466fe-ca73-11ea-b77e-5452c0a8c863 storagedomainID=6685cca5-f0e1-4831-acdf-6f7b50596142 imageID=4d2f7009-5b79-4b44-b0ef-e152bc51649f volumeID=7b953d0e-662d-4e72-9fdc-823ea867262b', 'warn': True, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['{', ' "apparentsize": "134217728",', ' "capacity": "134217728",', ' "children": [],', ' "ctime": "1595241259",', ' "description": "{\\"Updated\\":false,\\"Last Updated\\":null,\\"Storage Domains\\":[{\\"uuid\\":\\"6685cca5-f0e1-4831-acdf-6f7b50596142\\"}],\\"Disk Description\\":\\"OVF_STORE\\"}",', ' "disktype": "OVFS",', ' "domain": "6685cca5-f0e1-4831-acdf-6f7b50596142",', ' "format": "RAW",', ' "generation": 0,', ' "image": "4d2f7009-5b79-4b44-b0ef-e152bc51649f",', ' "lease": {', ' "offset": 0,', ' "owners": [],', ' "path": "/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/6685cca5-f0e1-4831-acdf-6f7b50596142/images/4d2f7009-5b79-4b44-b0ef-e152bc51649f/7b953d0e-662d-4e72-9fdc-823ea867262b.lease",', ' "version": null', ' },', ' "legality": "LEGAL",', ' "mtime": "0",', ' "parent": "00000000-0000-0000-0000-000000000000",', ' "pool": "",', ' "status": "OK",', ' "truesize": "134217728",', ' "type": "PREALLOCATED",', ' "uuid": "7b953d0e-662d-4e72-9fdc-823ea867262b",', ' "voltype": "LEAF"', '}'], 'stderr_lines': [], '_ansible_no_log': False, 'failed': True, 'attempts': 12, 'item': {'name': 'OVF_STORE', 'image_id': '7b953d0e-662d-4e72-9fdc-823ea867262b', 'id': '4d2f7009-5b79-4b44-b0ef-e152bc51649f'}, 'ansible_loop_var': 'item', '_ansible_item_label': {'name': 'OVF_STORE', 'image_id': '7b953d0e-662d-4e72-9fdc-823ea867262b', 'id': '4d2f7009-5b79-4b44-b0ef-e152bc51649f'}}, {'cmd': ['vdsm-client', 'Volume', 'getInfo', 'storagepoolID=03a466fe-ca73-11ea-b77e-5452c0a8c863', 'storagedomainID=6685cca5-f0e1-4831-acdf-6f7b50596142', 'imageID=044e384a-dedf-4589-8dfb-beca170138ee', 'volumeID=033d64fd-6f93-42be-84bc-082b03095ef3'], 'stdout': '{\n "apparentsize": "134217728",\n "capacity": "134217728",\n "children": [],\n "ctime": "1595241260",\n "description": "{\\"Updated\\":false,\\"Last Updated\\":null,\\"Storage Domains\\":[{\\"uuid\\":\\"6685cca5-f0e1-4831-acdf-6f7b50596142\\"}],\\"Disk Description\\":\\"OVF_STORE\\"}",\n "disktype": "OVFS",\n "domain": "6685cca5-f0e1-4831-acdf-6f7b50596142",\n "format": "RAW",\n "generation": 0,\n "image": "044e384a-dedf-4589-8dfb-beca170138ee",\n "lease": {\n "offset": 0,\n "owners": [],\n "path": "/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/6685cca5-f0e1-4831-acdf-6f7b50596142/images/044e384a-dedf-4589-8dfb-beca170138ee/033d64fd-6f93-42be-84bc-082b03095ef3.lease",\n "version": null\n },\n "legality": "LEGAL",\n "mtime": "0",\n "parent": "00000000-0000-0000-0000-000000000000",\n "pool": "",\n "status": "OK",\n "truesize": "134217728",\n "type": "PREALLOCATED",\n "uuid": "033d64fd-6f93-42be-84bc-082b03095ef3",\n "voltype": "LEAF"\n}', 'stderr': '', 'rc': 0, 'start': '2020-07-20 06:40:22.272462', 'end': '2020-07-20 06:40:22.621943', 'delta': '0:00:00.349481', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'vdsm-client Volume getInfo storagepoolID=03a466fe-ca73-11ea-b77e-5452c0a8c863 storagedomainID=6685cca5-f0e1-4831-acdf-6f7b50596142 imageID=044e384a-dedf-4589-8dfb-beca170138ee volumeID=033d64fd-6f93-42be-84bc-082b03095ef3', 'warn': True, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['{', ' "apparentsize": "134217728",', ' "capacity": "134217728",', ' "children": [],', ' "ctime": "1595241260",', ' "description": "{\\"Updated\\":false,\\"Last Updated\\":null,\\"Storage Domains\\":[{\\"uuid\\":\\"6685cca5-f0e1-4831-acdf-6f7b50596142\\"}],\\"Disk Description\\":\\"OVF_STORE\\"}",', ' "disktype": "OVFS",', ' "domain": "6685cca5-f0e1-4831-acdf-6f7b50596142",', ' "format": "RAW",', ' "generation": 0,', ' "image": "044e384a-dedf-4589-8dfb-beca170138ee",', ' "lease": {', ' "offset": 0,', ' "owners": [],', ' "path": "/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/6685cca5-f0e1-4831-acdf-6f7b50596142/images/044e384a-dedf-4589-8dfb-beca170138ee/033d64fd-6f93-42be-84bc-082b03095ef3.lease",', ' "version": null', ' },', ' "legality": "LEGAL",', ' "mtime": "0",', ' "parent": "00000000-0000-0000-0000-000000000000",', ' "pool": "",', ' "status": "OK",', ' "truesize": "134217728",', ' "type": "PREALLOCATED",', ' "uuid": "033d64fd-6f93-42be-84bc-082b03095ef3",', ' "voltype": "LEAF"', '}'], 'stderr_lines': [], '_ansible_no_log': False, 'failed': True, 'attempts': 12, 'item': {'name': 'OVF_STORE', 'image_id': '033d64fd-6f93-42be-84bc-082b03095ef3', 'id': '044e384a-dedf-4589-8dfb-beca170138ee'}, 'ansible_loop_var': 'item', '_ansible_item_label': {'name': 'OVF_STORE', 'image_id': '033d64fd-6f93-42be-84bc-082b03095ef3', 'id': '044e384a-dedf-4589-8dfb-beca170138ee'}}], 'changed': True, 'msg': 'All items completed'} 2020-07-20 06:40:22,919-0400 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'cmd': ['vdsm-client', 'Volume', 'getInfo', 'storagepoolID=03a466fe-ca73-11ea-b77e-5452c0a8c863', 'storagedomainID=6685cca5-f0e1-4831-acdf-6f7b50596142', 'imageID=4d2f7009-5b79-4b44-b0ef-e152bc51649f', 'volumeID=7b953d0e-662d-4e72-9fdc-823ea867262b'], 'stdout': '{\n "apparentsize": "134217728",\n "capacity": "134217728",\n "children": [],\n "ctime": "1595241259",\n "description": "{\\"Updated\\":false,\\"Last Updated\\":null,\\"Storage Domains\\":[{\\"uuid\\":\\"6685cca5-f0e1-4831-acdf-6f7b50596142\\"}],\\"Disk Description\\":\\"OVF_STORE\\"}",\n "disktype": "OVFS",\n "domain": "6685cca5-f0e1-4831-acdf-6f7b50596142",\n "format": "RAW",\n "generation": 0,\n "image": "4d2f7009-5b79-4b44-b0ef-e152bc51649f",\n "lease": {\n "offset": 0,\n "owners": [],\n "path": "/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/6685cca5-f0e1-4831-acdf-6f7b50596142/images/4d2f7009-5b79-4b44-b0ef-e152bc51649f/7b953d0e-662d-4e72-9fdc-823ea867262b.lease",\n "version": null\n },\n "legality": "LEGAL",\n "mtime": "0",\n "parent": "00000000-0000-0000-0000-000000000000",\n "pool": "",\n "status": "OK",\n "truesize": "134217728",\n "type": "PREALLOCATED",\n "uuid": "7b953d0e-662d-4e72-9fdc-823ea867262b",\n "voltype": "LEAF"\n}', 'stderr': '', 'rc': 0, 'start': '2020-07-20 06:38:13.456845', 'end': '2020-07-20 06:38:13.897280', 'delta': '0:00:00.440435', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'vdsm-client Volume getInfo storagepoolID=03a466fe-ca73-11ea-b77e-5452c0a8c863 storagedomainID=6685cca5-f0e1-4831-acdf-6f7b50596142 imageID=4d2f7009-5b79-4b44-b0ef-e152bc51649f volumeID=7b953d0e-662d-4e72-9fdc-823ea867262b', 'warn': True, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['{', ' "apparentsize": "134217728",', ' "capacity": "134217728",', ' "children": [],', ' "ctime": "1595241259",', ' "description": "{\\"Updated\\":false,\\"Last Updated\\":null,\\"Storage Domains\\":[{\\"uuid\\":\\"6685cca5-f0e1-4831-acdf-6f7b50596142\\"}],\\"Disk Description\\":\\"OVF_STORE\\"}",', ' "disktype": "OVFS",', ' "domain": "6685cca5-f0e1-4831-acdf-6f7b50596142",', ' "format": "RAW",', ' "generation": 0,', ' "image": "4d2f7009-5b79-4b44-b0ef-e152bc51649f",', ' "lease": {', ' "offset": 0,', ' "owners": [],', ' "path": "/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/6685cca5-f0e1-4831-acdf-6f7b50596142/images/4d2f7009-5b79-4b44-b0ef-e152bc51649f/7b953d0e-662d-4e72-9fdc-823ea867262b.lease",', ' "version": null', ' },', ' "legality": "LEGAL",', ' "mtime": "0",', ' "parent": "00000000-0000-0000-0000-000000000000",', ' "pool": "",', ' "status": "OK",', ' "truesize": "134217728",', ' "type": "PREALLOCATED",', ' "uuid": "7b953d0e-662d-4e72-9fdc-823ea867262b",', ' "voltype": "LEAF"', '}'], 'stderr_lines': [], '_ansible_no_log': False, 'failed': True, 'attempts': 12, 'item': {'name': 'OVF_STORE', 'image_id': '7b953d0e-662d-4e72-9fdc-823ea867262b', 'id': '4d2f7009-5b79-4b44-b0ef-e152bc51649f'}, 'ansible_loop_var': 'item', '_ansible_item_label': {'name': 'OVF_STORE', 'image_id': '7b953d0e-662d-4e72-9fdc-823ea867262b', 'id': '4d2f7009-5b79-4b44-b0ef-e152bc51649f'}} 2020-07-20 06:40:23,026-0400 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'cmd': ['vdsm-client', 'Volume', 'getInfo', 'storagepoolID=03a466fe-ca73-11ea-b77e-5452c0a8c863', 'storagedomainID=6685cca5-f0e1-4831-acdf-6f7b50596142', 'imageID=044e384a-dedf-4589-8dfb-beca170138ee', 'volumeID=033d64fd-6f93-42be-84bc-082b03095ef3'], 'stdout': '{\n "apparentsize": "134217728",\n "capacity": "134217728",\n "children": [],\n "ctime": "1595241260",\n "description": "{\\"Updated\\":false,\\"Last Updated\\":null,\\"Storage Domains\\":[{\\"uuid\\":\\"6685cca5-f0e1-4831-acdf-6f7b50596142\\"}],\\"Disk Description\\":\\"OVF_STORE\\"}",\n "disktype": "OVFS",\n "domain": "6685cca5-f0e1-4831-acdf-6f7b50596142",\n "format": "RAW",\n "generation": 0,\n "image": "044e384a-dedf-4589-8dfb-beca170138ee",\n "lease": {\n "offset": 0,\n "owners": [],\n "path": "/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/6685cca5-f0e1-4831-acdf-6f7b50596142/images/044e384a-dedf-4589-8dfb-beca170138ee/033d64fd-6f93-42be-84bc-082b03095ef3.lease",\n "version": null\n },\n "legality": "LEGAL",\n "mtime": "0",\n "parent": "00000000-0000-0000-0000-000000000000",\n "pool": "",\n "status": "OK",\n "truesize": "134217728",\n "type": "PREALLOCATED",\n "uuid": "033d64fd-6f93-42be-84bc-082b03095ef3",\n "voltype": "LEAF"\n}', 'stderr': '', 'rc': 0, 'start': '2020-07-20 06:40:22.272462', 'end': '2020-07-20 06:40:22.621943', 'delta': '0:00:00.349481', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'vdsm-client Volume getInfo storagepoolID=03a466fe-ca73-11ea-b77e-5452c0a8c863 storagedomainID=6685cca5-f0e1-4831-acdf-6f7b50596142 imageID=044e384a-dedf-4589-8dfb-beca170138ee volumeID=033d64fd-6f93-42be-84bc-082b03095ef3', 'warn': True, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['{', ' "apparentsize": "134217728",', ' "capacity": "134217728",', ' "children": [],', ' "ctime": "1595241260",', ' "description": "{\\"Updated\\":false,\\"Last Updated\\":null,\\"Storage Domains\\":[{\\"uuid\\":\\"6685cca5-f0e1-4831-acdf-6f7b50596142\\"}],\\"Disk Description\\":\\"OVF_STORE\\"}",', ' "disktype": "OVFS",', ' "domain": "6685cca5-f0e1-4831-acdf-6f7b50596142",', ' "format": "RAW",', ' "generation": 0,', ' "image": "044e384a-dedf-4589-8dfb-beca170138ee",', ' "lease": {', ' "offset": 0,', ' "owners": [],', ' "path": "/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/6685cca5-f0e1-4831-acdf-6f7b50596142/images/044e384a-dedf-4589-8dfb-beca170138ee/033d64fd-6f93-42be-84bc-082b03095ef3.lease",', ' "version": null', ' },', ' "legality": "LEGAL",', ' "mtime": "0",', ' "parent": "00000000-0000-0000-0000-000000000000",', ' "pool": "",', ' "status": "OK",', ' "truesize": "134217728",', ' "type": "PREALLOCATED",', ' "uuid": "033d64fd-6f93-42be-84bc-082b03095ef3",', ' "voltype": "LEAF"', '}'], 'stderr_lines': [], '_ansible_no_log': False, 'failed': True, 'attempts': 12, 'item': {'name': 'OVF_STORE', 'image_id': '033d64fd-6f93-42be-84bc-082b03095ef3', 'id': '044e384a-dedf-4589-8dfb-beca170138ee'}, 'ansible_loop_var': 'item', '_ansible_item_label': {'name': 'OVF_STORE', 'image_id': '033d64fd-6f93-42be-84bc-082b03095ef3', 'id': '044e384a-dedf-4589-8dfb-beca170138ee'}} 2020-07-20 06:40:23,128-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:105 PLAY RECAP [localhost] : ok: 69 changed: 19 unreachable: 0 skipped: 10 failed: 1

3 2

ovirt-engine has been tagged (ovirt-engine-4.4.1.10)
by Tal Nisan 21 Jul '20

21 Jul '20

1 0

el8 CI breaking on iprocess mirrors
by Amit Bawer 20 Jul '20

20 Jul '20

Hi This can be seen on [1] 13:47:57 Error: Error downloading packages: 13:47:57 Cannot download x86_64/ioprocess-1.4.1-1.202007151811.gitc41863d.el8.x86_64.rpm: All mirrors were tried [1] https://jenkins.ovirt.org/job/vdsm_standard-check-patch/22702/consoleFull Thanks

2 2