July 2020 - Devel - oVirt List Archives

[VDSM] Started getting virNetTLSContextCheckCertTimes:154 : The server certificate /etc/pki/vdsm/certs/vdsmcert.pem has expired
by Edward Haas 11 Jan '24

11 Jan '24

Hi, I have a VM which acts as an oVirt host with VDSM installed. I use it for testing without connecting it to Engine. Recently VDSM fails to come up and I see this error: Sep 26 17:58:18 localhost libvirtd: 2018-09-26 14:58:18.978+0000: 12176: error : virNetTLSContextCheckCertTimes:154 : The server certificate /etc/pki/vdsm/certs/vdsmcert.pem has expired Any hint on how to generate or fetch a new certificate without the help of Engine? Thanks, Edy.

3 5

Error: Adding new Host to ovirt-engine
by Ahmad Khiet 14 Aug '23

14 Aug '23

Hi, Can't add new host to ovirt engine, because the following error: 2019-06-12 12:23:09,664 p=4134 u=engine | TASK [ovirt-host-deploy-facts : Set facts] ************************************* 2019-06-12 12:23:09,684 p=4134 u=engine | ok: [10.35.1.17] => { "ansible_facts": { "ansible_python_interpreter": "/usr/bin/python2", "host_deploy_vdsm_version": "4.40.0" }, "changed": false } 2019-06-12 12:23:09,697 p=4134 u=engine | TASK [ovirt-provider-ovn-driver : Install ovs] ********************************* 2019-06-12 12:23:09,726 p=4134 u=engine | fatal: [10.35.1.17]: FAILED! => {} MSG: The conditional check 'cluster_switch == "ovs" or (ovn_central is defined and ovn_central | ipaddr and ovn_engine_cluster_version is version_compare('4.2', '>='))' failed. The error was: The ipaddr filter requires python's netaddr be installed on the ansible controller The error appears to be in '/home/engine/apps/engine/share/ovirt-engine/playbooks/roles/ovirt-provider-ovn-driver/tasks/configure.yml': line 3, column 5, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - block: - name: Install ovs ^ here 2019-06-12 12:23:09,728 p=4134 u=engine | PLAY RECAP ********************************************************************* 2019-06-12 12:23:09,728 p=4134 u=engine | 10.35.1.17 : ok=3 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 whats missing!? Thanks -- Ahmad Khiet Red Hat <https://www.redhat.com/> akhiet(a)redhat.com M: +972-54-6225629 <https://red.ht/sig>

7 10

Purging inactive maintainers from vdsm-master-maintainers
by Nir Soffer 21 Jun '21

21 Jun '21

I want to remove inactive contributors from vdsm-master-maintainers. I suggest the simple rule of 2 years of inactivity for removing from this group, based on git log. See the list below for current status: https://gerrit.ovirt.org/#/admin/groups/106,members Thoughts? Nir

6 11

Error Java SDK Issue??
by Geschwentner, Patrick 02 Apr '21

02 Apr '21

Dear Ladies and Gentlemen! I am currently working with the java-sdk and I encountered a problem. If I would like to retrieve the disk details, I get the following error: Disk currDisk = ovirtConnection.followLink(diskAttachment.disk()); The Error is occurring in this line: [cid:image001.png@01D44537.AF127FD0] The getResponst looks quiet ok. (I inspected: [cid:image002.png@01D44537.AF127FD0] and it looks ok). Error: wrong number of arguments The code is quiet similar to what you published on github (https://github.com/oVirt/ovirt-engine-sdk-java/blob/master/sdk/src/test/jav… ). Can you confirm the defect? Best regards Patrick

6 11

CentOS Stream support
by Michal Skrivanek 01 Feb '21

01 Feb '21

Hi all, we would like to ask about interest in community about oVirt moving to CentOS Stream. There were some requests before but it’s hard to see how many people would really like to see that. With CentOS releases lagging behind RHEL for months it’s interesting to consider moving to CentOS Stream as it is much more up to date and allows us to fix bugs faster, with less workarounds and overhead for maintaining old code. E.g. our current integration tests do not really pass on CentOS 8.1 and we can’t really do much about that other than wait for more up to date packages. It would also bring us closer to make oVirt run smoothly on RHEL as that is also much closer to Stream than it is to outdated CentOS. So..would you like us to support CentOS Stream? We don’t really have capacity to run 3 different platforms, would you still want oVirt to support CentOS Stream if it means “less support” for regular CentOS? There are some concerns about Stream being a bit less stable, do you share those concerns? Thank you for your comments, michal

6 12

device compatibility interface for live migration with assigned devices
by Yan Zhao 15 Sep '20

15 Sep '20

hi folks, we are defining a device migration compatibility interface that helps upper layer stack like openstack/ovirt/libvirt to check if two devices are live migration compatible. The "devices" here could be MDEVs, physical devices, or hybrid of the two. e.g. we could use it to check whether - a src MDEV can migrate to a target MDEV, - a src VF in SRIOV can migrate to a target VF in SRIOV, - a src MDEV can migration to a target VF in SRIOV. (e.g. SIOV/SRIOV backward compatibility case) The upper layer stack could use this interface as the last step to check if one device is able to migrate to another device before triggering a real live migration procedure. we are not sure if this interface is of value or help to you. please don't hesitate to drop your valuable comments. (1) interface definition The interface is defined in below way: __ userspace /\ \ / \write / read \ ________/__________ ___\|/_____________ | migration_version | | migration_version |-->check migration --------------------- --------------------- compatibility device A device B a device attribute named migration_version is defined under each device's sysfs node. e.g. (/sys/bus/pci/devices/0000\:00\:02.0/$mdev_UUID/migration_version). userspace tools read the migration_version as a string from the source device, and write it to the migration_version sysfs attribute in the target device. The userspace should treat ANY of below conditions as two devices not compatible: - any one of the two devices does not have a migration_version attribute - error when reading from migration_version attribute of one device - error when writing migration_version string of one device to migration_version attribute of the other device The string read from migration_version attribute is defined by device vendor driver and is completely opaque to the userspace. for a Intel vGPU, string format can be defined like "parent device PCI ID" + "version of gvt driver" + "mdev type" + "aggregator count". for an NVMe VF connecting to a remote storage. it could be "PCI ID" + "driver version" + "configured remote storage URL" for a QAT VF, it may be "PCI ID" + "driver version" + "supported encryption set". (to avoid namespace confliction from each vendor, we may prefix a driver name to each migration_version string. e.g. i915-v1-8086-591d-i915-GVTg_V5_8-1) (2) backgrounds The reason we hope the migration_version string is opaque to the userspace is that it is hard to generalize standard comparing fields and comparing methods for different devices from different vendors. Though userspace now could still do a simple string compare to check if two devices are compatible, and result should also be right, it's still too limited as it excludes the possible candidate whose migration_version string fails to be equal. e.g. an MDEV with mdev_type_1, aggregator count 3 is probably compatible with another MDEV with mdev_type_3, aggregator count 1, even their migration_version strings are not equal. (assumed mdev_type_3 is of 3 times equal resources of mdev_type_1). besides that, driver version + configured resources are all elements demanding to take into account. So, we hope leaving the freedom to vendor driver and let it make the final decision in a simple reading from source side and writing for test in the target side way. we then think the device compatibility issues for live migration with assigned devices can be divided into two steps: a. management tools filter out possible migration target devices. Tags could be created according to info from product specification. we think openstack/ovirt may have vendor proprietary components to create those customized tags for each product from each vendor. e.g. for Intel vGPU, with a vGPU(a MDEV device) in source side, the tags to search target vGPU are like: a tag for compatible parent PCI IDs, a tag for a range of gvt driver versions, a tag for a range of mdev type + aggregator count for NVMe VF, the tags to search target VF may be like: a tag for compatible PCI IDs, a tag for a range of driver versions, a tag for URL of configured remote storage. b. with the output from step a, openstack/ovirt/libvirt could use our proposed device migration compatibility interface to make sure the two devices are indeed live migration compatible before launching the real live migration process to start stream copying, src device stopping and target device resuming. It is supposed that this step would not bring any performance penalty as -in kernel it's just a simple string decoding and comparing -in openstack/ovirt, it could be done by extending current function check_can_live_migrate_destination, along side claiming target resources.[1] [1] https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/libvi… Thanks Yan

14 117

[ARM64] Possiblity to support oVirt on ARM64
by Zhenyu Zheng 11 Sep '20

11 Sep '20

Hi oVirt, We are currently trying to make oVirt work on ARM64 platform, since I'm quite new to oVirt community, I'm wondering what is the current status about ARM64 support in the oVirt upstream, as I saw the oVirt Wikipedia page mentioned there is an ongoing efforts to support ARM platform. We have a small team here and we are willing to also help to make this work. Thanks alot, Zhenyu Zheng

6 8

OST for 4.3 is broken on build stage
by Amit Bawer 26 Aug '20

26 Aug '20

Hi Lookup for el7 repo fails: [2020-06-22T07:46:48.800Z] ERROR: Command failed: [2020-06-22T07:46:48.800Z] # /usr/bin/yum --installroot /var/lib/mock/epel-7-ppc64le-60da95f193baa5a745076a0c6b674c73-bootstrap-6968/root/ --releasever 7 install yum yum-utils --setopt=tsflags=nocontexts [2020-06-22T07:46:48.800Z] Failed to set locale, defaulting to C [2020-06-22T07:46:48.800Z] http://cbs.centos.org/repos/virt7-ovirt-common-candidate/ppc64le/os/repodat…: [Errno 14] HTTPS Error 404 - Not Found [2020-06-22T07:46:48.800Z] Trying other mirror. [2020-06-22T07:46:48.800Z] To address this issue please refer to the below wiki article [2020-06-22T07:46:48.800Z] Taken from https://jenkins.ovirt.org/job/vdsm_standard-check-patch/21966/consoleText Thanks

2 1

implementing hotplugCd/hotunplugCd in vdsm
by Fedor Gavrilov 25 Aug '20

25 Aug '20

Hey, So in an attempt to fix change CD functionality we discovered a few other potential issues and what Nir suggested was to implement two [somewhat] new functions in VDSM: hotplug and hotunplug for CDs similar to how it works for normal disks now. Existing changeCD function will be left as is for backwards compatibility. As I found out, engine already calculates iface and index before invoking VDSM functions, so we will just pass these along with PDIV to the VDSM. Suggested flow is, let me quote: >So the complete change CD flow should be: > >1. get the previous drivespec from vm metadata >2. prepare new drivespec >3. add new drivespec to vm metadata >4. attach a new device to vm >5. teardown the previous drivespec >6. remove previous drivespec from vm metadata > >When the vm is stopped, it must do: > >1. get drive spec from vm metadata >2. teardown drivespec > >During attach, there are interesting races: >- what happens if vdsm crashes after step 2? who will teardown the volume? > maybe you need to add the new drivespec to the metadata first, >before preparing it. >- what happens if attach failed? who will remove the new drive from >the metadata? Now, what makes hotplugDisk/hotunplugDisk different? From what I understand, the flow is same there, so what difference is there as far as VDSM is concerned? If none, this means if I more or less copy that code, changing minor details and data accordingly for CDs, this should work, shouldn't it? Thanks! Fedor

4 11

Improving VM behavior in case of IO errors
by Shubha Kulkarni 16 Aug '20

16 Aug '20

Hello, In OVirt, we have a property propagate_error at the disk level that decides in case of an error, how this error be propagated to the VM. This value is maintained in the database table with the default value set as Off. The default setting(Off) results in a policy that ends up pausing the VM rather than propagating the errors to VM. There is no provision in the UI currently to configure this property for disk (images or luns). So there is no easy way to set this value. Further, even if the value is manually set to "On" in db, it gets overwriiten by UI everytime some other property is updated as described here - https://bugzilla.redhat.com/show_bug.cgi?id=1669367 Setting the value to "Off" is not ideal for multipath devices where a single path failure causes vm to pause. It puts serious restrictions for the DR situation and unlike VMWare * Hyper-V, oVirt is not able to support the DR functionality - https://bugzilla.redhat.com/show_bug.cgi?id=1314160 While we wait for RFE, the proposal here is to revise the out of the box behavior for LUNs. For LUNs, we should propagate the errors to VM rather than directly stopping those. This will allow us to handle short-term multipath outages and improve availability. This is a simple change in behavior but will have good positive impact. I would like to seek feedback about this to make sure that everyone is ok with the proposal. Thanks, Shubha

3 11

[VDSM] Trouble merging patch, related to jenkins whitelist or new RHEL CI?
by Nir Soffer 03 Aug '20

03 Aug '20

I could not merge: https://gerrit.ovirt.org/c/109402/ Although it was approved and verified, and got +1 from Continuous Integration. There is a new "RHEL-Continuous-Integration", which does not run with this change, even when I trigger the tests manually with "ci test". I tried to add +1 for "RHEL-Continuous-Integration" but this is not possible, I see: RHEL-Continuous-Integration You don't have permission to edit this label. So finally I downloaded the patch and pushed it manually. I think this patch will fix the problem: https://gerrit.ovirt.org/#/c/110576/ But I need CI experts to review this. Since we have 50 projects that need this, this probably should be fixed elsewhere, and inherited by all projects. I know that Germano was not able to trigger tests because he was missing in the jenkins whitelist, but this was fixed last week. Germano, maybe just to check that everything works for you, you can post some trivial patch? Nir

4 9

please i'm facing a with making el8
by Amit Chaudhary 30 Jul '20

30 Jul '20

ovirt.zip <https://drive.google.com/file/d/1I1Kcbnmi3IIE1taUsWEe2uQaWFWvERyE/view?usp=…> i have attached my ovirt zip. the issue im facing is that the kickstart is not generated for el8 and im not able to figure out how to use .j2 file that is being generated. i've manage to frankenstein a kickstart file that is in the zip with all the settings but its also giving error while compressing (dracut module 'livenet' cannot be found or installed) while if i search installed modules of dracut i can see livenet is installed. see see my zip it have logs of last compilation also. sir/mam it would be a great help for my college final year project project on hci and my research paper. please sir/mam please help me Amit Chaudhary +919650596355 -- *CONFIDENTIALITY NOTICE* This communication (including any attachment/s) is intended only for the use of the addressee(s) and contains information that is PRIVILEGED AND CONFIDENTIAL. Unauthorized reading, dissemination, distribution, or copying of this communication is prohibited. * DISCLAIMER* Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the University. WARNING:- This communication, including any attachments, may not be free of viruses, interceptions or interference, and may not be compatible with your systems. You should carry out your own virus checks before opening any attachment to this e-mail. The sender of this e-mail and Galgotias University shall not be liable for any damage that you may sustain as a result of viruses, incompleteness of this message, a delay in receipt of this message or computer problems experienced.

2 1

OST switching back to CirrOS
by Michal Skrivanek 30 Jul '20

30 Jul '20

Hi, FYI we’ve just switched back from CentOS to CirrOS VMs in OST basic suite, saving 900MB of data transferred all the time (and copied back and forth during test) as well as lower memory requirements (down to 128MB from 384MB, times 3 VMs). For a good measure a memory hot unplug test was added too. Let me know if you see anything unusual, other than faster run times. Thanks, michal

2 2

Error while executing action New NFS Storage Domain: Unexpected exception
by Dana Elfassy 30 Jul '20

30 Jul '20

Hi, When I'm trying to add a storage domain via the webadmin I'm getting the error message that's on the subject. in the vdsm.log I see this error: 2020-07-29 04:53:48,045-0400 ERROR (check/loop) [storage.Monitor] Error checking path /rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata (monitor:525) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 523, in _pathChecked delay = result.delay() File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line 391, in delay raise exception.MiscFileReadException(self.path, self.rc, self.err) vdsm.storage.exception.MiscFileReadException: Internal file read failure: ('/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata', 1, bytearray(b"/usr/bin/dd: failed to open \'/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata\': No such file or directory\n")) when looking at the directory I do see that metadata was created under this directory, with the following content: [delfassy@delfassy mnt]$ cd new_nfs/cddff7bb-bb0a-4fc8-8ba4-023be91d822d/dom_md/ [delfassy@delfassy dom_md]$ less metadata CLASS=Data DESCRIPTION=spider IOOPTIMEOUTSEC=10 LEASERETRIES=3 LEASETIMESEC=60 LOCKPOLICY= LOCKRENEWALINTERVALSEC=5 POOL_UUID= REMOTE_PATH=vserver-spider.eng.lab.tlv.redhat.com:/pub/delfassy/new_nfs ROLE=Regular SDUUID=cddff7bb-bb0a-4fc8-8ba4-023be91d822d TYPE=NFS VERSION=4 _SHA_CKSUM=f4953016c94f205127b00a5b10f0e7d5d9cf24f7 The permissions on the directory are: drwxrwxrwx. 3 vdsm kvm 4096 Jul 29 11:49 new_mnt Are there any other configurations I'm missing? Any idea how this issue can be resolved? Thanks, Dana

2 3

Re: Issue with ovirt 4.4 after doing some incremental backups.
by Eyal Shenitzky 29 Jul '20

29 Jul '20

According to the vdsm.log it seems that the host reports checkpoint 'ef0dfe55-c08c-4d9e-ad32-d6b6d5cbdac6' as defined checkpoint for the VM - 7-28 16:24:06,488+0200 INFO (jsonrpc/3) [api.virt] FINISH list_checkpoints return={'result': ['cd078706-84c0-4370-a6ec-654ccd6a21aa', '906fe1fd-4e81-473c-9582-de693985f462', 'd20de661-fcc1-43ff-86a3-6294a0a94e1b', '805b447e-d033-4e b1-905a-d287ca50c85a', '9ae68264-a0c9-4f75-9ffa-20c40323abe9', '8664da63-6d92-44b8-a690-04e17afb483a', 'fb6fa799-3b6f-44fa-a60e-c66ce8c76e35', 'f9f53ee7-2a8c-4dfc-a669-2dd788361eae', '7a0f37db-02bc-49c4-bfea-a03d3474f1f8', '95d8d5c7-55fc- 4c39-8a72-fc3aaec72c39', 'aa71a6e9-174b-4d8d-a2da-1a12d5e2ab2e', 'ac833f22-c8fa-4925-becc-dcf4dac81d3f', '7fc9afd1-076f-4985-b412-4ea029fd2af7', 'ae299a18-c4b0-46c4-81e1-f9583e6954f6', 'cee97ef9-6e7c-4518-91b1-dfa3b467f5c4', '51b3bf23-3cc 3-4a16-b6d9-942c6e4f7573', '0f376950-a25e-477a-93b8-d1d0ffbff716', 'ef0dfe55-c08c-4d9e-ad32-d6b6d5cbdac6'], 'status': {'code': 0, 'message': 'Done'}} from=::ffff:10.40.1.66,56474, flow_id=32ba9d9d-32c2-4652-ac01-895e92123323, vmId=116aa6e b-31a1-43db-9b1e-ad6e32fb9260 (api:54) It means as Nir said that the host assumes the corresponding bitmap is also exists in QEMU. The logs only contain a short period of time so I cannot tell if some bitmap destructive operation occurs on the VM (Live merge, cold snapshot creation etc..) Peter, is there any other log that may assist? On Wed, 29 Jul 2020 at 11:28, Łukasz Kołaciński <l.kolacinski(a)storware.eu> wrote: > Hello, > I am sending logs from vdsm, maybe they will help. I cannot find useful > libvirt logs and file libvirt.log doesn't exists. > > Best Regards, > > Łukasz Kołaciński > > Junior Java Developer > > e-mail: l.kolacinski(a)storware.eu > <m.helbert(a)storware.eu> > > > > > *[image: STORWARE]* <http://www.storware.eu/> > > > > *ul. Leszno 8/44 01-192 Warszawa www.storware.eu > <https://www.storware.eu/>* > > *[image: facebook]* <https://www.facebook.com/storware> > > *[image: twitter]* <https://twitter.com/storware> > > *[image: linkedin]* <https://www.linkedin.com/company/storware> > > *[image: Storware_Stopka_09]* > <https://www.youtube.com/channel/UCKvLitYPyAplBctXibFWrkw> > > > > *Storware Spółka z o.o. nr wpisu do ewidencji KRS dla M.St. Warszawa > 000510131* *, NIP 5213672602.** Wiadomość ta jest przeznaczona jedynie > dla osoby lub podmiotu, który jest jej adresatem i może zawierać poufne > i/lub uprzywilejowane informacje. Zakazane jest jakiekolwiek przeglądanie, > przesyłanie, rozpowszechnianie lub inne wykorzystanie tych informacji lub > podjęcie jakichkolwiek działań odnośnie tych informacji przez osoby lub > podmioty inne niż zamierzony adresat. Jeżeli Państwo otrzymali przez > pomyłkę tę informację prosimy o poinformowanie o tym nadawcy i usunięcie > tej wiadomości z wszelkich komputerów. **This message is intended only > for the person or entity to which it is addressed and may contain > confidential and/or privileged material. Any review, retransmission, > dissemination or other use of, or taking of any action in reliance upon, > this information by persons or entities other than the intended recipient > is prohibited. If you have received this message in error, please contact > the sender and remove the material from all of your computer systems.* > > ------------------------------ > *Od:* Nir Soffer <nsoffer(a)redhat.com> > *Wysłane:* środa, 29 lipca 2020 00:31 > *Do:* Łukasz Kołaciński <l.kolacinski(a)storware.eu> > *DW:* Eyal Shenitzky <eshenitz(a)redhat.com>; Mateusz Maziarz < > m.maziarz(a)storware.eu>; Marcin Kubacki <m.kubacki(a)storware.eu>; Peter > Krempa <pkrempa(a)redhat.com>; devel <devel(a)ovirt.org> > *Temat:* Re: Issue with ovirt 4.4 after doing some incremental backups. > > On Tue, Jul 28, 2020 at 4:05 PM Łukasz Kołaciński < > l.kolacinski(a)storware.eu> wrote: > > Hello, > > > Hi Łukasz > > I moved the discussion to devel(a)ovirt.org since it is more appropriate > for this issue. > > After doing a few vm backups, something breaks and I am unable to perform > any operations. I cannot do incremental backups and even full backups > doesn't work. I have this issue third time. I don't know how to fix this so > I am currently making new vms for testing purposes > > VDSM ovirt44-h2.storware.local command StartVmBackupVDS failed: Backup > Error: {'vm_id': '116aa6eb-31a1-43db-9b1e-ad6e32fb9260', 'backup': > <vdsm.virt.backup.BackupConfig object at 0x7f42602bba20>, 'reason': "Error > starting backup: internal error: unable to execute QEMU command > 'transaction': Dirty bitmap 'ef0dfe55-c08c-4d9e-ad32-d6b6d5cbdac6' not > found"} > > This means that libvirt cannot find the dirty bitmap when starting the > backup. > > When we start a backup, we get the list of checkpoints from libvirt and we > redefine all checkpoints. We assume that all redefined checkpoints have > a bitmap in qemu at the time of the redefine. > > Peter, is this assumption correct? > > If libvirt and engine agree on the existing checkpoints, we start the > backup. > In this case one of the bitmaps was missing, so the backup failed. > > We know about some flows that may cause loss of the bitmaps: > - copying disks (bitmaps are not copied yet) > - live storage migration (it copy the disks) > - deleting snapshots > - live migration may cause this, not tested yet > - unclean shutdown of the vm > - storage is not accessible when vm is terminated > > Did you do any of these operations on the tested vm? > > Some of the issues can never be fixed, like unclean shutdown or storage > issue when qemu try to write the bitmaps to disk. So your backup > application > must be able to recover from this error. > > However oVirt does not provide a useful error that enables recovery. We > have > a special error when full backup is required, but it seem that this error > is > not returned in this case, and instead we return internal error. > > Also since you cannot do a full backup after this error, I guess that > engine > did not delete the checkpoint with the missing bitmap. This is not > surprising > since the error returned from vdsm is a generic error (BackupError), so > engine > cannot tell what is the reason for the failure. > > Did you check the backup events? What was the backup completion event? > > See this example how to get backup events: > > https://github.com/oVirt/ovirt-engine-sdk/blob/4a143351fcd3cdb0df8c50888931… > > https://github.com/oVirt/ovirt-engine-sdk/blob/4a143351fcd3cdb0df8c50888931… > > Eyal, looking at the API docs: > http://ovirt.github.io/ovirt-engine-api-model/master/#types/event > > Event code is an integer. This is not usable for detecting errors since > the value > is not part of the API. We need an enum like: > > http://ovirt.github.io/ovirt-engine-api-model/4.4/#types/image_transfer_pha… > > Peter, do we have a specific error code in libvirt about missing bitmap? > we need > this to pass useful error to engine, and engine needs this error to pass > useful > error to the user. > > The error seen here is generated by: > > flags = libvirt.VIR_DOMAIN_BACKUP_BEGIN_REUSE_EXTERNAL > try: > dom.backupBegin(backup_xml, checkpoint_xml, flags=flags) > except libvirt.libvirtError as e: > raise exception.BackupError( > reason="Error starting backup: {}".format(e), > vm_id=vm.id, > backup=backup_cfg) > > Looking in the documentation: > https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBackupBegin > > There are no specified errors so we cannot detect the reason for the > failure > and return meaningful error to our caller. > > Peter, how do you suggest to recover from internal errors? how can we > tell if this temporary error that can succeed in the next attempt, or an > error > that requires starting from full backup? > > Nir > -- Regards, Eyal Shenitzky

2 1

Re: Issue with ovirt 4.4 after doing some incremental backups.
by Eyal Shenitzky 29 Jul '20

29 Jul '20

Libvirt logs need to be activated on the host in order to collect them, please try: - systemctl stop vdsmd - systemctl stop libvirtd - Under /etc/libvirt/libvirtd.conf add: - log_filters="1:qemu 1:libvirt 4:object 4:json 4:event 1:util" - log_outputs="1:file:/var/log/libvirt/libvirtd.log" - systemctl start libvirtd - systemctl start vdsmd On Wed, 29 Jul 2020 at 11:28, Łukasz Kołaciński <l.kolacinski(a)storware.eu> wrote: > Hello, > I am sending logs from vdsm, maybe they will help. I cannot find useful > libvirt logs and file libvirt.log doesn't exists. > > Best Regards, > > Łukasz Kołaciński > > Junior Java Developer > > e-mail: l.kolacinski(a)storware.eu > <m.helbert(a)storware.eu> > > > > > *[image: STORWARE]* <http://www.storware.eu/> > > > > *ul. Leszno 8/44 01-192 Warszawa www.storware.eu > <https://www.storware.eu/>* > > *[image: facebook]* <https://www.facebook.com/storware> > > *[image: twitter]* <https://twitter.com/storware> > > *[image: linkedin]* <https://www.linkedin.com/company/storware> > > *[image: Storware_Stopka_09]* > <https://www.youtube.com/channel/UCKvLitYPyAplBctXibFWrkw> > > > > *Storware Spółka z o.o. nr wpisu do ewidencji KRS dla M.St. Warszawa > 000510131* *, NIP 5213672602.** Wiadomość ta jest przeznaczona jedynie > dla osoby lub podmiotu, który jest jej adresatem i może zawierać poufne > i/lub uprzywilejowane informacje. Zakazane jest jakiekolwiek przeglądanie, > przesyłanie, rozpowszechnianie lub inne wykorzystanie tych informacji lub > podjęcie jakichkolwiek działań odnośnie tych informacji przez osoby lub > podmioty inne niż zamierzony adresat. Jeżeli Państwo otrzymali przez > pomyłkę tę informację prosimy o poinformowanie o tym nadawcy i usunięcie > tej wiadomości z wszelkich komputerów. **This message is intended only > for the person or entity to which it is addressed and may contain > confidential and/or privileged material. Any review, retransmission, > dissemination or other use of, or taking of any action in reliance upon, > this information by persons or entities other than the intended recipient > is prohibited. If you have received this message in error, please contact > the sender and remove the material from all of your computer systems.* > > ------------------------------ > *Od:* Nir Soffer <nsoffer(a)redhat.com> > *Wysłane:* środa, 29 lipca 2020 00:31 > *Do:* Łukasz Kołaciński <l.kolacinski(a)storware.eu> > *DW:* Eyal Shenitzky <eshenitz(a)redhat.com>; Mateusz Maziarz < > m.maziarz(a)storware.eu>; Marcin Kubacki <m.kubacki(a)storware.eu>; Peter > Krempa <pkrempa(a)redhat.com>; devel <devel(a)ovirt.org> > *Temat:* Re: Issue with ovirt 4.4 after doing some incremental backups. > > On Tue, Jul 28, 2020 at 4:05 PM Łukasz Kołaciński < > l.kolacinski(a)storware.eu> wrote: > > Hello, > > > Hi Łukasz > > I moved the discussion to devel(a)ovirt.org since it is more appropriate > for this issue. > > After doing a few vm backups, something breaks and I am unable to perform > any operations. I cannot do incremental backups and even full backups > doesn't work. I have this issue third time. I don't know how to fix this so > I am currently making new vms for testing purposes > > VDSM ovirt44-h2.storware.local command StartVmBackupVDS failed: Backup > Error: {'vm_id': '116aa6eb-31a1-43db-9b1e-ad6e32fb9260', 'backup': > <vdsm.virt.backup.BackupConfig object at 0x7f42602bba20>, 'reason': "Error > starting backup: internal error: unable to execute QEMU command > 'transaction': Dirty bitmap 'ef0dfe55-c08c-4d9e-ad32-d6b6d5cbdac6' not > found"} > > This means that libvirt cannot find the dirty bitmap when starting the > backup. > > When we start a backup, we get the list of checkpoints from libvirt and we > redefine all checkpoints. We assume that all redefined checkpoints have > a bitmap in qemu at the time of the redefine. > > Peter, is this assumption correct? > > If libvirt and engine agree on the existing checkpoints, we start the > backup. > In this case one of the bitmaps was missing, so the backup failed. > > We know about some flows that may cause loss of the bitmaps: > - copying disks (bitmaps are not copied yet) > - live storage migration (it copy the disks) > - deleting snapshots > - live migration may cause this, not tested yet > - unclean shutdown of the vm > - storage is not accessible when vm is terminated > > Did you do any of these operations on the tested vm? > > Some of the issues can never be fixed, like unclean shutdown or storage > issue when qemu try to write the bitmaps to disk. So your backup > application > must be able to recover from this error. > > However oVirt does not provide a useful error that enables recovery. We > have > a special error when full backup is required, but it seem that this error > is > not returned in this case, and instead we return internal error. > > Also since you cannot do a full backup after this error, I guess that > engine > did not delete the checkpoint with the missing bitmap. This is not > surprising > since the error returned from vdsm is a generic error (BackupError), so > engine > cannot tell what is the reason for the failure. > > Did you check the backup events? What was the backup completion event? > > See this example how to get backup events: > > https://github.com/oVirt/ovirt-engine-sdk/blob/4a143351fcd3cdb0df8c50888931… > > https://github.com/oVirt/ovirt-engine-sdk/blob/4a143351fcd3cdb0df8c50888931… > > Eyal, looking at the API docs: > http://ovirt.github.io/ovirt-engine-api-model/master/#types/event > > Event code is an integer. This is not usable for detecting errors since > the value > is not part of the API. We need an enum like: > > http://ovirt.github.io/ovirt-engine-api-model/4.4/#types/image_transfer_pha… > > Peter, do we have a specific error code in libvirt about missing bitmap? > we need > this to pass useful error to engine, and engine needs this error to pass > useful > error to the user. > > The error seen here is generated by: > > flags = libvirt.VIR_DOMAIN_BACKUP_BEGIN_REUSE_EXTERNAL > try: > dom.backupBegin(backup_xml, checkpoint_xml, flags=flags) > except libvirt.libvirtError as e: > raise exception.BackupError( > reason="Error starting backup: {}".format(e), > vm_id=vm.id, > backup=backup_cfg) > > Looking in the documentation: > https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBackupBegin > > There are no specified errors so we cannot detect the reason for the > failure > and return meaningful error to our caller. > > Peter, how do you suggest to recover from internal errors? how can we > tell if this temporary error that can succeed in the next attempt, or an > error > that requires starting from full backup? > > Nir > -- Regards, Eyal Shenitzky

2 1

Re: Issue with ovirt 4.4 after doing some incremental backups.
by Nir Soffer 29 Jul '20

29 Jul '20

On Tue, Jul 28, 2020 at 4:05 PM Łukasz Kołaciński <l.kolacinski(a)storware.eu> wrote: > Hello, > Hi Łukasz I moved the discussion to devel(a)ovirt.org since it is more appropriate for this issue. After doing a few vm backups, something breaks and I am unable to perform > any operations. I cannot do incremental backups and even full backups > doesn't work. I have this issue third time. I don't know how to fix this so > I am currently making new vms for testing purposes > > VDSM ovirt44-h2.storware.local command StartVmBackupVDS failed: Backup > Error: {'vm_id': '116aa6eb-31a1-43db-9b1e-ad6e32fb9260', 'backup': > <vdsm.virt.backup.BackupConfig object at 0x7f42602bba20>, 'reason': "Error > starting backup: internal error: unable to execute QEMU command > 'transaction': Dirty bitmap 'ef0dfe55-c08c-4d9e-ad32-d6b6d5cbdac6' not > found"} > > This means that libvirt cannot find the dirty bitmap when starting the backup. When we start a backup, we get the list of checkpoints from libvirt and we redefine all checkpoints. We assume that all redefined checkpoints have a bitmap in qemu at the time of the redefine. Peter, is this assumption correct? If libvirt and engine agree on the existing checkpoints, we start the backup. In this case one of the bitmaps was missing, so the backup failed. We know about some flows that may cause loss of the bitmaps: - copying disks (bitmaps are not copied yet) - live storage migration (it copy the disks) - deleting snapshots - live migration may cause this, not tested yet - unclean shutdown of the vm - storage is not accessible when vm is terminated Did you do any of these operations on the tested vm? Some of the issues can never be fixed, like unclean shutdown or storage issue when qemu try to write the bitmaps to disk. So your backup application must be able to recover from this error. However oVirt does not provide a useful error that enables recovery. We have a special error when full backup is required, but it seem that this error is not returned in this case, and instead we return internal error. Also since you cannot do a full backup after this error, I guess that engine did not delete the checkpoint with the missing bitmap. This is not surprising since the error returned from vdsm is a generic error (BackupError), so engine cannot tell what is the reason for the failure. Did you check the backup events? What was the backup completion event? See this example how to get backup events: https://github.com/oVirt/ovirt-engine-sdk/blob/4a143351fcd3cdb0df8c50888931… https://github.com/oVirt/ovirt-engine-sdk/blob/4a143351fcd3cdb0df8c50888931… Eyal, looking at the API docs: http://ovirt.github.io/ovirt-engine-api-model/master/#types/event Event code is an integer. This is not usable for detecting errors since the value is not part of the API. We need an enum like: http://ovirt.github.io/ovirt-engine-api-model/4.4/#types/image_transfer_pha… Peter, do we have a specific error code in libvirt about missing bitmap? we need this to pass useful error to engine, and engine needs this error to pass useful error to the user. The error seen here is generated by: flags = libvirt.VIR_DOMAIN_BACKUP_BEGIN_REUSE_EXTERNAL try: dom.backupBegin(backup_xml, checkpoint_xml, flags=flags) except libvirt.libvirtError as e: raise exception.BackupError( reason="Error starting backup: {}".format(e), vm_id=vm.id, backup=backup_cfg) Looking in the documentation: https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBackupBegin There are no specified errors so we cannot detect the reason for the failure and return meaningful error to our caller. Peter, how do you suggest to recover from internal errors? how can we tell if this temporary error that can succeed in the next attempt, or an error that requires starting from full backup? Nir

3 2

ovirt-engine has been tagged (ovirt-engine-4.4.2)
by Tal Nisan 29 Jul '20

29 Jul '20

3 2

ovirt-engine has been tagged (ovirt-engine-4.3.11.3)
by Tal Nisan 28 Jul '20

28 Jul '20

1 0

Engine on Fedora 32 (help needed)
by Pavel Bar 27 Jul '20

27 Jul '20

Hi, I upgraded the OS on my laptop from Fedora 30 to Fedora 32 and now have issues with running an engine - some libraries are missing. I know that the new Fedora releases are not supported. I couldn't downgrade back to Fedora 30 and I don't want to totally reinstall the OS (either to an old Fedora 30 or to a new CentOS 8.2). Issues that I encountered: 1) Some repositories do not exist for Fedora 32. Copr repo for ovirt-katello owned by sbonazzo 383 B/s | 341 B 00:00 Errors during downloading metadata for repository '*sbonazzo-ovirt-katello* ': - Status code: 404 for https://copr-be.cloud.fedoraproject.org/results/sbonazzo/ovirt-katello/fedo… (IP: 34.227.76.72) Error: Failed to download metadata for repo 'sbonazzo-ovirt-katello': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried Copr repo for ovirt-cockpit-sso owned by mlibra 380 B/s | 341 B 00:00 Errors during downloading metadata for repository ' *mlibra-ovirt-cockpit-sso*': - Status code: 404 for https://copr-be.cloud.fedoraproject.org/results/mlibra/ovirt-cockpit-sso/fe… (IP: 34.227.76.72) Error: Failed to download metadata for repo 'mlibra-ovirt-cockpit-sso': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried Copr repo for ovsdbapp owned by mdbarroso 339 B/s | 341 B 00:01 Errors during downloading metadata for repository '*mdbarroso-ovsdbapp*': - Status code: 404 for https://copr-be.cloud.fedoraproject.org/results/mdbarroso/ovsdbapp/fedora-3… (IP: 34.227.76.72) Error: Failed to download metadata for repo 'mdbarroso-ovsdbapp': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried Ignoring repositories: *sbonazzo-ovirt-katello*, *mlibra-ovirt-cockpit-sso*, *mdbarroso-ovsdbapp* Workaround (at least stopped the errors): I updated the " /etc/yum.repos.d/ovirt-master-dependencies.repo" and changed the " $releasever" into the hardcoded "30" for the 3 repositories above. 2) Unable to install "python3-otopi" and probably some other libraries. Workaround that didn't fully work: I updated " /etc/yum.repos.d/ovirt-master-snapshot.repo" and changed the "$releasever" into the hardcoded "30" for both "ovirt-master-snapshot" & " ovirt-master-snapshot-static". Now it finds "python3-otopi", but complains about a missing "python(abi)". The latter is actually installed, but I guess there is a collision between fc30 vs. fc32 :( pbar@laptop /etc/yum.repos.d $ sudo dnf install -y python3-otopi Latest oVirt master nightly snapshot 1.8 kB/s | 3.0 kB 00:01 Latest oVirt master additional nightly snapshot 1.8 kB/s | 3.0 kB 00:01 Error: Problem: conflicting requests - nothing provides python(abi) = 3.7 needed by python3-otopi-1.9.3-0.0.master.20200607140659.git775c2c5.*fc30*.noarch (try to add '--skip-broken' to skip uninstallable packages) pbar@laptop /etc/yum.repos.d $ sudo dnf install -y "python(abi)" Last metadata expiration check: 0:00:19 ago on Mon 27 Jul 2020 03:13:34 PM IDT. Package python27-2.7.18-2.*fc32*.x86_64 is already installed. Package python3-3.8.3-2.*fc32*.x86_64 is already installed. Dependencies resolved. Nothing to do. Complete! Any suggestions that do not require a clean new OS install? Thank you in advance! Pavel

3 4

Upgrade GWT to 2.9.0
by Radoslaw Szwajkowski 27 Jul '20

27 Jul '20

Hi all, we plan to bump the GWT version to 2.9.0 (patch [1]). Key changes: 1. support for building with Java 11 2. removing support for classic dev mode ( use super dev mode instead) 3. accumulated improvements and bug fixes (from versions 2.8.1, 2.8.2, 2.9.0) Please review and check if your areas are impacted. For details see BZ [2] best regards, radek [1] https://gerrit.ovirt.org/110359/ [2] https://bugzilla.redhat.com/1860309

2 2

vdsm.storage.exception.UnknownTask: Task id unknown (was: [oVirt Jenkins] ovirt-system-tests_he-basic-suite-master - Build # 1641 - Still Failing!)
by Yedidyah Bar David 21 Jul '20

21 Jul '20

On Wed, Jun 17, 2020 at 6:28 AM <jenkins(a)jenkins.phx.ovirt.org> wrote: > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1641/ This one failed while trying to create the disk image for the hosted-egnine VM: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1641… : 2020-06-16 23:03:20,527-0400 INFO ansible task start {'status': 'OK', 'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', 'ansible_task': 'ovirt.hosted_engine_setup : Add HE disks'} ... 2020-06-16 23:14:12,702-0400 DEBUG var changed: host "localhost" var "add_disks" type "<class 'dict'>" value: "{ ... "msg": "Timeout exceed while waiting on result state of the entity." https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1641… : 2020-06-16 23:03:22,612-04 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (default task-1) [16c24599-0048-44eb-a410-d39b7ce98712] CommandMultiAsyncTasks::attachTask: Attaching task '6b2a7648-748c-430b-94b6-5e3f719df2ac' to command 'fa81759d-c57a-4237-81e0-beb210faa64d'. 2020-06-16 23:03:22,659-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (default task-1) [16c24599-0048-44eb-a410-d39b7ce98712] Adding task '6b2a7648-748c-430b-94b6-5e3f719df2ac' (Parent Command 'AddImageFromScratch', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet.. 2020-06-16 23:03:22,699-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (default task-1) [16c24599-0048-44eb-a410-d39b7ce98712] BaseAsyncTask::startPollingTask: Starting to poll task '6b2a7648-748c-430b-94b6-5e3f719df2ac'. ... 2020-06-16 23:03:25,835-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] SPMAsyncTask::PollTask: Polling task '6b2a7648-748c-430b-94b6-5e3f719df2ac' (Parent Command 'AddImageFromScratch', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'success'. 2020-06-16 23:03:25,863-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] BaseAsyncTask::onTaskEndSuccess: Task '6b2a7648-748c-430b-94b6-5e3f719df2ac' (Parent Command 'AddImageFromScratch', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. But then: 2020-06-16 23:03:25,897-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-29) [16c24599-0048-44eb-a410-d39b7ce98712] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'AddImageFromScratch' succeeded, clearing tasks. 2020-06-16 23:03:25,897-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-29) [16c24599-0048-44eb-a410-d39b7ce98712] SPMAsyncTask::ClearAsyncTask: Attempting to clear task '6b2a7648-748c-430b-94b6-5e3f719df2ac' 2020-06-16 23:03:25,899-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-29) [16c24599-0048-44eb-a410-d39b7ce98712] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='3bcde3b4-b044-11ea-bbb6-5452c0a8c863', ignoreFailoverLimit='false', taskId='6b2a7648-748c-430b-94b6-5e3f719df2ac'}), log id: 481c2d3d 2020-06-16 23:03:25,900-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-29) [16c24599-0048-44eb-a410-d39b7ce98712] START, HSMClearTaskVDSCommand(HostName = lago-he-basic-suite-master-host-0, HSMTaskGuidBaseVDSCommandParameters:{hostId='85ecc51c-f2cb-46a1-9452-fd487399d8dd', taskId='6b2a7648-748c-430b-94b6-5e3f719df2ac'}), log id: 17360b3d ... 2020-06-16 23:03:26,054-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-29) [16c24599-0048-44eb-a410-d39b7ce98712] BaseAsyncTask::removeTaskFromDB: Removed task '6b2a7648-748c-430b-94b6-5e3f719df2ac' from DataBase But then: 2020-06-16 23:03:26,315-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-55) [7fe7b467] Command 'UploadStreamVDSCommand(HostName = lago-he-basic-suite-master-host-0, UploadStreamVDSCommandParameters:{hostId='85ecc51c-f2cb-46a1-9452-fd487399d8dd'})' execution failed: javax.net.ssl.SSLPeerUnverifiedException: Certificate for <lago-he-basic-suite-master-host-0.lago.local> doesn't match any of the subject alternative names: [lago-he-basic-suite-master-host-0.lago.local] 2020-06-16 23:03:26,315-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-55) [7fe7b467] FINISH, UploadStreamVDSCommand, return: , log id: 7e3a3e80 2020-06-16 23:03:26,316-04 ERROR [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-55) [7fe7b467] Command 'org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: javax.net.ssl.SSLPeerUnverifiedException: Certificate for <lago-he-basic-suite-master-host-0.lago.local> doesn't match any of the subject alternative names: [lago-he-basic-suite-master-host-0.lago.local] (Failed with error VDS_NETWORK_ERROR and code 5022) Any idea why? Anything changed in how we check the certificate? Perhaps related to upgrade to CentOS 8.2? And, how come it failed only this late? Don't we check the certificate earlier? Anyway, this left the host in "not responding" state, so: 2020-06-16 23:03:29,994-04 ERROR [org.ovirt.engine.core.bll.storage.disk.AddDiskCommandCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-79) [16c24599-0048-44eb-a410-d39b7ce98712] Failed to get volume info: org.ovirt.engine.core.common.errors.EngineException: EngineException: No host was found to perform the operation (Failed with error RESOURCE_MANAGER_VDS_NOT_FOUND and code 5004) And perhaps due to an unrelated issue, also: 2020-06-16 23:03:31,177-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMRevertTaskVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [16c24599-0048-44eb-a410-d39b7ce98712] Trying to revert unknown task '6b2a7648-748c-430b-94b6-5e3f719df2ac' I looked a bit also at: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1641… and see there some relevant stuff, but nothing I can spot about the root cause (e.g. the word "cert" does not appear there). Can anyone please have a look? Thanks. > Build Number: 1641 > Build Status: Still Failing > Triggered By: Started by timer > > ------------------------------------- > Changes Since Last Success: > ------------------------------------- > Changes for Build #1633 > [Marcin Sobczyk] ost-images: Drop rebasing of qcows > > [Ehud Yonasi] mock: fix yum repos injection. > > [Ehud Yonasi] onboard ost-images to stdci. > > > Changes for Build #1634 > [Marcin Sobczyk] ost-images: Drop rebasing of qcows > > > Changes for Build #1635 > [Marcin Sobczyk] ost-images: Drop rebasing of qcows > > > Changes for Build #1636 > [Marcin Sobczyk] ost-images: Drop rebasing of qcows > > > Changes for Build #1637 > [Marcin Sobczyk] ost-images: Drop rebasing of qcows > > > Changes for Build #1638 > [Marcin Sobczyk] ost-images: Drop rebasing of qcows > > [Ehud Yonasi] stdci_runner: update templates node to ost-images. > > > Changes for Build #1639 > [Marcin Sobczyk] ost-images: Drop rebasing of qcows > > > Changes for Build #1640 > [Yedidyah Bar David] Allow engine 20 minutes to come up after VM restart > > > Changes for Build #1641 > [Michal Skrivanek] test live storage migration again > > [Ehud Yonasi] poll: add ost-images to nightly. > > > > > ----------------- > Failed Tests: > ----------------- > No tests ran. -- Didi

5 12

Check OVF_STORE volume status task failures
by Artem Hrechanychenko 21 Jul '20

21 Jul '20

Hi all, maybe I miss some information, but I'm still have troubles with he installation using ost CI. Is that already fixed ? https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/10415/ https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/10415… 2020-07-20 06:36:04,455-0400 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:111 TASK [ovirt.hosted_engine_setup : Check OVF_STORE volume status] 2020-07-20 06:40:22,815-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:105 {'results': [{'cmd': ['vdsm-client', 'Volume', 'getInfo', 'storagepoolID=03a466fe-ca73-11ea-b77e-5452c0a8c863', 'storagedomainID=6685cca5-f0e1-4831-acdf-6f7b50596142', 'imageID=4d2f7009-5b79-4b44-b0ef-e152bc51649f', 'volumeID=7b953d0e-662d-4e72-9fdc-823ea867262b'], 'stdout': '{\n "apparentsize": "134217728",\n "capacity": "134217728",\n "children": [],\n "ctime": "1595241259",\n "description": "{\\"Updated\\":false,\\"Last Updated\\":null,\\"Storage Domains\\":[{\\"uuid\\":\\"6685cca5-f0e1-4831-acdf-6f7b50596142\\"}],\\"Disk Description\\":\\"OVF_STORE\\"}",\n "disktype": "OVFS",\n "domain": "6685cca5-f0e1-4831-acdf-6f7b50596142",\n "format": "RAW",\n "generation": 0,\n "image": "4d2f7009-5b79-4b44-b0ef-e152bc51649f",\n "lease": {\n "offset": 0,\n "owners": [],\n "path": "/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/6685cca5-f0e1-4831-acdf-6f7b50596142/images/4d2f7009-5b79-4b44-b0ef-e152bc51649f/7b953d0e-662d-4e72-9fdc-823ea867262b.lease",\n "version": null\n },\n "legality": "LEGAL",\n "mtime": "0",\n "parent": "00000000-0000-0000-0000-000000000000",\n "pool": "",\n "status": "OK",\n "truesize": "134217728",\n "type": "PREALLOCATED",\n "uuid": "7b953d0e-662d-4e72-9fdc-823ea867262b",\n "voltype": "LEAF"\n}', 'stderr': '', 'rc': 0, 'start': '2020-07-20 06:38:13.456845', 'end': '2020-07-20 06:38:13.897280', 'delta': '0:00:00.440435', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'vdsm-client Volume getInfo storagepoolID=03a466fe-ca73-11ea-b77e-5452c0a8c863 storagedomainID=6685cca5-f0e1-4831-acdf-6f7b50596142 imageID=4d2f7009-5b79-4b44-b0ef-e152bc51649f volumeID=7b953d0e-662d-4e72-9fdc-823ea867262b', 'warn': True, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['{', ' "apparentsize": "134217728",', ' "capacity": "134217728",', ' "children": [],', ' "ctime": "1595241259",', ' "description": "{\\"Updated\\":false,\\"Last Updated\\":null,\\"Storage Domains\\":[{\\"uuid\\":\\"6685cca5-f0e1-4831-acdf-6f7b50596142\\"}],\\"Disk Description\\":\\"OVF_STORE\\"}",', ' "disktype": "OVFS",', ' "domain": "6685cca5-f0e1-4831-acdf-6f7b50596142",', ' "format": "RAW",', ' "generation": 0,', ' "image": "4d2f7009-5b79-4b44-b0ef-e152bc51649f",', ' "lease": {', ' "offset": 0,', ' "owners": [],', ' "path": "/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/6685cca5-f0e1-4831-acdf-6f7b50596142/images/4d2f7009-5b79-4b44-b0ef-e152bc51649f/7b953d0e-662d-4e72-9fdc-823ea867262b.lease",', ' "version": null', ' },', ' "legality": "LEGAL",', ' "mtime": "0",', ' "parent": "00000000-0000-0000-0000-000000000000",', ' "pool": "",', ' "status": "OK",', ' "truesize": "134217728",', ' "type": "PREALLOCATED",', ' "uuid": "7b953d0e-662d-4e72-9fdc-823ea867262b",', ' "voltype": "LEAF"', '}'], 'stderr_lines': [], '_ansible_no_log': False, 'failed': True, 'attempts': 12, 'item': {'name': 'OVF_STORE', 'image_id': '7b953d0e-662d-4e72-9fdc-823ea867262b', 'id': '4d2f7009-5b79-4b44-b0ef-e152bc51649f'}, 'ansible_loop_var': 'item', '_ansible_item_label': {'name': 'OVF_STORE', 'image_id': '7b953d0e-662d-4e72-9fdc-823ea867262b', 'id': '4d2f7009-5b79-4b44-b0ef-e152bc51649f'}}, {'cmd': ['vdsm-client', 'Volume', 'getInfo', 'storagepoolID=03a466fe-ca73-11ea-b77e-5452c0a8c863', 'storagedomainID=6685cca5-f0e1-4831-acdf-6f7b50596142', 'imageID=044e384a-dedf-4589-8dfb-beca170138ee', 'volumeID=033d64fd-6f93-42be-84bc-082b03095ef3'], 'stdout': '{\n "apparentsize": "134217728",\n "capacity": "134217728",\n "children": [],\n "ctime": "1595241260",\n "description": "{\\"Updated\\":false,\\"Last Updated\\":null,\\"Storage Domains\\":[{\\"uuid\\":\\"6685cca5-f0e1-4831-acdf-6f7b50596142\\"}],\\"Disk Description\\":\\"OVF_STORE\\"}",\n "disktype": "OVFS",\n "domain": "6685cca5-f0e1-4831-acdf-6f7b50596142",\n "format": "RAW",\n "generation": 0,\n "image": "044e384a-dedf-4589-8dfb-beca170138ee",\n "lease": {\n "offset": 0,\n "owners": [],\n "path": "/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/6685cca5-f0e1-4831-acdf-6f7b50596142/images/044e384a-dedf-4589-8dfb-beca170138ee/033d64fd-6f93-42be-84bc-082b03095ef3.lease",\n "version": null\n },\n "legality": "LEGAL",\n "mtime": "0",\n "parent": "00000000-0000-0000-0000-000000000000",\n "pool": "",\n "status": "OK",\n "truesize": "134217728",\n "type": "PREALLOCATED",\n "uuid": "033d64fd-6f93-42be-84bc-082b03095ef3",\n "voltype": "LEAF"\n}', 'stderr': '', 'rc': 0, 'start': '2020-07-20 06:40:22.272462', 'end': '2020-07-20 06:40:22.621943', 'delta': '0:00:00.349481', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'vdsm-client Volume getInfo storagepoolID=03a466fe-ca73-11ea-b77e-5452c0a8c863 storagedomainID=6685cca5-f0e1-4831-acdf-6f7b50596142 imageID=044e384a-dedf-4589-8dfb-beca170138ee volumeID=033d64fd-6f93-42be-84bc-082b03095ef3', 'warn': True, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['{', ' "apparentsize": "134217728",', ' "capacity": "134217728",', ' "children": [],', ' "ctime": "1595241260",', ' "description": "{\\"Updated\\":false,\\"Last Updated\\":null,\\"Storage Domains\\":[{\\"uuid\\":\\"6685cca5-f0e1-4831-acdf-6f7b50596142\\"}],\\"Disk Description\\":\\"OVF_STORE\\"}",', ' "disktype": "OVFS",', ' "domain": "6685cca5-f0e1-4831-acdf-6f7b50596142",', ' "format": "RAW",', ' "generation": 0,', ' "image": "044e384a-dedf-4589-8dfb-beca170138ee",', ' "lease": {', ' "offset": 0,', ' "owners": [],', ' "path": "/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/6685cca5-f0e1-4831-acdf-6f7b50596142/images/044e384a-dedf-4589-8dfb-beca170138ee/033d64fd-6f93-42be-84bc-082b03095ef3.lease",', ' "version": null', ' },', ' "legality": "LEGAL",', ' "mtime": "0",', ' "parent": "00000000-0000-0000-0000-000000000000",', ' "pool": "",', ' "status": "OK",', ' "truesize": "134217728",', ' "type": "PREALLOCATED",', ' "uuid": "033d64fd-6f93-42be-84bc-082b03095ef3",', ' "voltype": "LEAF"', '}'], 'stderr_lines': [], '_ansible_no_log': False, 'failed': True, 'attempts': 12, 'item': {'name': 'OVF_STORE', 'image_id': '033d64fd-6f93-42be-84bc-082b03095ef3', 'id': '044e384a-dedf-4589-8dfb-beca170138ee'}, 'ansible_loop_var': 'item', '_ansible_item_label': {'name': 'OVF_STORE', 'image_id': '033d64fd-6f93-42be-84bc-082b03095ef3', 'id': '044e384a-dedf-4589-8dfb-beca170138ee'}}], 'changed': True, 'msg': 'All items completed'} 2020-07-20 06:40:22,919-0400 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'cmd': ['vdsm-client', 'Volume', 'getInfo', 'storagepoolID=03a466fe-ca73-11ea-b77e-5452c0a8c863', 'storagedomainID=6685cca5-f0e1-4831-acdf-6f7b50596142', 'imageID=4d2f7009-5b79-4b44-b0ef-e152bc51649f', 'volumeID=7b953d0e-662d-4e72-9fdc-823ea867262b'], 'stdout': '{\n "apparentsize": "134217728",\n "capacity": "134217728",\n "children": [],\n "ctime": "1595241259",\n "description": "{\\"Updated\\":false,\\"Last Updated\\":null,\\"Storage Domains\\":[{\\"uuid\\":\\"6685cca5-f0e1-4831-acdf-6f7b50596142\\"}],\\"Disk Description\\":\\"OVF_STORE\\"}",\n "disktype": "OVFS",\n "domain": "6685cca5-f0e1-4831-acdf-6f7b50596142",\n "format": "RAW",\n "generation": 0,\n "image": "4d2f7009-5b79-4b44-b0ef-e152bc51649f",\n "lease": {\n "offset": 0,\n "owners": [],\n "path": "/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/6685cca5-f0e1-4831-acdf-6f7b50596142/images/4d2f7009-5b79-4b44-b0ef-e152bc51649f/7b953d0e-662d-4e72-9fdc-823ea867262b.lease",\n "version": null\n },\n "legality": "LEGAL",\n "mtime": "0",\n "parent": "00000000-0000-0000-0000-000000000000",\n "pool": "",\n "status": "OK",\n "truesize": "134217728",\n "type": "PREALLOCATED",\n "uuid": "7b953d0e-662d-4e72-9fdc-823ea867262b",\n "voltype": "LEAF"\n}', 'stderr': '', 'rc': 0, 'start': '2020-07-20 06:38:13.456845', 'end': '2020-07-20 06:38:13.897280', 'delta': '0:00:00.440435', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'vdsm-client Volume getInfo storagepoolID=03a466fe-ca73-11ea-b77e-5452c0a8c863 storagedomainID=6685cca5-f0e1-4831-acdf-6f7b50596142 imageID=4d2f7009-5b79-4b44-b0ef-e152bc51649f volumeID=7b953d0e-662d-4e72-9fdc-823ea867262b', 'warn': True, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['{', ' "apparentsize": "134217728",', ' "capacity": "134217728",', ' "children": [],', ' "ctime": "1595241259",', ' "description": "{\\"Updated\\":false,\\"Last Updated\\":null,\\"Storage Domains\\":[{\\"uuid\\":\\"6685cca5-f0e1-4831-acdf-6f7b50596142\\"}],\\"Disk Description\\":\\"OVF_STORE\\"}",', ' "disktype": "OVFS",', ' "domain": "6685cca5-f0e1-4831-acdf-6f7b50596142",', ' "format": "RAW",', ' "generation": 0,', ' "image": "4d2f7009-5b79-4b44-b0ef-e152bc51649f",', ' "lease": {', ' "offset": 0,', ' "owners": [],', ' "path": "/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/6685cca5-f0e1-4831-acdf-6f7b50596142/images/4d2f7009-5b79-4b44-b0ef-e152bc51649f/7b953d0e-662d-4e72-9fdc-823ea867262b.lease",', ' "version": null', ' },', ' "legality": "LEGAL",', ' "mtime": "0",', ' "parent": "00000000-0000-0000-0000-000000000000",', ' "pool": "",', ' "status": "OK",', ' "truesize": "134217728",', ' "type": "PREALLOCATED",', ' "uuid": "7b953d0e-662d-4e72-9fdc-823ea867262b",', ' "voltype": "LEAF"', '}'], 'stderr_lines': [], '_ansible_no_log': False, 'failed': True, 'attempts': 12, 'item': {'name': 'OVF_STORE', 'image_id': '7b953d0e-662d-4e72-9fdc-823ea867262b', 'id': '4d2f7009-5b79-4b44-b0ef-e152bc51649f'}, 'ansible_loop_var': 'item', '_ansible_item_label': {'name': 'OVF_STORE', 'image_id': '7b953d0e-662d-4e72-9fdc-823ea867262b', 'id': '4d2f7009-5b79-4b44-b0ef-e152bc51649f'}} 2020-07-20 06:40:23,026-0400 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'cmd': ['vdsm-client', 'Volume', 'getInfo', 'storagepoolID=03a466fe-ca73-11ea-b77e-5452c0a8c863', 'storagedomainID=6685cca5-f0e1-4831-acdf-6f7b50596142', 'imageID=044e384a-dedf-4589-8dfb-beca170138ee', 'volumeID=033d64fd-6f93-42be-84bc-082b03095ef3'], 'stdout': '{\n "apparentsize": "134217728",\n "capacity": "134217728",\n "children": [],\n "ctime": "1595241260",\n "description": "{\\"Updated\\":false,\\"Last Updated\\":null,\\"Storage Domains\\":[{\\"uuid\\":\\"6685cca5-f0e1-4831-acdf-6f7b50596142\\"}],\\"Disk Description\\":\\"OVF_STORE\\"}",\n "disktype": "OVFS",\n "domain": "6685cca5-f0e1-4831-acdf-6f7b50596142",\n "format": "RAW",\n "generation": 0,\n "image": "044e384a-dedf-4589-8dfb-beca170138ee",\n "lease": {\n "offset": 0,\n "owners": [],\n "path": "/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/6685cca5-f0e1-4831-acdf-6f7b50596142/images/044e384a-dedf-4589-8dfb-beca170138ee/033d64fd-6f93-42be-84bc-082b03095ef3.lease",\n "version": null\n },\n "legality": "LEGAL",\n "mtime": "0",\n "parent": "00000000-0000-0000-0000-000000000000",\n "pool": "",\n "status": "OK",\n "truesize": "134217728",\n "type": "PREALLOCATED",\n "uuid": "033d64fd-6f93-42be-84bc-082b03095ef3",\n "voltype": "LEAF"\n}', 'stderr': '', 'rc': 0, 'start': '2020-07-20 06:40:22.272462', 'end': '2020-07-20 06:40:22.621943', 'delta': '0:00:00.349481', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'vdsm-client Volume getInfo storagepoolID=03a466fe-ca73-11ea-b77e-5452c0a8c863 storagedomainID=6685cca5-f0e1-4831-acdf-6f7b50596142 imageID=044e384a-dedf-4589-8dfb-beca170138ee volumeID=033d64fd-6f93-42be-84bc-082b03095ef3', 'warn': True, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['{', ' "apparentsize": "134217728",', ' "capacity": "134217728",', ' "children": [],', ' "ctime": "1595241260",', ' "description": "{\\"Updated\\":false,\\"Last Updated\\":null,\\"Storage Domains\\":[{\\"uuid\\":\\"6685cca5-f0e1-4831-acdf-6f7b50596142\\"}],\\"Disk Description\\":\\"OVF_STORE\\"}",', ' "disktype": "OVFS",', ' "domain": "6685cca5-f0e1-4831-acdf-6f7b50596142",', ' "format": "RAW",', ' "generation": 0,', ' "image": "044e384a-dedf-4589-8dfb-beca170138ee",', ' "lease": {', ' "offset": 0,', ' "owners": [],', ' "path": "/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/6685cca5-f0e1-4831-acdf-6f7b50596142/images/044e384a-dedf-4589-8dfb-beca170138ee/033d64fd-6f93-42be-84bc-082b03095ef3.lease",', ' "version": null', ' },', ' "legality": "LEGAL",', ' "mtime": "0",', ' "parent": "00000000-0000-0000-0000-000000000000",', ' "pool": "",', ' "status": "OK",', ' "truesize": "134217728",', ' "type": "PREALLOCATED",', ' "uuid": "033d64fd-6f93-42be-84bc-082b03095ef3",', ' "voltype": "LEAF"', '}'], 'stderr_lines': [], '_ansible_no_log': False, 'failed': True, 'attempts': 12, 'item': {'name': 'OVF_STORE', 'image_id': '033d64fd-6f93-42be-84bc-082b03095ef3', 'id': '044e384a-dedf-4589-8dfb-beca170138ee'}, 'ansible_loop_var': 'item', '_ansible_item_label': {'name': 'OVF_STORE', 'image_id': '033d64fd-6f93-42be-84bc-082b03095ef3', 'id': '044e384a-dedf-4589-8dfb-beca170138ee'}} 2020-07-20 06:40:23,128-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:105 PLAY RECAP [localhost] : ok: 69 changed: 19 unreachable: 0 skipped: 10 failed: 1

3 2

ovirt-engine has been tagged (ovirt-engine-4.4.1.10)
by Tal Nisan 21 Jul '20

21 Jul '20

1 0

el8 CI breaking on iprocess mirrors
by Amit Bawer 20 Jul '20

20 Jul '20

Hi This can be seen on [1] 13:47:57 Error: Error downloading packages: 13:47:57 Cannot download x86_64/ioprocess-1.4.1-1.202007151811.gitc41863d.el8.x86_64.rpm: All mirrors were tried [1] https://jenkins.ovirt.org/job/vdsm_standard-check-patch/22702/consoleFull Thanks

2 2

Re: [ovirt-users] Parent checkpoint ID does not match the actual leaf checkpoint
by Nir Soffer 19 Jul '20

19 Jul '20

On Sun, Jul 19, 2020 at 5:38 PM Łukasz Kołaciński <l.kolacinski(a)storware.eu> wrote: > Hello, > Thanks to previous answers, I was able to make backups. Unfortunately, we > had some infrastructure issues and after the host reboots new problems > appeared. I am not able to do any backup using the commands that worked > yesterday. I looked through the logs and there is something like this: > > 2020-07-17 15:06:30,644+02 ERROR > [org.ovirt.engine.core.bll.StartVmBackupCommand] > (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-54) > [944a1447-4ea5-4a1c-b971-0bc612b6e45e] Failed to execute VM backup > operation 'StartVmBackup': {}: > org.ovirt.engine.core.common.errors.EngineException: EngineException: > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: > VDSGenericException: VDSErrorException: Failed to StartVmBackupVDS, error = > Checkpoint Error: {'parent_checkpoint_id': None, 'leaf_checkpoint_id': > 'cd078706-84c0-4370-a6ec-654ccd6a21aa', 'vm_id': > '116aa6eb-31a1-43db-9b1e-ad6e32fb9260', 'reason': '*Parent checkpoint ID > does not match the actual leaf checkpoint*'}, code = 1610 (Failed with > error unexpected and code 16) > > It looks like engine sent: parent_checkpoint_id: None This issue was fix in engine few weeks ago. Which engine and vdsm versions are you testing? > at > deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:114) > at > deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) > at > deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2114) > at > deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.StartVmBackupCommand.performVmBackupOperation(StartVmBackupCommand.java:368) > at > deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.StartVmBackupCommand.runVmBackup(StartVmBackupCommand.java:225) > at > deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.StartVmBackupCommand.performNextOperation(StartVmBackupCommand.java:199) > at > deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback.childCommandsExecutionEnded(SerialChildCommandsExecutionCallback.java:32) > at > deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.ChildCommandsCallbackBase.doPolling(ChildCommandsCallbackBase.java:80) > at > deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:175) > at > deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at > java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) > at > org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) > at > org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:834) > at > org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) > > > And the last error is: > > 2020-07-17 15:13:45,835+02 ERROR > [org.ovirt.engine.core.bll.StartVmBackupCommand] > (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-14) > [f553c1f2-1c99-4118-9365-ba6b862da936] Failed to execute VM backup > operation 'GetVmBackupInfo': {}: > org.ovirt.engine.core.common.errors.EngineException: EngineException: > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: > VDSGenericException: VDSErrorException: Failed to GetVmBackupInfoVDS, error > = No such backup Error: {'vm_id': '116aa6eb-31a1-43db-9b1e-ad6e32fb9260', > 'backup_id': 'bf1c26f7-c3e5-437c-bb5a-255b8c1b3b73', 'reason': '*VM > backup not exists: Domain backup job id not found: no domain backup job > present'*}, code = 1601 (Failed with error unexpected and code 16) > > This is likely a result of the first error. If starting backup failed the backup entity is deleted. > (these errors are from full backup) > > Like I said this is very strange because everything was working correctly. > > > Regards > > Łukasz Kołaciński > > Junior Java Developer > > e-mail: l.kolacinski(a)storware.eu > <m.helbert(a)storware.eu> > > > > > *[image: STORWARE]* <http://www.storware.eu/> > > > > *ul. Leszno 8/44 01-192 Warszawa www.storware.eu > <https://www.storware.eu/>* > > *[image: facebook]* <https://www.facebook.com/storware> > > *[image: twitter]* <https://twitter.com/storware> > > *[image: linkedin]* <https://www.linkedin.com/company/storware> > > *[image: Storware_Stopka_09]* > <https://www.youtube.com/channel/UCKvLitYPyAplBctXibFWrkw> > > > > *Storware Spółka z o.o. nr wpisu do ewidencji KRS dla M.St. Warszawa > 000510131* *, NIP 5213672602.** Wiadomość ta jest przeznaczona jedynie > dla osoby lub podmiotu, który jest jej adresatem i może zawierać poufne > i/lub uprzywilejowane informacje. Zakazane jest jakiekolwiek przeglądanie, > przesyłanie, rozpowszechnianie lub inne wykorzystanie tych informacji lub > podjęcie jakichkolwiek działań odnośnie tych informacji przez osoby lub > podmioty inne niż zamierzony adresat. Jeżeli Państwo otrzymali przez > pomyłkę tę informację prosimy o poinformowanie o tym nadawcy i usunięcie > tej wiadomości z wszelkich komputerów. **This message is intended only > for the person or entity to which it is addressed and may contain > confidential and/or privileged material. Any review, retransmission, > dissemination or other use of, or taking of any action in reliance upon, > this information by persons or entities other than the intended recipient > is prohibited. If you have received this message in error, please contact > the sender and remove the material from all of your computer systems.* > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/S3PLYPOZGT6TL… >

1 0

Re: Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
by Lev Veyde 19 Jul '20

19 Jul '20

Hi Nir, Looks like the issue is fixed now, and everything is back to normal. Thanks in advance, On Sun, Jul 19, 2020 at 2:34 PM Lev Veyde <lveyde(a)redhat.com> wrote: > Hi Nir, > > Got this error only on the re-build attempt. > And looks like it's the Jenkins slave that ran out of disk space, not our > server. > > And the job was failing before on something else, if I understand > correctly it was timing out while waiting on some cron job to complete it's > work. > But I think that the CI team could have better answers for that. > > Thanks in advance, > > On Sun, Jul 19, 2020 at 2:27 PM Nir Soffer <nsoffer(a)redhat.com> wrote: > >> On Sun, Jul 19, 2020 at 2:18 PM Lev Veyde <lveyde(a)redhat.com> wrote: >> >>> Hi, >>> >>> Looks like my job re-build attempt fails on not enough disk space: >>> >>> *14:15:32* Traceback (most recent call last): >>> >>> >> Disk is full? >> >> >>> <https://jenkins.ovirt.org/job/ovirt_master_publish-rpms_nightly/1806/consol…>*14:15:32* IOError: [Errno 28] No space left on device >>> >>> >>> On Sun, Jul 19, 2020 at 2:15 PM Lev Veyde <lveyde(a)redhat.com> wrote: >>> >>>> Hi, >>>> >>>> Looks like the nightly snapshot job has failed, and it it caused the >>>> removal of all -snapshot repos. >>>> >>>> I'm now trying to re-run the failed job: >>>> https://jenkins.ovirt.org/job/ovirt_master_publish-rpms_nightly/1806/console >>>> >>>> But we need to understand why it failed - is it just the timeout issue >>>> we see in the log, or there is something wrong with it? >>>> >>>> Currently all snapshot repos are broken (i.e. 4.2, 4.3 and master). >>>> >>>> Thanks in advance, >>>> >>>> On Sun, Jul 19, 2020 at 1:38 PM Nir Soffer <nsoffer(a)redhat.com> wrote: >>>> >>>>> Downloading ovirt-release-master.rpm is Forbidden now. This breaks all >>>>> developers >>>>> workflows. >>>>> >>>>> Please fix as soon as possible. >>>>> >>>>> Example failure when trying to build vdsm centos-8 docker image: >>>>> >>>>> [MIRROR] ovirt-release-master.rpm: Status code: 403 for >>>>> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm >>>>> [FAILED] ovirt-release-master.rpm: Status code: 403 for >>>>> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm >>>>> Status code: 403 for >>>>> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm >>>>> >>>>> >>>> >>>> -- >>>> >>>> Lev Veyde >>>> >>>> Senior Software Engineer, RHCE | RHCVA | MCITP >>>> >>>> Red Hat Israel >>>> >>>> <https://www.redhat.com> >>>> >>>> lev(a)redhat.com | lveyde(a)redhat.com >>>> <https://red.ht/sig> >>>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >>>> >>> >>> >>> -- >>> >>> Lev Veyde >>> >>> Senior Software Engineer, RHCE | RHCVA | MCITP >>> >>> Red Hat Israel >>> >>> <https://www.redhat.com> >>> >>> lev(a)redhat.com | lveyde(a)redhat.com >>> <https://red.ht/sig> >>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >>> >> > > -- > > Lev Veyde > > Senior Software Engineer, RHCE | RHCVA | MCITP > > Red Hat Israel > > <https://www.redhat.com> > > lev(a)redhat.com | lveyde(a)redhat.com > <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > -- Lev Veyde Senior Software Engineer, RHCE | RHCVA | MCITP Red Hat Israel <https://www.redhat.com> lev(a)redhat.com | lveyde(a)redhat.com <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

2 2

ovirt-engine has been tagged (ovirt-engine-4.4.1.9)
by Tal Nisan 16 Jul '20

16 Jul '20

1 0

Huge queue for OST check patch job
by Marcin Sobczyk 16 Jul '20

16 Jul '20

Hi All, 'check-patch' pipeline for OST has currently a huge queue of jobs to be executed, some stuck since 14 hrs [1]. When testing patches for OST please consider trimming down the triggered pipelines like this [2], and marking the patch with "WIP" until it's ready. Also, when doing a whole series of patches, getting them in more quickly will help a lot too. Thanks, Marcin [1] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… [2] https://gerrit.ovirt.org/#/c/110304/6/stdci.yaml

3 3

Re: [ovirt-users] Problem with backuping ovirt 4.4 with SDK
by Nir Soffer 15 Jul '20

15 Jul '20

On Tue, Jul 14, 2020 at 9:33 AM Łukasz Kołaciński <l.kolacinski(a)storware.eu> wrote: > Hello, > Hi Lukaz, Lets move the discussion to devel(a)ovirt.org, I think it will be more productive. Also, always CC me and Eyal on incremental backup questions for a quicker response. > I am trying to do full backup on ovirt 4.4 using sdk. > Which version of oVirt? libvirt? > I used steps from this youtube video: > https://www.youtube.com/watch?v=E2VWUVcycj4 and I got error after running > backup_vm.py. I see that sdk has imported disks and created backup entity > and then I got sdk.NotFoundError exception. > This means that starting backup failed. Unfortunately the API does not have a good way to get the error that caused the backup to fai. You should be able to see the error in the event log in the UI, and in engine log. > I also tried to do full backup with API and after finalizing backup > disappeared (I think) > So backup from the API was successful? Backups are expected to disappear, they are temporary objects used to manage the backup process. Once the backup process was finished you can do nothing with the backup object, and you cannot fetch the same backup data again. > and I couldn't try incremental. > The fact that the backup disappeared should not prevent the next backup. After you create a backup, you need to poll backup status until the backup is ready. while backup.phase != BackupPhase.READY: time.sleep(1) backup = backup_service.get()# to_checkpoint_id will be used as If the backup does not end in ready state, it failed, and you cannot do anything with this backup. When the backup is ready, you can fetch the to_checkpoint_id created for this backup. checkpoint_id = backup.to_checkpoint_id At this point you need to persist the checkpoint id. This will be used to create the incremental backup. [ 0.0 ] Starting full backup for VM '51708c8e-6671-480b-b2d8-199a1af9cbdc' > Password: > [ 4.2 ] Waiting until backup 0458bf7f-868c-4859-9fa7-767b3ec62b52 is > ready > Traceback (most recent call last): > File "./backup_vm.py", line 343, in start_backup > backup = backup_service.get() > File "/usr/lib64/python3.7/site-packages/ovirtsdk4/services.py", line > 32333, in get > return self._internal_get(headers, query, wait) > File "/usr/lib64/python3.7/site-packages/ovirtsdk4/service.py", line > 211, in _internal_get > return future.wait() if wait else future > File "/usr/lib64/python3.7/site-packages/ovirtsdk4/service.py", line 55, > in wait > return self._code(response) > File "/usr/lib64/python3.7/site-packages/ovirtsdk4/service.py", line > 208, in callback > self._check_fault(response) > File "/usr/lib64/python3.7/site-packages/ovirtsdk4/service.py", line > 130, in _check_fault > body = self._internal_read_body(response) > File "/usr/lib64/python3.7/site-packages/ovirtsdk4/service.py", line > 312, in _internal_read_body > self._raise_error(response) > File "/usr/lib64/python3.7/site-packages/ovirtsdk4/service.py", line > 118, in _raise_error > raise error > ovirtsdk4.NotFoundError: HTTP response code is 404. > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "./backup_vm.py", line 476, in <module> > main() > File "./backup_vm.py", line 173, in main > args.command(args) > File "./backup_vm.py", line 230, in cmd_start > backup = start_backup(connection, args) > File "./backup_vm.py", line 345, in start_backup > raise RuntimeError("Backup {} failed".format(backup.id)) > RuntimeError: Backup 0458bf7f-868c-4859-9fa7-767b3ec62b52 failed > This is correct, backup has failed. Please check the event log to understand the failure. Eyal, can you show how to get the error from the backup using the SDK, in a way that can be used by a program? e.g. a public error code that can be used to decide on the next step, and an error message that can be used for displaying error to users of the backup application. This should be added to the backup_vm.py example. Nir

2 1

code merges in ovirt-engine repo
by Michal Skrivanek 15 Jul '20

15 Jul '20

Hi all, we’re moving to 4.4.z development now and we need to keep a closer eye on automation results and making sure the build is not broken. For these reasons we’re considering moving to a similar model as vdsm, having a smaller set of people with merge rights to make sure the patches get in in the right order and they meet our sanity standards (OST, bug’s TM) Any objections/comments? Thanks, michal

3 6

oVirt 2020 online conference - free registration now open
by Sandro Bonazzola 15 Jul '20

15 Jul '20

It is our pleasure to renew our invite to oVirt 2020 online conference and announce free registration <https://www.eventbrite.it/e/ovirt-2020-online-conference-registration-11198…> is now open. The conference,organized by oVirt community, will take place online on Monday, September 7th 2020! You can read updated information about the conference at https://blogs.ovirt.org/ovirt-2020-online-conference/ A kind reminder that if you'd like to present at the conference, the deadline for the presentation proposal is July 26th 2020. -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

Re: device compatibility interface for live migration with assigned devices
by Sean Mooney 14 Jul '20

14 Jul '20

resending with full cc list since i had this typed up i would blame my email provier but my email client does not seam to like long cc lists. we probably want to continue on alex's thread to not split the disscusion. but i have responed inline with some example of how openstack schdules and what i ment by different mdev_types On Tue, 2020-07-14 at 20:29 +0100, Sean Mooney wrote: > On Tue, 2020-07-14 at 11:01 -0600, Alex Williamson wrote: > > On Tue, 14 Jul 2020 13:33:24 +0100 > > Sean Mooney <smooney(a)redhat.com> wrote: > > > > > On Tue, 2020-07-14 at 11:21 +0100, Daniel P. Berrangé wrote: > > > > On Tue, Jul 14, 2020 at 07:29:57AM +0800, Yan Zhao wrote: > > > > > hi folks, > > > > > we are defining a device migration compatibility interface that helps upper > > > > > layer stack like openstack/ovirt/libvirt to check if two devices are > > > > > live migration compatible. > > > > > The "devices" here could be MDEVs, physical devices, or hybrid of the two. > > > > > e.g. we could use it to check whether > > > > > - a src MDEV can migrate to a target MDEV, > > > > > > mdev live migration is completely possible to do but i agree with Dan barrange's comments > > > from the point of view of openstack integration i dont see calling out to a vender sepecific > > > tool to be an accpetable > > > > As I replied to Dan, I'm hoping Yan was referring more to vendor > > specific knowledge rather than actual tools. > > > > > solutions for device compatiablity checking. the sys filesystem > > > that describs the mdevs that can be created shoudl also > > > contain the relevent infomation such > > > taht nova could integrate it via libvirt xml representation or directly retrive the > > > info from > > > sysfs. > > > > > - a src VF in SRIOV can migrate to a target VF in SRIOV, > > > > > > so vf to vf migration is not possible in the general case as there is no standarised > > > way to transfer teh device state as part of the siorv specs produced by the pci-sig > > > as such there is not vender neutral way to support sriov live migration. > > > > We're not talking about a general case, we're talking about physical > > devices which have vfio wrappers or hooks with device specific > > knowledge in order to support the vfio migration interface. The point > > is that a discussion around vfio device migration cannot be limited to > > mdev devices. > > ok upstream in openstack at least we do not plan to support generic livemigration > for passthough devivces. we cheat with network interfaces since in generaly operating > systems handel hotplug of a nic somewhat safely so wehre no abstraction layer like > an mdev is present or a macvtap device we hot unplug the nic before the migration > and attach a new one after. for gpus or crypto cards this likely would not be viable > since you can bond generic hardware devices to hide the removal and readdtion of a generic > pci device. we were hoping that there would be a convergenca around MDEVs as a way to provide > that abstraction going forward for generic device or some other new mechanisum in the future. > > > > > > > - a src MDEV can migration to a target VF in SRIOV. > > > > > > that also makes this unviable > > > > > (e.g. SIOV/SRIOV backward compatibility case) > > > > > > > > > > The upper layer stack could use this interface as the last step to check > > > > > if one device is able to migrate to another device before triggering a real > > > > > live migration procedure. > > > > > > well actully that is already too late really. ideally we would want to do this compaiablity > > > check much sooneer to avoid the migration failing. in an openstack envionment at least > > > by the time we invoke libvirt (assuming your using the libvirt driver) to do the migration we have alreaedy > > > finished schduling the instance to the new host. if if we do the compatiablity check at this point > > > and it fails then the live migration is aborted and will not be retired. These types of late check lead to a > > > poor user experince as unless you check the migration detial it basically looks like the migration was ignored > > > as it start to migrate and then continuge running on the orgininal host. > > > > > > when using generic pci passhotuhg with openstack, the pci alias is intended to reference a single vendor > > > id/product > > > id so you will have 1+ alias for each type of device. that allows openstack to schedule based on the availability > > > of > > > a > > > compatibale device because we track inventories of pci devices and can query that when selecting a host. > > > > > > if we were to support mdev live migration in the future we would want to take the same declarative approch. > > > 1 interospec the capability of the deivce we manage > > > 2 create inventories of the allocatable devices and there capabilities > > > 3 schdule the instance to a host based on the device-type/capabilities and claim it atomicly to prevent raceces > > > 4 have the lower level hyperviors do addtional validation if need prelive migration. > > > > > > this proposal seams to be targeting extending step 4 where as ideally we should focuse on providing the info that > > > would > > > be relevant in set 1 preferably in a vendor neutral way vai a kernel interface like /sys. > > > > I think this is reading a whole lot into the phrase "last step". We > > want to make the information available for a management engine to > > consume as needed to make informed decisions regarding likely > > compatible target devices. > > well openstack as a management engin has 3 stages for schdule and asignment,. > in respocne to a live migration request the api does minimal valaidation then hand the task off to the conductor > service > ot orchestrate. the conductor invokes an rpc to the schduler service which makes a rest call to the plamcent service. > the placment cervice generate a set of allocation candiate for host based on qunataive and qulaitivly > queries agains an abstract resouce provider tree model of the hosts. > currently device pasthough is not modeled in placment so plamcnet is basicaly returning a set of host that have enough > cpu ram and disk for the instance. in the spacial of vGPU they technically are modelled in placement but not in a way > that would gurarentee compatiablity for migration. a generic pci device request is haneled in the second phase of > schduling called filtering and weighing. in this pahse the nova schuleer apply a series of filter to the list of host > returned by plamcnet to assert things like anit afintiy, tenant isolation or in the case of this converation nuam > affintiy and pci device avaiablity. when we have filtered the posible set of host down to X number we weigh the > listing > to select an optimal host and set of alternitive hosts. we then enter the code that this mail suggest modfiying which > does an rpc call to the destiation host form teh conductor to have it assert compatiablity which internaly calls back > to > the sourc host. > > so my point is we have done a lot of work by the time we call check_can_live_migrate_destination and failing > at this point is considerd quite a late failure but its still better then failing when qemu actully tries to migrate. > in general we would prefer to move compatiablity check as early in that workflow as possible but to be fair we dont > actully check cpu model compatiablity until check_can_live_migrate_destination. > https://github.com/openstack/nova/blob/8988316b8c132c9662dea6cf0345975e87ce… > > if we needed too we could read the version string on the source and write the version string on the dest at this > point. > doing so however would be considerd, inelegant, we have found this does not scale as the first copmpatabilty check. > for cpu for example there are way to filter hosts by groups sets fo host with the same cpu or filtering on cpu feature > flags that happen in the placment or filter stage both of which are very early and cheap to do at runtime. > > the "read for version, write for compatibility" workflow could be used as a final safe check if required but > probing for compatibility via writes is basicaly considered an anti patteren in openstack. we try to always > assert compatibility by reading avaiable info and asserting requirement over it not testing to see if it works. > > this has come up in the past in the context of virtio feature flag where the idea of spawning an instrance or trying > to add a virtio port to ovs dpdk that reqested a specific feature flag was rejected as unacceptable from a performance > and security point of view. > > > > > > > > we are not sure if this interface is of value or help to you. please don't > > > > > hesitate to drop your valuable comments. > > > > > > > > > > > > > > > (1) interface definition > > > > > The interface is defined in below way: > > > > > > > > > > __ userspace > > > > > /\ \ > > > > > / \write > > > > > / read \ > > > > > ________/__________ ___\|/_____________ > > > > > | migration_version | | migration_version |-->check migration > > > > > --------------------- --------------------- compatibility > > > > > device A device B > > > > > > > > > > > > > > > a device attribute named migration_version is defined under each device's > > > > > sysfs node. e.g. (/sys/bus/pci/devices/0000\:00\:02.0/$mdev_UUID/migration_version). > > > > > > this might be useful as we could tag the inventory with the migration version and only might to > > > devices with the same version > > > > Is cross version compatibility something that you'd consider using? > > yes but it would depend on what cross version actully ment. > > the version of an mdev is not something we would want to be exposed to endusers. > it would be a security risk to do so as the version sting would potentaily allow the untrused user > to discover if a device has an unpatch vulnerablity. as a result in the context of live migration > we can only support cross verion compatiabilyt if the device in the guest does not alter as > part of the migration and the behavior does not change. > > going form version 1.0 with feature X to verions 1.1 with feature X and Y but only X enabled would > be fine. going gorm 1.0 to 2.0 where thre is only feature Y would not be ok. > being abstract makes it a little harder to readabout but i guess i would sumerisei if its > transparent to the guest for the lifetime of the qemu process then its ok for the backing version to change. > if a vm is rebooted its also ok fo the vm to pick up feature Y form the 1.1 device although at that point > it could not be migrated back to the 1.0 host as it now has feature X and Y and 1.0 only has X so that woudl be > an obserable change if it was drop as a reult of the live migration. > > > > > > > userspace tools read the migration_version as a string from the source device, > > > > > and write it to the migration_version sysfs attribute in the target device. > > > > > > this would not be useful as the schduler cannot directlly connect to the compute host > > > and even if it could it would be extreamly slow to do this for 1000s of hosts and potentally > > > multiple devices per host. > > > > Seems similar to Dan's requirement, looks like the 'read for version, > > write for compatibility' test idea isn't really viable. > > its ineffiecnt and we have reject adding such test in the case of virtio-feature flag compatiabilty > in the past, so its more an option of last resourt if we have no other way to support compatiablity > checking. > > > > > > > > > > > > The userspace should treat ANY of below conditions as two devices not compatible: > > > > > - any one of the two devices does not have a migration_version attribute > > > > > - error when reading from migration_version attribute of one device > > > > > - error when writing migration_version string of one device to > > > > > migration_version attribute of the other device > > > > > > > > > > The string read from migration_version attribute is defined by device vendor > > > > > driver and is completely opaque to the userspace. > > > > > > opaque vendor specific stings that higher level orchestros have to pass form host > > > to host and cant reason about are evil, when allowed they prolifroate and > > > makes any idea of a vendor nutral abstraction and interoperablity between systems > > > impossible to reason about. that said there is a way to make it opaue but still useful > > > to userspace. see below > > > > > for a Intel vGPU, string format can be defined like > > > > > "parent device PCI ID" + "version of gvt driver" + "mdev type" + "aggregator count". > > > > > > > > > > for an NVMe VF connecting to a remote storage. it could be > > > > > "PCI ID" + "driver version" + "configured remote storage URL" > > > > > > > > > > for a QAT VF, it may be > > > > > "PCI ID" + "driver version" + "supported encryption set". > > > > > > > > > > (to avoid namespace confliction from each vendor, we may prefix a driver name to > > > > > each migration_version string. e.g. i915-v1-8086-591d-i915-GVTg_V5_8-1) > > > > > > honestly i would much prefer if the version string was just a semver string. > > > e.g. {major}.{minor}.{bugfix} > > > > > > if you do a driver/frimware update and break compatiablity with an older version bump the > > > major version. > > > > > > if you add optional a feature that does not break backwards compatiablity if you migrate > > > an older instance to the new host then just bump the minor/feature number. > > > > > > if you have a fix for a bug that does not change the feature set or compatiblity backwards or > > > forwards then bump the bugfix number > > > > > > then the check is as simple as > > > 1.) is the mdev type the same > > > 2.) is the major verion the same > > > 3.) am i going form the same version to same version or same version to newer version > > > > > > if all 3 are true we can migrate. > > > e.g. > > > 2.0.1 -> 2.1.1 (ok same major version and migrating from older feature release to newer feature release) > > > 2.1.1 -> 2.0.1 (not ok same major version and migrating from new feature release to old feature release may be > > > incompatable) > > > 2.0.0 -> 3.0.0 (not ok chaning major version) > > > 2.0.1 -> 2.0.0 (ok same major and minor version, all bugfixs in the same minor release should be compatibly) > > > > What's the value of the bugfix field in this scheme? > > its not require but really its for a non visable chagne form a feature standpoint. > a rather contrived example but if it was quadratic to inital a set of queues or device bufferes > in 1.0.0 and you made it liniar in 1.0.1 that is a performace improvment in the device intialisation time > which is great but it would not affect the feature set or compatiablity in any way. you could call it > a feature but its really just an internal change but you might want to still bump the version number. > > > > The simplicity is good, but is it too simple. It's not immediately > > clear to me whether all features can be hidden behind a minor version. > > For instance, if we have an mdev device that supports this notion of > > aggregation, which is proposed as a solution to the problem that > > physical hardware might support lots and lots of assignable interfaces > > which can be combined into arbitrary sets for mdev devices, making it > > impractical to expose an mdev type for every possible enumeration of > > assignable interfaces within a device. > > so this is a modeling problem and likely a limitation of the current way an mdev_type is exposed. > stealing some linux doc eamples > > > |- [parent physical device] > |--- Vendor-specific-attributes [optional] > |--- [mdev_supported_types] > | |--- [<type-id>] > | | |--- create > | | |--- name > | | |--- available_instances > | | |--- device_api > | | |--- description > > you could adress this in 1 of at least 3 ways. > 1.) mdev type for each enmartion which is fine for 1-2 variabley othersize its a combinitroial explotions. > 2.) report each of the consomable sub componetns as an mdev type and create mupltipel mdevs and assign them to the vm. > 3.) provider an api to dynamically compose mdevs types which staticaly partion the reqouese and can then be consomed > perferably embeding the resouce infomation in the description filed in a huma/machince readable form. > > 2 and 3 woudl work well with openstack however they both have there challanges > 1 doesnt really work for anyone out side of a demo. > > We therefore expose a base type > > where the aggregation is built later. This essentially puts us in a > > scenario where even within an mdev type running on the same driver, > > there are devices that are not directly compatible with each other. > > > > > we dont need vendor to rencode the driver name or vendor id and product id in the string. that info is alreay > > > available both to the device driver and to userspace via /sys already we just need to know if version of > > > the same mdev are compatiable so a simple semver version string which is well know in the software world > > > at least is a clean abstration we can reuse. > > > > This presumes there's no cross device migration. > > no but it does assume no cross mdev_type migration. > it assuems that nvida_mdev_type_x on host 1 is the same as nvida_mdev_type_x on host 2. > if the parent device differese but support the same mdev type we are asserting that they > should be compatiable or a differnt mdev_type name should be used on each device. > > so we are presuming the mdev type cant change as part of a live migration and if the type > was to change it would no longer be a live migration operation it would be something else. > that is based on the premis that changing the mdev type would change the capabilities of the mdev > > > An mdev type can only > > be migrated to the same mdev type, all of the devices within that type > > have some based compatibility, a phsyical device can only be migrated to > > the same physical device. In the latter case what defines the type? > > the type-id in /sysfs > > /sys/devices/virtual/mtty/mtty/ > |-- mdev_supported_types > | |-- mtty-1 <---- this is an mdev type > | | |-- available_instances > | | |-- create > | | |-- device_api > | | |-- devices > | | `-- name > | `-- mtty-2 <---- as is this > | |-- available_instances > | |-- create > | |-- device_api > | |-- devices > | `-- name > > |- [parent phy device] > |--- [$MDEV_UUID] > |--- remove > |--- mdev_type {link to its type} <-- here > |--- vendor-specific-attributes [optional] > > > If > > it's a PCI device, is it only vendor:device IDs? > > no the mdev type is not defined by the vendor:device id of the parent device > although the capablityes of that device will determin what mdev types if any it supprots. > > What about revision? > > What about subsystem IDs? > > at least for nvidia gpus i dont think if you by an evga branded v100 vs an pny branded one the capability > would change but i do know that certenly the capablities of a dell branding intel nic and an intel branded > one can. e.g. i have seen oem sku nics without sriov eventhoguh the same nic form intel supports it. > sriov was deliberatly disabled in the dell firmware even though it share dhte same vendor and prodcut id but differnt > subsystem id. > > if the odm made an incomatipable change like that which affect an mdev type in some way i guess i would expect them to > change the name or the description filed content to signal that. > > > What about possibly an onboard ROM or > > internal firmware? > > i would expect that updating the firmware/rom could result in changing a version string. that is how i was imagining > it would change. > > The information may be available, but which things > > are relevant to migration? > > that i dont know an i really would not like to encode that knolage in the vendor specific way in higher level > tools like openstack or even libvirt. declarative version sting comparisons or even simile feature flag > check where an abstract huristic that can be applied across vendors would be fine. but yes i dont know > what info would be needed in this case. > > We already see desires to allow migration > > between physical and mdev, > > migration between a phsical device and an mdev would not generally be considered a live migration in openstack. > that would be a different operation as it would be user visible withing the guest vm. > > but also to expose mdev types that might be > > composable to be compatible with other types. Thanks, > > i think composable mdev types are really challanging without some kind of feature flag concept > like cpu flags or ethtool nic capablities that are both human readable and easily parsable. > > we have the capability to schedule on cpu flags or gpu cuda level using a traits abstraction > so instead of saying i want an vm on a host with an intel 2695v3 to ensure it has AVX > you say i want an vm that is capable of using AVX > https://github.com/openstack/os-traits/blob/master/os_traits/hw/cpu/x86/__i… > > we also have trait for cuda level so instead of asking for a specifc mdev type or nvida > gpu the idea was you woudl describe what feature cuda in this exmple you need > https://github.com/openstack/os-traits/blob/master/os_traits/hw/gpu/cuda.py… > > That is what we call qualitative schudleing and is why we create teh placement service. > with out going in to the weeds we try to decouple quantaitive request such as 4 cpus and 1G of ram > form the qunative i need AVX supprot > > e.g. resouces:VCPU=4,resouces:MEMORY_MB=1024 triats:required=HW_CPU_X86_AVX > > declarative quantitive and capablites reporting of resouces fits easily into that model. > dynamic quantities that change as other mdev are allocated from the parent device or as > new mdevs types are composed on the fly are very challenging. > > > > > Alex > > > >

1 0

Import failures for networking vdsm modules in basic and he suite
by Marcin Sobczyk 13 Jul '20

13 Jul '20

Hi, I'm observing some issues with network imports failing. For basic suite supervdsmd fails to start with: l 13 17:24:11 lago-basic-suite-master-host-0 python3[29380]: detected unhandled Python exception in '/usr/share/vdsm/supervdsmd' Jul 13 17:24:11 lago-basic-suite-master-host-0 abrt-server[29382]: Not saving repeating crash in '/usr/share/vdsm/supervdsmd' Jul 13 17:24:11 lago-basic-suite-master-host-0 daemonAdapter[29380]: Traceback (most recent call last): Jul 13 17:24:11 lago-basic-suite-master-host-0 daemonAdapter[29380]: File "/usr/share/vdsm/supervdsmd", line 24, in <module> Jul 13 17:24:11 lago-basic-suite-master-host-0 daemonAdapter[29380]: from vdsm import supervdsm_server Jul 13 17:24:11 lago-basic-suite-master-host-0 daemonAdapter[29380]: File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 66, in <module> Jul 13 17:24:11 lago-basic-suite-master-host-0 daemonAdapter[29380]: from vdsm.network.initializer import init_privileged_network_components Jul 13 17:24:11 lago-basic-suite-master-host-0 daemonAdapter[29380]: File "/usr/lib/python3.6/site-packages/vdsm/network/initializer.py", line 32, in <module> Jul 13 17:24:11 lago-basic-suite-master-host-0 daemonAdapter[29380]: from vdsm.network import nmstate Jul 13 17:24:11 lago-basic-suite-master-host-0 daemonAdapter[29380]: ImportError: cannot import name 'nmstate' While deploying he basic suite I can see smth like (this one's a bit weird cause it's extracted from ansible logs, so might be trimmed): File \"/usr/libexec/vdsm/vm_libvirt_hook.py\", line 29, in <module> from vdsm.virt.vmdevices import storage File \"/usr/$ib/python3.6/site-packages/vdsm/virt/vmdevices/__init__.py\", line 27, in <module> from . import graphics File \"/usr/lib/python3.6/site-packages/vdsm/virt/vmdevices$graphics.py\", line 27, in <module> from vdsm.virt import displaynetwork File \"/usr/lib/python3.6/site-packages/vdsm/virt/displaynetwork.py\", line 23, in <module>\$ from vdsm.network import api as net_api File \"/usr/lib/python3.6/site-packages/vdsm/network/api.py\", line 34, in <module> from vdsm.network import netswitch File \"/usr/lib/python3.6/site-packages/vdsm/network/netswitch/__init__.py\", line 23, in <module> from . import configurator File \"/usr/lib/python3.6/site-packages$vdsm/ Ales, could you take a look? Regards, Marcin

2 2

Re: [ovirt-users] Using ovirt imageio
by Nir Soffer 13 Jul '20

13 Jul '20

On Tue, Jul 7, 2020 at 5:05 PM Łukasz Kołaciński <l.kolacinski(a)storware.eu> wrote: > Dear ovirt community, > Hi Łukasz, Adding devel(a)ovit.org since this topic is more appropriate for the devel list. > I am trying to use ovirt imageio api to receive changed blocks (dirty > bitmap) on ovirt 4.4. Could anyone tell me how to get them step by step? On > the documentation I saw endpoint "GET /images/ticket-uuid/map". I don't > know what ticket-uuid is and how to generate it. I also need to know how to > use this api because I can't reach it via /ovirt-engine/api/ > > I am asking about this endpoint: > > https://www.ovirt.org/documentation/incremental-backup-guide/incremental-ba… > This guide is outdated and should not be used now. The most up to date information is here: https://www.ovirt.org/develop/release-management/features/storage/increment… However the extents API is also outdated in the feature page. We are working on updating it. So here is example: First you must start backup with from_checkpoint_id argument: backup = backups_service.add( types.Backup( disks=disks, from_checkpoint_id="checkpoint-id", ) ) > "checkpoint-id" is the checkpoint created in the last backup. This starts a backup in in incremental mode. Dirty extents are available only in this mode. Then you start a transfer for download, using the backup id: transfer = imagetransfer.create_transfer( connection, disk, types.ImageTransferDirection.DOWNLOAD, backup=types.Backup(id=backup_uuid)) The transfer.transfer_url is the URL to download from, for example: https://host:54322/images/53787351-3f72-44a1-8a26-1323524fac4a Connect to host:54322 and send this request: GET /images/53787351-3f72-44a1-8a26-1323524fac4a/extents?context=dirty And parse the return json list, containing objects like: [ {"start": 0, "length": 65536, "dirty": true}, {"start": 65536, "length": 1048576, "dirty": false}, ... ] For example code of using the imageio API, see imageio http backend: https://github.com/oVirt/ovirt-imageio/blob/d5aa0e1fe659f1bf1247516f83c71e0… https://github.com/oVirt/ovirt-imageio/blob/d5aa0e1fe659f1bf1247516f83c71e0… We are adding a ImageioClient API that makes it easier to consume without writing any HTTP code: https://gerrit.ovirt.org/c/110068 With this you can use: with ImageioClient(transfer.transfer_url, cafile=args.cafile) as client: for extent in client.extent("dirty"): if extent.dirty: print("##dirty start={} length={}".format(extent.start, extent.length)) client.write_to(sys.stdout.buffer, extent.start, extent.length) print() This will stream the dirty extents to stdout. Not very useful as is, but illustrates how you can consume the data. Here is an example writing extents to a sparse stream format: https://gerrit.ovirt.org/c/110069 For complete backup example code see: https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/backup_v… Note the new imagetransfer helper module: https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/helpers/… Nir e-mail: l.kolacinski(a)storware.eu > <m.helbert(a)storware.eu> > > > > > *[image: STORWARE]* <http://www.storware.eu/> > > > > *ul. Leszno 8/44 01-192 Warszawa www.storware.eu > <https://www.storware.eu/>* > > *[image: facebook]* <https://www.facebook.com/storware> > > *[image: twitter]* <https://twitter.com/storware> > > *[image: linkedin]* <https://www.linkedin.com/company/storware> > > *[image: Storware_Stopka_09]* > <https://www.youtube.com/channel/UCKvLitYPyAplBctXibFWrkw> > > > > *Storware Spółka z o.o. nr wpisu do ewidencji KRS dla M.St. Warszawa > 000510131* *, NIP 5213672602.** Wiadomość ta jest przeznaczona jedynie > dla osoby lub podmiotu, który jest jej adresatem i może zawierać poufne > i/lub uprzywilejowane informacje. Zakazane jest jakiekolwiek przeglądanie, > przesyłanie, rozpowszechnianie lub inne wykorzystanie tych informacji lub > podjęcie jakichkolwiek działań odnośnie tych informacji przez osoby lub > podmioty inne niż zamierzony adresat. Jeżeli Państwo otrzymali przez > pomyłkę tę informację prosimy o poinformowanie o tym nadawcy i usunięcie > tej wiadomości z wszelkich komputerów. **This message is intended only > for the person or entity to which it is addressed and may contain > confidential and/or privileged material. Any review, retransmission, > dissemination or other use of, or taking of any action in reliance upon, > this information by persons or entities other than the intended recipient > is prohibited. If you have received this message in error, please contact > the sender and remove the material from all of your computer systems.* > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/AIJMMJOR354VK… >

3 4

VM export: task stucked
by francesco＠shellrent.com 10 Jul '20

10 Jul '20

Hi all, everyday at 01:00 AM we perform a daily backup on many VMs hosted in multiple hosts (all with oVirt 4.3) using a custom sciprt written in python3 (using SDK) and everything works "almost" fine. There is one single VM (Windows Server 2016) with a disk of 600 GB (the real disk usage is about 150 GB) hosted on a single node that has a strange behaviour. 1) The export start after a few seconds of the execution of the script, we use "vm_service.export_to_path_on_host" class for exporting the VM as ova file; 2) After few minutes i see in the engine the command "START, DumpXmlsVDSCommand" that is completely istantly both on the host side ad the engine side, and it's fine: 2020-07-02 01:05:45,428+0200 INFO (jsonrpc/7) [api.host] START dumpxmls(vmList=[u'10e88cab-ec4f-4491-b51f-94e3d2e81a0a']) from=::ffff:$ENGINE_IP,39308 (api:48) 2020-07-02 01:05:45,428+0200 INFO (jsonrpc/7) [api.host] FINISH dumpxmls return={... LONG XML ...} 3) after 3 hours i see the following logs about the export task: 2020-07-02 04:11:39,201+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-91) [895d6936-f45c-4766-afd4-408a4e4e9a41] START, GetVolumeInfoVDSCommand(HostName = $OVIRT_HOST, GetVolumeInfoVDSCommandParameters:{hostId='e8c07142-fcd8-4f78-9158-ffb2caa06dc5', storagePoolId='79d774b7-ca5b-49c2-baf8-9275ba3f1a84', storageDomainId='6775c41c-7d67-451b-8beb-4fd086eade2e', imageGroupId='a084fa36-0f93-45c2-a323-ea9ca2d16677', imageId='55b3eac5-05b2-4bae-be50-37cde7050697'}), log id: 3cbf2c7c 2020-07-02 04:11:39,299+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetQemuImageInfoVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-91) [895d6936-f45c-4766-afd4-408a4e4e9a41] START, GetQemuImageInfoVDSCommand(HostName = $OVIRT_HOST, GetVolumeInfoVDSCommandParameters:{hostId='e8c07142-fcd8-4f78-9158-ffb2caa06dc5', storagePoolId='79d774b7-ca5b-49c2-baf8-9275ba3f1a84', storageDomainId='6775c41c-7d67-451b-8beb-4fd086eade2e', imageGroupId='a084fa36-0f93-45c2-a323-ea9ca2d16677', imageId='55b3eac5-05b2-4bae-be50-37cde7050697'}), log id: 47d5122e 2020-07-02 04:43:21,339+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [312261bb] START, PrepareImageVDSCommand(HostName = $OVIRT_HOST, PrepareImageVDSCommandParameters:{hostId='e8c07142-fcd8-4f78-9158-ffb2caa06dc5'}), log id: 28c880d12020-07-02 04:43:21,650+02 INFO [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [312261bb] Executing Ansible command: ANSIBLE_STDOUT_CALLBACK=imagemeasureplugin /usr/bin/ansible-playbook --ssh-common-args=-F /var/lib/ovirt-engine/.ssh/config --private-key=/etc/pki/ovirt-engine/keys/engine_id_rsa --inventory=/tmp/ansible-inventory1121551177697847734 --extra-vars=image_path="/rhev/data-center/mnt/_data/6775c41c-7d67-451b-8beb-4fd086eade2e/images/a084fa36-0f93-45c2-a323-ea9ca2d16677/5b76cc6c-1a6b-4e02-8ce4-dc80faf9ed04" /usr/share/ovirt-engine/playbooks/ovirt-image-measure.yml [Logfile: /var/log/ovirt-engine/ova /ovirt-image-measure-ansible-20200702044321-$OVIRT_HOST-312261bb.log] 2020-07-02 05:05:20,428+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-8) [64c95c4d] START, UploadStreamVDSCommand(HostName = $OVIRT_HOST, UploadStreamVDSCommandParameters:{hostId='e8c07142-fcd8-4f78-9158-ffb2caa06dc5'}), log id: 666809cb 2020-07-02 05:05:22,104+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-8) [64c95c4d] START, UploadStreamVDSCommand(HostName = $OVIRT_HOST, UploadStreamVDSCommandParameters:{hostId='e8c07142-fcd8-4f78-9158-ffb2caa06dc5'}), log id: 33ff46bf 2020-07-02 05:05:29,602+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-287) [64c95c4d] START, HSMClearTaskVDSCommand(HostName = $OVIRT_HOST, HSMTaskGuidBaseVDSCommandParameters:{hostId='e8c07142-fcd8-4f78-9158-ffb2caa06dc5', taskId='e40a2740-37f6-455d-bed7-554efef761ff'}), log id: f01ff26 2020-07-02 05:05:29,619+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-269) [64c95c4d] START, HSMClearTaskVDSCommand(HostName = $OVIRT_HOST, HSMTaskGuidBaseVDSCommandParameters:{hostId='e8c07142-fcd8-4f78-9158-ffb2caa06dc5', taskId='1c94df38-88c1-4bf8-89a2-8c322513b21b'}), log id: 6150e13b 4) And then, after several hours, the export fail with the following logs: engine logs. 2020-07-02 12:43:21,653+02 ERROR [org.ovirt.engine.core.bll.CreateOvaCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [312261bb] Failed to measure image: null. Please check logs for more details: /var/log/ovirt-engine/ova/ovirt-image-measure-ansible-20200702044321-$OVIRT_HOST-312261bb.log 2020-07-02 12:43:23,741+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-28) [] EVENT_ID: IMPORTEXPORT_EXPORT_VM_TO_OVA_FAILED(1,225), Failed to export Vm $OVIRT_VM_TO_EXPORT as a Virtual Appliance to path /backup/$OVIRT_VM_TO_EXPORT_daily_2020Jul02.ova on Host $OVIRT_HOST 2020-07-02 12:43:26,162+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-9) [54b569e8] START, MergeVDSCommand(HostName = $OVIRT_HOST, MergeVDSCommandParameters:{hostId='e8c07142-fcd8-4f78-9158-ffb2caa06dc5', vmId='10e88cab-ec4f-4491-b51f-94e3d2e81a0a', storagePoolId='79d774b7-ca5b-49c2-baf8-9275ba3f1a84', storageDomainId='6775c41c-7d67-451b-8beb-4fd086eade2e', imageGroupId='a084fa36-0f93-45c2-a323-ea9ca2d16677', imageId='55b3eac5-05b2-4bae-be50-37cde7050697', baseImageId='5b76cc6c-1a6b-4e02-8ce4-dc80faf9ed04', topImageId='55b3eac5-05b2-4bae-be50-37cde7050697', bandwidth='0'}), log id: a0b417d 2020-07-02 12:43:26,328+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-commandCoordinator-Thread-9) [54b569e8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM $OVIRT_HOST command MergeVDS failed: Merge failed 2020-07-02 12:43:26,328+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-9) [54b569e8] HostName = $OVIRT_HOST 2020-07-02 12:43:26,328+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-9) [54b569e8] Command 'MergeVDSCommand(HostName = $OVIRT_HOST, MergeVDSCommandParameters:{hostId='e8c07142-fcd8-4f78-9158-ffb2caa06dc5', vmId='10e88cab-ec4f-4491-b51f-94e3d2e81a0a', storagePoolId='79d774b7-ca5b-49c2-baf8-9275ba3f1a84', storageDomainId='6775c41c-7d67-451b-8beb-4fd086eade2e', imageGroupId='a084fa36-0f93-45c2-a323-ea9ca2d16677', imageId='55b3eac5-05b2-4bae-be50-37cde7050697', baseImageId='5b76cc6c-1a6b-4e02-8ce4-dc80faf9ed04', topImageId='55b3eac5-05b2-4bae-be50-37cde7050697', bandwidth='0'})' execution failed: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 2020-07-02 12:43:28,996+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DumpXmlsVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [54b569e8] START, DumpXmlsVDSCommand(HostName = $OVIRT_HOST, Params:{hostId='e8c07142-fcd8-4f78-9158-ffb2caa06dc5', vmIds='[10e88cab-ec4f-4491-b51f-94e3d2e81a0a]'}), log id: 65709a96 2020-07-02 13:05:38,206+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-45) [5179fcb] START, UploadStreamVDSCommand(HostName = $OVIRT_HOST, UploadStreamVDSCommandParameters:{hostId='e8c07142-fcd8-4f78-9158-ffb2caa06dc5'}), log id: 44e9ec23 2020-07-02 13:05:39,851+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-45) [5179fcb] START, UploadStreamVDSCommand(HostName = $OVIRT_HOST, UploadStreamVDSCommandParameters:{hostId='e8c07142-fcd8-4f78-9158-ffb2caa06dc5'}), log id: 68231c70 2020-07-02 13:05:40,356+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-240) [5179fcb] START, HSMClearTaskVDSCommand(HostName = $OVIRT_HOST, HSMTaskGuidBaseVDSCommandParameters:{hostId='e8c07142-fcd8-4f78-9158-ffb2caa06dc5', taskId='5d61682b-cf47-4637-9014-c647eb5265ee'}), log id: 718885f3 2020-07-02 13:05:50,357+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-290) [5179fcb] START, HSMClearTaskVDSCommand(HostName = $OVIRT_HOST, HSMTaskGuidBaseVDSCommandParameters:{hostId='e8c07142-fcd8-4f78-9158-ffb2caa06dc5', taskId='5709eb6c-a656-4a4d-b67f-72dd0bebf3e4'}), log id: 40c18d57 And here the vdsm.log of the host: 2020-07-02 12:43:26,169+0200 INFO (jsonrpc/6) [api.virt] START merge(drive={u'imageID': u'a084fa36-0f93-45c2-a323-ea9ca2d16677', u'volumeID': u'55b3eac5-05b2-4bae-be50-37cde7050697', u'domainID': u'6775c41c-7d67-451b-8beb-4fd086eade2e', u'poolID': u'79d774b7-ca5b-49c2-baf8-9275ba3f1a84'}, baseVolUUID=u'5b76cc6c-1a6b-4e02-8ce4-dc80faf9ed04', topVolUUID=u'55b3eac5-05b2-4bae-be50-37cde7050697', bandwidth=u'0', jobUUID=u'b49fa67b-4a5a-4f73-823f-9e33a8e20cd2') from=::ffff:$ENGINE_IP,39308, flow_id=54b569e8, vmId=10e88cab-ec4f-4491-b51f-94e3d2e81a0a (api:48) 2020-07-02 12:43:26,196+0200 INFO (jsonrpc/6) [vdsm.api] START getVolumeInfo(sdUUID=u'6775c41c-7d67-451b-8beb-4fd086eade2e', spUUID='79d774b7-ca5b-49c2-baf8-9275ba3f1a84', imgUUID=u'a084fa36-0f93-45c2-a323-ea9ca2d16677', volUUID=u'5b76cc6c-1a6b-4e02-8ce4-dc80faf9ed04', options=None) from=::ffff:$ENGINE_IP,39308, flow_id=54b569e8, task_id=bb1d1fd0-9746-4949-9c6c-b069947f5875 (api:48) 2020-07-02 12:43:26,199+0200 INFO (jsonrpc/6) [storage.VolumeManifest] Info request: sdUUID=6775c41c-7d67-451b-8beb-4fd086eade2e imgUUID=a084fa36-0f93-45c2-a323-ea9ca2d16677 volUUID = 5b76cc6c-1a6b-4e02-8ce4-dc80faf9ed04 (volume:240) 2020-07-02 12:43:26,205+0200 INFO (jsonrpc/6) [storage.VolumeManifest] 6775c41c-7d67-451b-8beb-4fd086eade2e/a084fa36-0f93-45c2-a323-ea9ca2d16677/5b76cc6c-1a6b-4e02-8ce4-dc80faf9ed04 info is {'status': 'OK', 'domain': '6775c41c-7d67-451b-8beb-4fd086eade2e', 'voltype': 'INTERNAL', 'description': '', 'parent': '7e802d0d-d6cd-4979-8f8d-5f4c5d5c3013', 'format': 'COW', 'generation': 0, 'image': 'a084fa36-0f93-45c2-a323-ea9ca2d16677', 'disktype': 'DATA', 'legality': 'LEGAL', 'mtime': '0', 'apparentsize': '6635716608', 'children': [], 'pool': '', 'ctime': '1593558283', 'capacity': '644245094400', 'uuid': u'5b76cc6c-1a6b-4e02-8ce4-dc80faf9ed04', 'truesize': '6637277184', 'type': 'SPARSE'} (volume:279) 2020-07-02 12:43:26,205+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getVolumeInfo return={'info': {'status': 'OK', 'domain': '6775c41c-7d67-451b-8beb-4fd086eade2e', 'voltype': 'INTERNAL', 'description': '', 'parent': '7e802d0d-d6cd-4979-8f8d-5f4c5d5c3013', 'format': 'COW', 'generation': 0, 'image': 'a084fa36-0f93-45c2-a323-ea9ca2d16677', 'disktype': 'DATA', 'legality': 'LEGAL', 'mtime': '0', 'apparentsize': '6635716608','children': [], 'pool': '', 'ctime': '1593558283', 'capacity': '644245094400', 'uuid': u'5b76cc6c-1a6b-4e02-8ce4-dc80faf9ed04', 'truesize': '6637277184', 'type': 'SPARSE'}} from=::ffff:$ENGINE_IP,39308, flow_id=54b569e8, task_id=bb1d1fd0-9746-4949-9c6c-b069947f5875 (api:54) 2020-07-02 12:43:26,206+0200 INFO (jsonrpc/6) [vdsm.api] START getVolumeInfo(sdUUID=u'6775c41c-7d67-451b-8beb-4fd086eade2e', spUUID='79d774b7-ca5b-49c2-baf8-9275ba3f1a84', imgUUID=u'a084fa36-0f93-45c2-a323-ea9ca2d16677', volUUID=u'55b3eac5-05b2-4bae-be50-37cde7050697', options=None) from=::ffff:$ENGINE_IP,39308, flow_id=54b569e8, task_id=32ce9b21-b9f3-4319-a229-b32df1fb9283 (api:48) 2020-07-02 12:43:26,208+0200 INFO (jsonrpc/6) [storage.VolumeManifest] Info request: sdUUID=6775c41c-7d67-451b-8beb-4fd086eade2e imgUUID=a084fa36-0f93-45c2-a323-ea9ca2d16677 volUUID = 55b3eac5-05b2-4bae-be50-37cde7050697 (volume:240) 2020-07-02 12:43:26,219+0200 INFO (jsonrpc/6) [storage.VolumeManifest] 6775c41c-7d67-451b-8beb-4fd086eade2e/a084fa36-0f93-45c2-a323-ea9ca2d16677/55b3eac5-05b2-4bae-be50-37cde7050697 info is {'status': 'OK', 'domain': '6775c41c-7d67-451b-8beb-4fd086eade2e', 'voltype': 'LEAF', 'description': '', 'parent': '5b76cc6c-1a6b-4e02-8ce4-dc80faf9ed04', 'format': 'COW', 'generation': 0, 'image': 'a084fa36-0f93-45c2-a323-ea9ca2d16677', 'disktype': 'DATA', 'legality': 'LEGAL', 'mtime': '0', 'apparentsize': '4077387776', 'children': [], 'pool': '', 'ctime': '1593644682', 'capacity': '644245094400', 'uuid': u'55b3eac5-05b2-4bae-be50-37cde7050697', 'truesize': '4077940736', 'type': 'SPARSE'} (volume:279) 2020-07-02 12:43:26,219+0200 INFO (jsonrpc/6) [vdsm.api] FINISH getVolumeInfo return={'info': {'status': 'OK', 'domain': '6775c41c-7d67-451b-8beb-4fd086eade2e', 'voltype': 'LEAF', 'description': '', 'parent': '5b76cc6c-1a6b-4e02-8ce4-dc80faf9ed04', 'format': 'COW', 'generation': 0, 'image': 'a084fa36-0f93-45c2-a323-ea9ca2d16677', 'disktype': 'DATA', 'legality': 'LEGAL', 'mtime': '0', 'apparentsize': '4077387776', 'children': [], 'pool': '', 'ctime': '1593644682', 'capacity': '644245094400', 'uuid': u'55b3eac5-05b2-4bae-be50-37cde7050697', 'truesize': '4077940736', 'type': 'SPARSE'}} from=::ffff:$ENGINE_IP,39308, flow_id=54b569e8, task_id=32ce9b21-b9f3-4319-a229-b32df1fb9283 (api:54) 2020-07-02 12:43:26,235+0200 INFO (jsonrpc/6) [virt.vm] (vmId='10e88cab-ec4f-4491-b51f-94e3d2e81a0a') Starting merge with jobUUID=u'b49fa67b-4a5a-4f73-823f-9e33a8e20cd2', original chain=e2bad960-6ad4-44b6-8ff0-69dcca6bf722 < 7e802d0d-d6cd-4979-8f8d-5f4c5d5c3013 < 5b76cc6c-1a6b-4e02-8ce4-dc80faf9ed04 < 55b3eac5-05b2-4bae-be50-37cde7050697 (top), disk='sda', base='sda[1]', top=None, bandwidth=0, flags=12 (vm:5951) 2020-07-02 12:43:26,280+0200 ERROR (jsonrpc/6) [virt.vm] (vmId='10e88cab-ec4f-4491-b51f-94e3d2e81a0a') Live merge failed (job: b49fa67b-4a5a-4f73-823f-9e33a8e20cd2) (vm:5957) 2020-07-02 12:43:26,296+0200 INFO (jsonrpc/6) [api.virt] FINISH merge return={'status': {'message': 'Merge failed', 'code': 52}} from=::ffff:$ENGINE_IP,39308, flow_id=54b569e8, vmId=10e88cab-ec4f-4491-b51f-94e3d2e81a0a (api:54) 2020-07-02 12:43:26,296+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.merge failed (error 52) in 0.13 seconds (__init__:312) 2020-07-02 12:43:26,627+0200 INFO (jsonrpc/1) [vdsm.api] START getSpmStatus(spUUID=u'79d774b7-ca5b-49c2-baf8-9275ba3f1a84', options=None) from=::ffff:$ENGINE_IP,39308, flow_id=4968c8cb, task_id=ee82b2d6-39f5-43ad-83a3-803f1af8e234 (api:48) 2020-07-02 12:43:26,628+0200 INFO (jsonrpc/1) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': -1}} from=::ffff:$ENGINE_IP,39308, flow_id=4968c8cb, task_id=ee82b2d6-39f5-43ad-83a3-803f1af8e234 (api:54) 2020-07-02 12:43:26,628+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StoragePool.getSpmStatus succeeded in 0.00 seconds (__init__:312) 2020-07-02 12:43:26,645+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID=u'79d774b7-ca5b-49c2-baf8-9275ba3f1a84', options=None) from=::ffff:$ENGINE_IP,39304, flow_id=4968c8cb, task_id=ab1124bc-651d-4ade-aae4-ff9fea23f6a0 (api:48) 2020-07-02 12:43:26,646+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'name': 'No Description', 'isoprefix': '', 'pool_status': 'connected', 'lver': -1, 'domains': u'6775c41c-7d67-451b-8beb-4fd086eade2e:Active', 'master_uuid': u'6775c41c-7d67-451b-8beb-4fd086eade2e', 'version': '5', 'spm_id': 1, 'type': 'LOCALFS', 'master_ver': 1}, 'dominfo': {u'6775c41c-7d67-451b-8beb-4fd086eade2e': {'status': u'Active', 'diskfree': '3563234983936', 'isoprefix': '', 'alerts': [], 'disktotal': '3923683311616', 'version': 5}}} from=::ffff:$ENGINE_IP,39304, flow_id=4968c8cb, task_id=ab1124bc-651d-4ade-aae4-ff9fea23f6a0 (api:54) 2020-07-02 12:43:26,647+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.getInfo succeeded in 0.01 seconds (__init__:312) 5) His task seems block any other export tasks, leave them in pending. There were about 5 export tasks and 1 import temaplte task and as soon we killed the following process, the other 4 tasks and the import one completed after few minutes. 2020-07-02 04:43:21,339+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [312261bb] START, PrepareImageVDSCommand(HostName = $OVIRT_HOST, PrepareImageVDSCommandParameters:{hostId='e8c07142-fcd8-4f78-9158-ffb2caa06dc5'}), log id: 28c880d12020-07-02 04:43:21,650+02 INFO [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [312261bb] Executing Ansible command: ANSIBLE_STDOUT_CALLBACK=imagemeasureplugin /usr/bin/ansible-playbook --ssh-common-args=-F /var/lib/ovirt-engine/.ssh/config --private-key=/etc/pki/ovirt-engine/keys/engine_id_rsa --inventory=/tmp/ansible-inventory1121551177697847734 --extra-vars=image_path="/rhev/data-center/mnt/_data/6775c41c-7d67-451b-8beb-4fd086eade2e/images/a084fa36-0f93-45c2-a323-ea9ca2d16677/5b76cc6c-1a6b-4e02-8ce4-dc80faf9ed04" /usr/share/ovirt-engine/playbooks/ovirt-image-measure.yml [Logfile: /var/log/ovirt-engine/ova /ovirt-image-measure-ansible-20200702044321-$OVIRT_HOST-312261bb.log] The process that we killed was: vdsm 25338 25332 99 04:14 pts/0 07:40:09 qemu-img measure -O qcow2 /rhev/data-center/mnt/_data/6775c41c-7d67-451b-8beb-4fd086eade2e/images/a084fa36-0f93-45c2-a323-ea9ca2d16677/55b3eac5-05b2-4bae-be50-37cde705069 A strace -p of the PID 25338 showed following lines: lseek(11, 3056795648, SEEK_DATA) = 3056795648 lseek(11, 3056795648, SEEK_HOLE) = 13407092736 lseek(14, 128637468672, SEEK_DATA) = 128637468672 lseek(14, 128637468672, SEEK_HOLE) = 317708828672 lseek(14, 128646250496, SEEK_DATA) = 128646250496 lseek(14, 128646250496, SEEK_HOLE) = 317708828672 lseek(14, 128637730816, SEEK_DATA) = 128637730816 lseek(14, 128637730816, SEEK_HOLE) = 317708828672 lseek(14, 128646774784, SEEK_DATA) = 128646774784 lseek(14, 128646774784, SEEK_HOLE) = 317708828672 lseek(14, 128646709248, SEEK_DATA) = 128646709248 Can yu help us figuring out what is going on? If any other informations is needed I'll provide 'em ASAP. Thank you for your patience and your time, Regards Francesco

3 2

oVirt vdsm Bulk-API deep dive
by Nir Soffer 09 Jul '20

09 Jul '20

I want to share an interesting talk on the new StorageDomain dump API. When: Thursday, July 9⋅16:00 – 16:45 (Israel time) Where: https://redhat.bluejeans.com/212781836 Who: Amit Bawer StorageDomain dump API is a faster way to query oVirt storage domain contents, optimized for bulk operations such as collecting sos reports or importing storage domains during disaster recovery. In recent tests we found that it is 295 times faster compared with vdsm-tool dump-volume-chains: https://bugzilla.redhat.com/show_bug.cgi?id=1557147#c26 What can you do with this API? Query volumes metadata by disk id: # vdsm-client StorageDomain dump sd_id=56ecc03c-4bb5-4792-8971-3c51ea924d2e | jq '.volumes | .[] | select(.image=="d7ead22a-0fbf-475c-a62f-6f7bc8473acb")' { "apparentsize": 6442450944, "capacity": 6442450944, "ctime": 1593128884, "description": "{\"DiskAlias\":\"fedora-32.raw\",\"DiskDescription\":\"Uploaded disk\"}", "disktype": "DATA", "format": "RAW", "generation": 0, "image": "d7ead22a-0fbf-475c-a62f-6f7bc8473acb", "legality": "LEGAL", "parent": "00000000-0000-0000-0000-000000000000", "status": "OK", "truesize": 6442455040, "type": "PREALLOCATED", "voltype": "LEAF" } Find all templates: # vdsm-client StorageDomain dump sd_id=56ecc03c-4bb5-4792-8971-3c51ea924d2e | jq '.volumes | .[] | select(.voltype=="SHARED")' { "apparentsize": 6442450944, "capacity": 6442450944, "ctime": 1593116563, "description": "{\"DiskAlias\":\"fedora-32.raw\",\"DiskDescription\":\"Uploaded disk\"}", "disktype": "DATA", "format": "RAW", "generation": 0, "image": "9b62b5fa-920e-4d0c-baf6-40406106e48e", "legality": "LEGAL", "parent": "00000000-0000-0000-0000-000000000000", "status": "OK", "truesize": 6442516480, "type": "PREALLOCATED", "voltype": "SHARED" } Please join the talk if you want to learn more. Cheers, Nir

2 2

ovirt-engine has been tagged (ovirt-engine-4.3.11.2)
by Tal Nisan 09 Jul '20

09 Jul '20

1 0

Re: [oVirt Jenkins] ovirt-system-tests_he-basic-suite-master - Build # 1655 - Still Failing!
by Yedidyah Bar David 07 Jul '20

07 Jul '20

On Sun, Jun 28, 2020 at 6:23 AM <jenkins(a)jenkins.phx.ovirt.org> wrote: > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1655/ This fails for some time now. Checked last one ^^, and it failed in: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1655… 2020-06-27 23:05:02,253-0400 INFO ansible task start {'status': 'OK', 'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', 'ansible_task': 'ovirt.hosted_engine_setup : Check OVF_STORE volume status'} 2020-06-27 23:05:02,253-0400 DEBUG ansible on_any args TASK: ovirt.hosted_engine_setup : Check OVF_STORE volume status kwargs is_conditional:False 2020-06-27 23:05:02,254-0400 DEBUG ansible on_any args localhostTASK: ovirt.hosted_engine_setup : Check OVF_STORE volume status kwargs 2020-06-27 23:05:03,816-0400 DEBUG ansible on_any args <ansible.executor.task_result.TaskResult object at 0x7f454e3d1eb8> kwargs ... 2020-06-27 23:09:39,166-0400 DEBUG var changed: host "localhost" var "ovf_store_status" type "<class 'dict'>" value: "{ "changed": true, "failed": true, "msg": "All items completed", "results": [ { "ansible_loop_var": "item", "attempts": 12, Meaning, it ran 12 times the command: vdsm-client Volume getInfo storagepoolID=41c9fdea-b8e9-11ea-ae2a-5452c0a8c863 storagedomainID=10a69775-8fb6-437d-9e78-2ecfd77c0a45 imageID=c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9 volumeID=6b835f55-a512-4f83-9d25-f6837d8b5cb1 and never got a result with output including "Updated". Any idea? Perhaps it's just a bug in the test, and we should test something else other than "Updated"? Or perhaps there was some other problem but I failed to find it? Thanks, > Build Number: 1655 > Build Status: Still Failing > Triggered By: Started by timer > > ------------------------------------- > Changes Since Last Success: > ------------------------------------- > Changes for Build #1643 > [Galit] Add missing repo which has collectd-write_syslog > > [Sandro Bonazzola] Revert "ovirt-release: run only on fc29 nodes" > > > Changes for Build #1644 > [Michal Skrivanek] make GLANCE failures fatal > > > Changes for Build #1645 > [Michal Skrivanek] use // in glance URL > > > Changes for Build #1646 > [Galit] Fix: No module named ost_utils.memoized > > > Changes for Build #1647 > [Martin Necas] ansible: fix deploy scripts to el8 > > [arachmani] Add arachman(a)redhat.com to jenkins recipient lists > > > Changes for Build #1648 > [Martin Necas] ansible: fix deploy scripts to el8 > > [Evgheni Dereveanchin] Do not run jenkins check-patch job on fc29 > > > Changes for Build #1649 > [Martin Necas] ansible: fix deploy scripts to el8 > > > Changes for Build #1650 > [Martin Necas] ansible: fix deploy scripts to el8 > > > Changes for Build #1651 > [Galit] Add centos8.2 official to templates > > > Changes for Build #1652 > [Evgeny Slutsky] he-basic-suite-master: Add Domain_name to the host when vm_run > > [Gal Ben Haim] Add gcc to fcraw > > [Sandro Bonazzola] pipelines: add ovirt-dependencies > > > Changes for Build #1653 > [Evgeny Slutsky] he-basic-suite-master: Add Domain_name to the host when vm_run > > > Changes for Build #1654 > [Galit] Add centos 8.2 image: el8.2-base > > [Evgheni Dereveanchin] do not expand values in run_oc_playbook > > > Changes for Build #1655 > [Galit] Add centos 8.2 image: el8.2-base > > > > > ----------------- > Failed Tests: > ----------------- > No tests ran. -- Didi

3 5

sic transit gloria mundi
by Dan Kenigsberg 06 Jul '20

06 Jul '20

Seven years ago, in June 2013, Giuseppe Valarelli introduced tests/functional/networkTests.py in https://gerrit.ovirt.org/#/c/14840/. It held high aspirations: to provide a comprehensive functional test for Vdsm's network API, in order to ease development of new code and reduce chances of reintroducing old bugs. 390 commits later, Andrej Cernak removed the last lines of code from that file in https://gerrit.ovirt.org/#/c/109182/. In between, many many other developers contributed to the set of tests, and later dissipated its content to multiple pytest-driven test modules. tests/functional/networkTests.py was never an emblem of great software design; it makes me happy to see it gone. But it was very useful and its spirit lives on: API tests are now the rule of Vdsm development, not an exception. I'd like to thank all the people who made this possible. $ git log 1c5f15c9b~ -- tests/functional/networkTests.py|grep Author|sort|uniq -c|sort -nr 72 Author: Edward Haas <edwardh(a)redhat.com> 58 Author: Ido Barkan <ibarkan(a)redhat.com> 46 Author: Ondřej Svoboda <osvoboda(a)redhat.com> 40 Author: Petr Horáček <phoracek(a)redhat.com> 35 Author: Dan Kenigsberg <danken(a)redhat.com> 34 Author: Antoni S. Puimedon <asegurap(a)redhat.com> 24 Author: Bell Levin <blevin(a)redhat.com> 23 Author: Leon Goldberg <lgoldber(a)redhat.com> 18 Author: Giuseppe Vallarelli <gvallare(a)redhat.com> 8 Author: Assaf Muller <amuller(a)redhat.com> 5 Author: Sveta Haas <sveta.haas(a)gmail.com> 4 Author: Mark Wu <wudxw(a)linux.vnet.ibm.com> 3 Author: Nir Soffer <nsoffer(a)redhat.com> 2 Author: Yaniv Bronhaim <ybronhei(a)redhat.com> 2 Author: Sagi Shnaidman <sshnaidm(a)redhat.com> 2 Author: Petr Sebek <psebek(a)redhat.com> 2 Author: Miguel Angel Ajo <miguelangel(a)ajo.es> 2 Author: Andrej Cernek <acernek(a)redhat.com> 1 Author: Viliam Serecun <v.serecun(a)gmail.com> 1 Author: Vered Volansky <vvolansk(a)redhat.com> 1 Author: Timothy Asir Jeyasingh <tjeyasin(a)redhat.com> 1 Author: shlad <a.shlado(a)gmail.com> 1 Author: pkliczewski <piotr.kliczewski(a)gmail.com> 1 Author: Petr Benas <pbenas(a)redhat.com> 1 Author: Marcin Sobczyk <msobczyk(a)redhat.com> 1 Author: Marcin Mirecki <mmirecki(a)redhat.com> 1 Author: Ilia Meerovich <iliam(a)redhat.com> 1 Author: Genadi Chereshnya <gcheresh(a)redhat.com> 1 Author: Dominik Holler <dholler(a)redhat.com>

2 1

ovirt-engine has been tagged (ovirt-engine-4.4.1.7)
by Sandro Bonazzola 03 Jul '20

03 Jul '20

1 0

Backup: how to download only used extents from imageio backend
by Michael Ablassmeier 02 Jul '20

02 Jul '20

hi, im currently looking at the new incremental backup api that has been part of the 4.4 and RHV 4.4-beta release. So far i was able to create full/incremental backups and restore without any problem. Now, using the backup_vm.py example from the ovirt-engine-sdk i get the following is happening during a full backup: 1) imageio client api requests transfer 2) starts qemu-img to create a local qemu image with same size 3) starts qemu-nbd to serve this image 4) reads used extents from provided imageio source, passes data to qemu-nbd process 5) resulting file is a thin provisioned qcow image with the actual data of the VM's used space. while this works great, it has one downside: if i backup a virtual machine with lots of used extents, or multiple virtual machines at the same time, i may run out of space, if my primary backup target is not a regular disk. Imagine i want to stream the FULL backup to tape directly like backup_vm.py full [..] <vm_uuid> /dev/nst0 thats currently not possible, because qemu-img is not able to open a tape device directly, given its nature of the qcow2 format. So what iam basically looking for, is a way to download only the extents from the imageio server that are really in use, not depending on qemu-* tools, to be able to pipe the data somehwere else. Standard tools, like for example curl, will allways download the full provisioned image from the imageio backend (of course). I noticed is that it is possible to query the extents via: https://tranfer_node:54322/images/d471c659-889f-4e7f-b55a-a475649c48a6/exte… As i failed to find them, are there any existing functions/api calls that could be used to download only the used extents to a file/fifo pipe? So far, i played around with the _internal.io.copy function, beeing able to at least read the data into a in memory BytesIO stream, but thats not the solution to my "problem" :) bye, - michael

3 7

ovirt-engine has been tagged (ovirt-engine-4.4.1.6)
by Tal Nisan 02 Jul '20

02 Jul '20

1 0

Issue while installing ovirt-imageio-daemon
by Sandro Bonazzola 02 Jul '20

02 Jul '20

Hi, just noticed in appliance build at https://jenkins.ovirt.org/job/ovirt-appliance_standard-check-patch/286/cons… 15:18:22,373 INFO anaconda:program: Installing : ovirt-imageio-daemon-2.0.9-1.el8.x86_64 90/223 <https://jenkins.ovirt.org/job/ovirt-appliance_standard-check-patch/286/cons…> 15:18:22,373 INFO anaconda:program: warning: group Development/Debug does not exist - using root <https://jenkins.ovirt.org/job/ovirt-appliance_standard-check-patch/286/cons…> 15:18:22,373 INFO anaconda:program: <https://jenkins.ovirt.org/job/ovirt-appliance_standard-check-patch/286/cons…> 15:18:22,374 INFO anaconda:program: Running scriptlet: ovirt-imageio-daemon-2.0.9-1.el8.x86_64 90/223 Sounds like a bug in the spec file. -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>*

3 3

Re: [ovirt-users] Re: ETL service aggregation error
by Shirly Radco 02 Jul '20

02 Jul '20

Hi, I was unable to view your log file. But if its indeed an the issue please run: systemctl stop ovirt-engine-dwhd ovirt_engine_history=# UPDATE history_configuration set var_datetime = date_trunc('hour', now())- interval '2 hour' WHERE var_name = 'lastHourAggr'; ovirt_engine_history=# UPDATE history_configuration set var_datetime = date_trunc('hour', now())- interval '1 day' WHERE var_name = 'lastDayAggr'; systemctl restart ovirt-engine-dwhd If you would like me to review the errors please attach the dwh log file. Best regards, -- Shirly Radco BI Principal Software Engineer Red Hat <https://www.redhat.com/> <https://www.redhat.com/> On Fri, Jun 26, 2020 at 10:28 PM Staniforth, Paul < P.Staniforth(a)leedsbeckett.ac.uk> wrote: > Hello Ayansh, > It looks like the lastHourAgg is wrong is should > be 2 hours before the runTime but is 1:30 hours before runtime. > > Try > > systemctl stop ovirt-engine-dwhd > > in ovirt_engine_history database > > UPDATE history_configuration set var_datetime = var_datetime - interval '24 hour' WHERE var_name = 'lastHourAggr'; > > UPDATE history_configuration set var_datetime = var_datetime - interval '1 day' WHERE var_name = 'lastDayAggr'; > > systemctl stop ovirt-engine-dwhd > > > see https://www.mail-archive.com/users@ovirt.org/msg53055.html > > > Regards, > > Paul Staniforth > > School of Built Environment Engineering and Computing. > > Leeds Beckett University > > Networked Systems Analyst, Research and Engineering Support. > Technical Lead Architect IMS and oVirt Virtualization > > tel: +44 (0)113 28123754 > email: p.staniforth(a)leedsbeckett.ac.uk > > <https://www.gofundme.com/f/bytes-for-heroes> > > ------------------------------ > *From:* Ayansh Rocks <shashank123rastogi(a)gmail.com> > *Sent:* 26 June 2020 17:10 > *To:* Staniforth, Paul <P.Staniforth(a)leedsbeckett.ac.uk>; users < > users(a)ovirt.org>; devel(a)ovirt.org <devel(a)ovirt.org> > *Subject:* Re: [ovirt-users] Re: ETL service aggregation error > > > *Caution External Mail:* Do not click any links or open any attachments > unless you trust the sender and know that the content is safe. > Can we fix this any how guys. > > On Thu, Jun 25, 2020 at 5:53 PM Ayansh Rocks <shashank123rastogi(a)gmail.com> > wrote: > > Is it a bug ? > > > On Thu, Jun 25, 2020 at 4:11 PM Ayansh Rocks <shashank123rastogi(a)gmail.com> > wrote: > > See below as well if it helps - > > [root@iondelvm149 ~]# date > Thu Jun 25 16:09:45 IST 2020 > [root@iondelvm149 ~]# psql -h localhost -U ovirt_engine_history -d > ovirt_engine_history > Password for user ovirt_engine_history: > psql (9.2.24, server 10.6) > WARNING: psql version 9.2, server version 10.0. > Some psql features might not work. > Type "help" for help. > > ovirt_engine_history=> select * from history_configuration; > var_name | var_value | var_datetime > -------------------+-----------+------------------------ > default_language | en_US | > firstSync | false | 2018-03-23 12:27:00-04 > lastHourAggr | | 2020-06-25 05:00:00-04 > HourlyAggFailed | false | > lastDayAggr | | 2020-06-24 01:00:00-04 > MinimalETLVersion | 4.3.0 | > (6 rows) > > ovirt_engine_history=> > > On Thu, Jun 25, 2020 at 4:06 PM Staniforth, Paul < > P.Staniforth(a)leedsbeckett.ac.uk> wrote: > > Hello Shashank, > is it on the hour? it sounds like it got a > problem aggregating the samples to create the hourly samples. > > I had to update the java memory usage as it was running out of heap memory. > > The defaults are > > DWH_HEAP_MIN=1g > DWH_HEAP_MAX=1g > > see > > /usr/share/ovirt-engine-dwh/services/ovirt-engine-dwhd/ovirt-engine-dwhd.conf > > for settings. > > Regards, > Paul S. > > > ------------------------------ > *From:* Ayansh Rocks <shashank123rastogi(a)gmail.com> > *Sent:* 25 June 2020 10:40 > *To:* Staniforth, Paul <P.Staniforth(a)leedsbeckett.ac.uk>; users < > users(a)ovirt.org> > *Subject:* Re: [ovirt-users] Re: ETL service aggregation error > > > *Caution External Mail:* Do not click any links or open any attachments > unless you trust the sender and know that the content is safe. > Hi Paul, > > Yes i am able to connect to the database from ovirt engine machine. > > port is already open. what could be issue, i am getting this error in > every hour. > > Thanks > > On Wed, Jun 24, 2020 at 6:30 PM Staniforth, Paul < > P.Staniforth(a)leedsbeckett.ac.uk> wrote: > > Hello Shashank, > it looks like it's had a problem for over 2 > years, is the Data Warehouse database local or remote? is there a firewall > port open? > > Can you connect to the database from the engine machine? > > The credentials should be in > /etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/10-setup-database.conf > > as your using 4.3 the version of postgresql is 10 and using scl > > scl enable rh-postgresql10 "psql -h localhost -U ovirt_engine_history > -d ovirt_engine_history" > > if it's using the standard user, database and is local. > > > Regards, > Paul S. > . > > ------------------------------ > *From:* Ayansh Rocks <shashank123rastogi(a)gmail.com> > *Sent:* 24 June 2020 12:47 > *To:* Staniforth, Paul <P.Staniforth(a)leedsbeckett.ac.uk>; users < > users(a)ovirt.org> > *Subject:* Re: [ovirt-users] Re: ETL service aggregation error > > > *Caution External Mail:* Do not click any links or open any attachments > unless you trust the sender and know that the content is safe. > Please find the attached error logs. > > On Thu, Jun 4, 2020 at 8:17 PM Staniforth, Paul < > P.Staniforth(a)leedsbeckett.ac.uk> wrote: > > Hello Shashank, > I can't see any of your images and also it > would be better to have the log file as text. > > Regards, > Paul S. > > > > ------------------------------ > *From:* Ayansh Rocks <shashank123rastogi(a)gmail.com> > *Sent:* 04 June 2020 15:16 > *To:* users <users(a)ovirt.org> > *Subject:* [ovirt-users] Re: ETL service aggregation error > > > *Caution External Mail:* Do not click any links or open any attachments > unless you trust the sender and know that the content is safe. > Any update on this ? > > On Tue, May 26, 2020 at 1:41 PM Ayansh Rocks <shashank123rastogi(a)gmail.com> > wrote: > > Hi, > > I am using 4.3.7 self hosted engine. From Few days i am getting regular > below error messages :- > [image: image.png] > > Logs in /var/log/ovirt-engine-dwh/ovirt-engine-dwhd.log > [image: image.png] > > What could be the reason for this? > > Thanks > Shashank > > To view the terms under which this email is distributed, please go to:- > http://leedsbeckett.ac.uk/disclaimer/email/ > <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fleedsbecke…> > > To view the terms under which this email is distributed, please go to:- > http://leedsbeckett.ac.uk/disclaimer/email/ > <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fleedsbecke…> > > To view the terms under which this email is distributed, please go to:- > http://leedsbeckett.ac.uk/disclaimer/email/ > <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fleedsbecke…> > > To view the terms under which this email is distributed, please go to:- > http://leedsbeckett.ac.uk/disclaimer/email/ > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/BCU7ZTSVHZR7P… >

1 0

VM rebooted during OST test_hotplug_cpu
by Yedidyah Bar David 01 Jul '20

01 Jul '20

Hi all, I am trying to verify fixes for ovirt-engine-rename, specifically for OVN. Engine top patch is [1], OST patch [2]. Ran the manual job on these [3]. In previous patches, OST failed in earlier tests. Now, it passed these tests, so I hope that my patches are enough for what I am trying to do. However, [3] did fail later, during test_hotplug_cpu - it set the number of CPUs, then tried to connect to the VM, and timed out. The logs imply that right after it changed the number of CPUs, the VM was rebooted, apparently by libvirtd. Relevant log snippets: vdsm [4]: 2020-06-29 10:21:10,889-0400 DEBUG (jsonrpc/1) [virt.vm] (vmId='7474280d-4501-4355-9425-63898757682b') Setting number of cpus to : 2 (vm:3089) 2020-06-29 10:21:10,952-0400 INFO (jsonrpc/1) [api.virt] FINISH setNumberOfCpus return={'status': {'code': 0, 'message': 'Done'}, 'vmList': {}} from=::ffff:192.168.201.4,54576, flow_id=7f9503ed, vmId=7474280d-4501-4355-9425-63898757682b (api:54) 2020-06-29 10:21:11,111-0400 DEBUG (periodic/0) [virt.sampling.VMBulkstatsMonitor] sampled timestamp 2925.602824355 elapsed 0.160 acquired True domains all (sampling:451) 2020-06-29 10:21:11,430-0400 DEBUG (jsonrpc/1) [jsonrpc.JsonRpcServer] Return 'VM.setNumberOfCpus' in bridge with {} (__init__:356) 2020-06-29 10:21:11,432-0400 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.setNumberOfCpus succeeded in 0.56 seconds (__init__:312) 2020-06-29 10:21:12,228-0400 INFO (libvirt/events) [virt.vm] (vmId='7474280d-4501-4355-9425-63898757682b') reboot event (vm:1033) qemu [5]: 2020-06-29T14:21:12.260303Z qemu-kvm: terminating on signal 15 from pid 42224 (<unknown process>) 2020-06-29 14:21:12.462+0000: shutting down, reason=destroyed libvirtd [6] itself does not log anything relevant AFAICT, but at least it shows that the above unknown process is itself: 2020-06-29 14:18:16.212+0000: 42224: error : qemuMonitorIO:620 : internal error: End of file from qemu monitor (Note that above line is from 3 minutes before the reboot, and the only place in the log with '42224'. No other log there has 42224, other than these and audit.log). Any idea? Is this a bug in libvirt? vdsm? I'd at least expect something in the log for such a severe step. [1] https://gerrit.ovirt.org/109961 [2] https://gerrit.ovirt.org/109734 [3] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… [4] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… [5] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… [6] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test… -- Didi

4 6