July 2021 - Devel - oVirt List Archives

[VDSM] Started getting virNetTLSContextCheckCertTimes:154 : The server certificate /etc/pki/vdsm/certs/vdsmcert.pem has expired
by Edward Haas 11 Jan '24

11 Jan '24

Hi, I have a VM which acts as an oVirt host with VDSM installed. I use it for testing without connecting it to Engine. Recently VDSM fails to come up and I see this error: Sep 26 17:58:18 localhost libvirtd: 2018-09-26 14:58:18.978+0000: 12176: error : virNetTLSContextCheckCertTimes:154 : The server certificate /etc/pki/vdsm/certs/vdsmcert.pem has expired Any hint on how to generate or fetch a new certificate without the help of Engine? Thanks, Edy.

3 5

Error: Adding new Host to ovirt-engine
by Ahmad Khiet 14 Aug '23

14 Aug '23

Hi, Can't add new host to ovirt engine, because the following error: 2019-06-12 12:23:09,664 p=4134 u=engine | TASK [ovirt-host-deploy-facts : Set facts] ************************************* 2019-06-12 12:23:09,684 p=4134 u=engine | ok: [10.35.1.17] => { "ansible_facts": { "ansible_python_interpreter": "/usr/bin/python2", "host_deploy_vdsm_version": "4.40.0" }, "changed": false } 2019-06-12 12:23:09,697 p=4134 u=engine | TASK [ovirt-provider-ovn-driver : Install ovs] ********************************* 2019-06-12 12:23:09,726 p=4134 u=engine | fatal: [10.35.1.17]: FAILED! => {} MSG: The conditional check 'cluster_switch == "ovs" or (ovn_central is defined and ovn_central | ipaddr and ovn_engine_cluster_version is version_compare('4.2', '>='))' failed. The error was: The ipaddr filter requires python's netaddr be installed on the ansible controller The error appears to be in '/home/engine/apps/engine/share/ovirt-engine/playbooks/roles/ovirt-provider-ovn-driver/tasks/configure.yml': line 3, column 5, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - block: - name: Install ovs ^ here 2019-06-12 12:23:09,728 p=4134 u=engine | PLAY RECAP ********************************************************************* 2019-06-12 12:23:09,728 p=4134 u=engine | 10.35.1.17 : ok=3 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 whats missing!? Thanks -- Ahmad Khiet Red Hat <https://www.redhat.com/> akhiet(a)redhat.com M: +972-54-6225629 <https://red.ht/sig>

7 10

lago is dead, long live ost!
by Michal Skrivanek 04 Aug '21

04 Aug '21

Hi, we worked for a while on removing our dependency on lago that has been more or less abandoned for a long time now. We got to the minimal feature set that is simple to replace with bunch of virsh commands, and most of the advanced logic is implemented in pytest. We’ve just merged the last patches that freed us up from lago for local and beaker CI runs. There’s a new ost.sh wrapper for the simple operations of running the suite, inspecting the environment and shutting it down. Hopefully it’s self explanatory…. ./ost.sh command [arguments] run <suite> <distro> [<pytest args>...] initializes the workspace with preinstalled distro ost-images, launches VMs and runs the whole suite add extra repos with --custom-repo=url skip check that extra repo is actually used with --skip-custom-repos-check status show environment status, VM details shell <host> [command ...] opens ssh connection console <host> opens virsh console destroy stop and remove the running environment Right now lago and run_suite.sh still works and it is still used by the mock-based jenkins.ovirt.org runs. It will go away in future. Thanks, michal

3 3

Re: RHV and oVirt CBT issue
by Nir Soffer 30 Jul '21

30 Jul '21

On Fri, Jul 30, 2021 at 5:47 AM luwen.zhang <luwen.zhang(a)vinchin.com> wrote: > Sorry I was trying to open a new thread for this issue, but it seems I failed to submit. Here let me explain how the issue is reproduced. > > It’s a regular backup by using CBT+imageip API, after a series of successful backup, at one of the backup session beginning, when we try to obtain the VM config and the snapshot list (obtain snapshot list can determine the VM virtual disk format is RAW or QCOW2) Why do you need the snapshot list when doing incremental backup? What you need is the list of disks in the vms, accessible via: GET /vms/{vm-id}/diskattachments For each disk attachment, get the disk using the diskattachment.disk.id: GET /disks/{disk-id}/ Please check how we do this in backup_vm.py example: https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/backup_v… > by using `GET vms/<vm-id>/snapshots`, but get the following error. > > <?xml version="1.0" encoding="UTF-8" standalone="yes"?> > > <fault> > > <detail>duplicate key acf1edaa-e950-4c4f-94df-1bd6b3da49c1 (attempted merging values org.ovirt.engine.core.common.businessentities.storage.diskimage@5103046c and org.ovirt.engine.core.common.businessentities.storage.diskimage@d973046c)</detail> > > <reason>Operation Failed</reason> > > </fault> We need a much more detailed steps. This is a typical backup flow: 1. Start incremental backup 2. Wait until backup is ready (phase == READY) 3. Start image transfer for incremental backup 4. Wait until image transfer is ready (phase == TRANSFERRING) 5. Download disk incremental data 6. Finalize transfer 7. Wait until transfer is finished (phase == FINISHED_SUCCESS/FINISHED_FAILURE) This is not easy, see this example: https://github.com/oVirt/ovirt-engine-sdk/blob/ac6f05bb5dcd8fdee2a67b2a296a… 8. Finalize backup 9. Wait until backup is finished (phase == FINISHED/FAILED) This is easier, but possible only since 4.4.7: https://github.com/oVirt/ovirt-engine-sdk/blob/ac6f05bb5dcd8fdee2a67b2a296a… 10. Rebase backup image on previous backup (if you store backup as qcow2 layers) Where in this flow you get the snapshot list (and other stuff?) Getting snapshots list is likely not needed for backup, but we need to fix it in case it is broken while running backups or image transfers. Do you run this flow in a loop? Maybe you do not wait until the previous image transfer was finished before starting a new backup? > After this, on oVirt engine web console, the VM show 2 disks (actually it only has 1) , and the disk status always showing “Finalizing”, it’s been more than 30 hours now, and during this, cannot modify VM disk or take snapshots. > > Before upgrading oVirt engine to 4.4.7.7-1.el8 this problem happened frequently, after upgrading, the frequency is reduced. > > Here I’m adding the engine logs and vdsm logs. > Engine logs: https://drive.google.com/file/d/1T3-EOxYYl3oFZOA9VMMBte5WyBoUO48U/view?usp=… > VDSM logs: https://drive.google.com/file/d/1x0B8lGqnKEDrgn666CuN3hqUGwD7fcYv/view?usp=… Thanks, we will check the logs next week. > Thanks & regards! > On 07/29/2021 19:20，Nir Soffer<nsoffer(a)redhat.com> wrote： > > On Thu, Jul 29, 2021 at 10:08 AM luwen.zhang <luwen.zhang(a)vinchin.com> wrote: > > The problem occurred yesterday, but we waited for more than 20 hours, still 2 disks and in Finalizing state. > > > If the image transfer is "finalizing" it means the image transfer is > trying to finalize, but the operation could not complete. > > In this phase the disk remains locked, and it should not be possible > to start a new image transfer > (e.g perform another backup). > > Engine and vdsm logs should explain why the image transfer is stuck in > the finalizing phase. > > Can you add detailed instructions on how to reproduce this issue? >

1 0

This year conference community updates
by Sandro Bonazzola 30 Jul '21

30 Jul '21

Hi, as announced we are going to have oVirt online conference coming in September. I would like to give usual community updates in and as part of that I'd like to report about oVirt downstream products (such as Red Hat Virtualization and Oracle Linux Virtualization Manager) and about companies using oVirt. I could just go googling and guessing by e-mail traffic on oVirt users list but I want to try a different approach this year. So, if you are shipping an oVirt downstream or if you're using oVirt and you'd like to get the company nominated during the community reports please let me know off-list. Also I'd like to remind that oVirt site has a section dedicated to user stories, if you want to share yours you can either push it to https://github.com/oVirt/ovirt-site or get in touch with me off-list to get help getting your story published. Thanks, -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 1

ovirt-provider-ovn git repo got broken
by Sandro Bonazzola 30 Jul '21

30 Jul '21

Hi, not sure what happened but ovirt-provider-ovn repo in gerrit (and its backup on github) now contains a bunch of ovirt-engine tags shipping ovirt-engine content. Can you please have a look? $ git fetch --tags $ git tag --list |grep ovirt-engine ovirt-engine-3.0.0_0001 ovirt-engine-3.1.0 ovirt-engine-3.2.0 ovirt-engine-3.2.0-4 ovirt-engine-3.2.1 ovirt-engine-3.2.1-1 ovirt-engine-3.2.2 ovirt-engine-3.3.0 ovirt-engine-3.3.0.1 ovirt-engine-3.3.1 ovirt-engine-3.3.1_beta ovirt-engine-3.3.1_rc1 ovirt-engine-3.3.2 ovirt-engine-3.3.2_beta1 ovirt-engine-3.3.2_rc1 ovirt-engine-3.3.3_beta1 ovirt-engine-3.3_beta1 ovirt-engine-3.3_rc ovirt-engine-3.3_rc2 ovirt-engine-3.5.2 ovirt-engine-3.5.2.1 ovirt-engine-3.5.4.1 ovirt-engine-3.6.0_alpha1 ovirt-engine-3.6.2.1 ovirt-engine-4.0.2.3 ovirt-engine-4.1.5 ovirt-engine-4.1.7 ovirt-engine-4.1.7.1 ovirt-engine-4.1.7.3 ovirt-engine-4.1.9.2 ovirt-engine-4.2.0 ovirt-engine-4.2.1.5 -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

2 1

Re: RHV and oVirt CBT issue
by Nir Soffer 30 Jul '21

30 Jul '21

On Thu, Jul 29, 2021 at 10:08 AM luwen.zhang <luwen.zhang(a)vinchin.com> wrote: > The problem occurred yesterday, but we waited for more than 20 hours, still 2 disks and in Finalizing state. If the image transfer is "finalizing" it means the image transfer is trying to finalize, but the operation could not complete. In this phase the disk remains locked, and it should not be possible to start a new image transfer (e.g perform another backup). Engine and vdsm logs should explain why the image transfer is stuck in the finalizing phase. Can you add detailed instructions on how to reproduce this issue?

2 1

after a regular backup, VM got duplicated disks and disks' status always in Finalizing
by luwen.zhang＠vinchin.com 30 Jul '21

30 Jul '21

Version: oVirt engine: 4.4.7.7-1.el8 oVirt node: 4.4.7 Steps of reproducing the problem: After a series of regular/successful backup tests by using CBT+imageio API, at one of the backup session beginning, when trying to obtain the VM config and snapshot list (obtain snapshot list can determine the VM disk is in RAW or QCOW2 format) by using `GET vms/<vm-id>/snapshots`, the below error is received. <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <fault> <detail>duplicate key acf1edaa-e950-4c4f-94df-1bd6b3da49c1 (attempted merging values org.ovirt.engine.core.common.businessentities.storage.diskimage@5103046c and org.ovirt.engine.core.common.businessentities.storage.diskimage@d973046c)</detail> <reason>Operation Failed</reason> </fault> After this on the oVirt engine web console, we saw the VM which should only have 1 disk is now showing 2 disks, the disk status always in Finalizing state. It's been more than 30 hours but the VM still got duplicated disks and still showing Finalizing, there's no way to modify the virtual disk or take snapshots for the VM.

1 0

Re: RHV and oVirt CBT issue
by Eyal Shenitzky 29 Jul '21

29 Jul '21

+Benny Zlotnik <bzlotnik(a)redhat.com> +Pavel Bar <pbar(a)redhat.com> On Thu, 29 Jul 2021 at 07:08, luwen.zhang <luwen.zhang(a)vinchin.com> wrote: > > > Dear all, > > FYI, after upgrading oVirt engine to 4.4.7.7-1.el8 the chance of > reproducing this issue is reduced, but it still persists. > Can you please open a bug for it and mention the steps you did to reproduce the problem? Also, please include all the relevant logs in the bug. Benny/Pavel, is there a WA for getting out of this issue besides removing the entry from the DB? > > The oVirt node version is also 4.4.7, the virtual disks are stored in > iSCSI LUN. > > The problem occurred yesterday, but we waited for more than 20 hours, > still 2 disks and in Finalizing state. > > > At this moment, we are unable to check/delete snapshots, or perform > backups. > > Please let us know if we can do anything to help with the debug. > > Thanks & regards! > > Luwen Zhang > On 07/27/2021 18:30，luwen.zhang<luwen.zhang(a)vinchin.com> > <luwen.zhang(a)vinchin.com> wrote： > > > Thank you all for your help, after upgrading oVirt engine the issue is now > fixed. > > Best regards! > Luwen Zhang > On 07/27/2021 18:03，Lev Veyde<lveyde(a)redhat.com> <lveyde(a)redhat.com> > wrote： > > Hi Sandro, > > Yes, it was released as part of the 0-day fix on July 20, including a new > appliance build. > > Thanks in advance, > > On Tue, Jul 27, 2021 at 12:20 PM Sandro Bonazzola <sbonazzo(a)redhat.com> > wrote: > >> >> >> Il giorno mar 27 lug 2021 alle ore 10:53 Eyal Shenitzky < >> eshenitz(a)redhat.com> ha scritto: >> >>> As +Michal Skrivanek <mskrivan(a)redhat.com> already replied, this issue >>> cause by - https://bugzilla.redhat.com/show_bug.cgi?id=1980428. >>> >>> It is already fixed in RHV 4.4.7 and on the master branch. >>> +Sandro Bonazzola <sbonazzo(a)redhat.com>, did we include it on oVirt >>> 4.4.7 also? >>> >> >> $ git tag --contains 6cd8bc532b0a4a6839ed667c67489256ae9ff5b9 >> ovirt-engine-4.4.7.7 >> >> It was shipped on July 20th but I don't see an announcement for it. +Lev >> Veyde <lveyde(a)redhat.com> ? >> >> >> >> >> >>> >>> On Tue, 27 Jul 2021 at 09:44, luwen.zhang <luwen.zhang(a)vinchin.com> >>> wrote: >>> >>>> >>>> Dear team could you please help on the below issue? >>>> >>>> Thanks Sandro for the advice! >>>> On 07/27/2021 14:22，Sandro Bonazzola<sbonazzo(a)redhat.com> >>>> <sbonazzo(a)redhat.com> wrote： >>>> >>>> I would recommend to ask this on devel(a)ovirt.org. >>>> Looping in +Eyal Shenitzky <eshenitz(a)redhat.com> >>>> >>>> Il giorno mar 27 lug 2021 alle ore 08:14 luwen.zhang < >>>> luwen.zhang(a)vinchin.com> ha scritto: >>>> >>>>> >>>>> Dear team, >>>>> >>>>> We had implemented CBT support for RHV and oVirt in our new version, >>>>> but when testing we encountered problems of obtain VM increments and also >>>>> had abnormal with the VM virtual disks. >>>>> >>>>> Our testing environment as below: >>>>> OS Version: RHEL - 8.4.2105.0 - 3.el8 >>>>> OS Description: oVirt Node 4.4.7 >>>>> Kernel Version: 4.18.0 - 315.el8.x86_64 >>>>> KVM Version: 6.0.0 - 19.el8s >>>>> LIBVIRT Version: libvirt-7.4.0-1.el8s >>>>> VDSM Version: vdsm-4.40.70.6-1.el8 >>>>> >>>>> One of the problem is after a successful backup, the VM which gets >>>>> only one virtual disk will get 2 identical virtual disks marked as Complete >>>>> on oVirt engine web console, like shown below. >>>>> >>>>> The additional disk might disappear after a while, but before it’s >>>>> gone there could be a lot of problems with the VM. >>>>> >>>>> 1. We cannot perform a new backup, the backup will fail directly. >>>>> >>>>> 2. The VM itself will run into exception, we cannot even power it off, >>>>> and it will also probably get crashed. >>>>> >>>>> 3. If the VM was initially powered off, after backing up, we cannot >>>>> power on the VM. >>>>> >>>>> And there could be some other problems related with this, is this >>>>> normal or there’s something we did wrong? >>>>> >>>>> Thank you in advance for your help! >>>>> >>>>> >>>>> *Luwen Zhang* | Product Manager >>>>> >>>>> *Mobile*: +8613880424687 | *Skype*: luwen.zhang_cn >>>>> >>>>> *WhatsApp*: +8613880424687 | *WeChat*: +8613880424687 >>>>> >>>>> *Website*: www.vinchin.com >>>>> >>>>> *Address*: F5, Block 8, National Information Security Industry Park, >>>>> >>>>> No.333 Yunhua Road, Hi-tech Zone, Chengdu, China | *Zip*: 610015 >>>>> >>>>> *INNOVATIVE VM BACKUP EXPERT* >>>>> >>>>> VMware Backup >>>>> <https://www.vinchin.com/en/hypervisor/vmware-backup.html> | XenServer/Citrix >>>>> Hypervisor/XCP-ng Backup >>>>> <https://www.vinchin.com/en/hypervisor/xcp-ng-xenserver-backup.html> >>>>> | Hyper-V Backup >>>>> <https://www.vinchin.com/en/hypervisor/hyper-v-backup.html> >>>>> >>>>> RHEV/oVirt Backup >>>>> <https://www.vinchin.com/en/hypervisor/red-hat-virtualization-ovirt-backup.h…> >>>>> | OLVM Backup >>>>> <https://www.vinchin.com/en/hypervisor/oracle-linux-virtualization-manager-b…> >>>>> | Sangfor HCI Backup >>>>> <https://www.vinchin.com/en/hypervisor/sangfor-hci-backup.html> | OpenStack >>>>> Backup <https://www.vinchin.com/en/hypervisor/openstack-backup.html> >>>>> >>>>> >>>>> >>>> >>>> -- >>>> >>>> Sandro Bonazzola >>>> >>>> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV >>>> >>>> Red Hat EMEA <https://www.redhat.com/> >>>> >>>> sbonazzo(a)redhat.com >>>> <https://www.redhat.com/> >>>> >>>> *Red Hat respects your work life balance. Therefore there is no need to >>>> answer this email out of your office hours.* >>>> >>>> >>>> >>> >>> -- >>> Regards, >>> Eyal Shenitzky >>> >> >> >> -- >> >> Sandro Bonazzola >> >> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV >> >> Red Hat EMEA <https://www.redhat.com/> >> >> sbonazzo(a)redhat.com >> <https://www.redhat.com/> >> >> *Red Hat respects your work life balance. Therefore there is no need to >> answer this email out of your office hours. >> <https://mojo.redhat.com/docs/DOC-1199578>* >> >> >> > > -- > > Lev Veyde > > Senior Software Engineer, RHCE | RHCVA | MCITP > > Red Hat Israel > > <https://www.redhat.com> > > lev(a)redhat.com | lveyde(a)redhat.com > <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > > -- Regards, Eyal Shenitzky

3 3

ovirt-engine has been tagged (ovirt-engine-4.4.8.2)
by Tal Nisan 29 Jul '21

29 Jul '21

1 0

Re: RHV and oVirt CBT issue
by luwen.zhang 29 Jul '21

29 Jul '21

1 0

Re: RHV and oVirt CBT issue
by Eyal Shenitzky 27 Jul '21

27 Jul '21

As +Michal Skrivanek <mskrivan(a)redhat.com> already replied, this issue cause by - https://bugzilla.redhat.com/show_bug.cgi?id=1980428. It is already fixed in RHV 4.4.7 and on the master branch. +Sandro Bonazzola <sbonazzo(a)redhat.com>, did we include it on oVirt 4.4.7 also? On Tue, 27 Jul 2021 at 09:44, luwen.zhang <luwen.zhang(a)vinchin.com> wrote: > > Dear team could you please help on the below issue? > > Thanks Sandro for the advice! > On 07/27/2021 14:22，Sandro Bonazzola<sbonazzo(a)redhat.com> > <sbonazzo(a)redhat.com> wrote： > > I would recommend to ask this on devel(a)ovirt.org. > Looping in +Eyal Shenitzky <eshenitz(a)redhat.com> > > Il giorno mar 27 lug 2021 alle ore 08:14 luwen.zhang < > luwen.zhang(a)vinchin.com> ha scritto: > >> >> Dear team, >> >> We had implemented CBT support for RHV and oVirt in our new version, but >> when testing we encountered problems of obtain VM increments and also had >> abnormal with the VM virtual disks. >> >> Our testing environment as below: >> OS Version: RHEL - 8.4.2105.0 - 3.el8 >> OS Description: oVirt Node 4.4.7 >> Kernel Version: 4.18.0 - 315.el8.x86_64 >> KVM Version: 6.0.0 - 19.el8s >> LIBVIRT Version: libvirt-7.4.0-1.el8s >> VDSM Version: vdsm-4.40.70.6-1.el8 >> >> One of the problem is after a successful backup, the VM which gets only >> one virtual disk will get 2 identical virtual disks marked as Complete on >> oVirt engine web console, like shown below. >> >> The additional disk might disappear after a while, but before it’s gone >> there could be a lot of problems with the VM. >> >> 1. We cannot perform a new backup, the backup will fail directly. >> >> 2. The VM itself will run into exception, we cannot even power it off, >> and it will also probably get crashed. >> >> 3. If the VM was initially powered off, after backing up, we cannot power >> on the VM. >> >> And there could be some other problems related with this, is this normal >> or there’s something we did wrong? >> >> Thank you in advance for your help! >> >> >> *Luwen Zhang* | Product Manager >> >> *Mobile*: +8613880424687 | *Skype*: luwen.zhang_cn >> >> *WhatsApp*: +8613880424687 | *WeChat*: +8613880424687 >> >> *Website*: www.vinchin.com >> >> *Address*: F5, Block 8, National Information Security Industry Park, >> >> No.333 Yunhua Road, Hi-tech Zone, Chengdu, China | *Zip*: 610015 >> >> *INNOVATIVE VM BACKUP EXPERT* >> >> VMware Backup <https://www.vinchin.com/en/hypervisor/vmware-backup.html> >> | XenServer/Citrix Hypervisor/XCP-ng Backup >> <https://www.vinchin.com/en/hypervisor/xcp-ng-xenserver-backup.html> | Hyper-V >> Backup <https://www.vinchin.com/en/hypervisor/hyper-v-backup.html> >> >> RHEV/oVirt Backup >> <https://www.vinchin.com/en/hypervisor/red-hat-virtualization-ovirt-backup.h…> >> | OLVM Backup >> <https://www.vinchin.com/en/hypervisor/oracle-linux-virtualization-manager-b…> >> | Sangfor HCI Backup >> <https://www.vinchin.com/en/hypervisor/sangfor-hci-backup.html> | OpenStack >> Backup <https://www.vinchin.com/en/hypervisor/openstack-backup.html> >> >> >> > > -- > > Sandro Bonazzola > > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > > Red Hat EMEA <https://www.redhat.com/> > > sbonazzo(a)redhat.com > <https://www.redhat.com/> > > *Red Hat respects your work life balance. Therefore there is no need to > answer this email out of your office hours.* > > > -- Regards, Eyal Shenitzky

4 4

Re: RHV and oVirt CBT issue
by Nir Soffer 27 Jul '21

27 Jul '21

On Tue, Jul 27, 2021 at 9:47 AM luwen.zhang <luwen.zhang(a)vinchin.com> wrote: > > Dear team could you please help on the below issue? > > Thanks Sandro for the advice! > On 07/27/2021 14:22，Sandro Bonazzola<sbonazzo(a)redhat.com> > <sbonazzo(a)redhat.com> wrote： > > I would recommend to ask this on devel(a)ovirt.org. > Looping in +Eyal Shenitzky <eshenitz(a)redhat.com> > > Il giorno mar 27 lug 2021 alle ore 08:14 luwen.zhang < > luwen.zhang(a)vinchin.com> ha scritto: > >> >> Dear team, >> >> We had implemented CBT support for RHV and oVirt in our new version, but >> when testing we encountered problems of obtain VM increments and also had >> abnormal with the VM virtual disks. >> >> Our testing environment as below: >> OS Version: RHEL - 8.4.2105.0 - 3.el8 >> OS Description: oVirt Node 4.4.7 >> Kernel Version: 4.18.0 - 315.el8.x86_64 >> KVM Version: 6.0.0 - 19.el8s >> LIBVIRT Version: libvirt-7.4.0-1.el8s >> VDSM Version: vdsm-4.40.70.6-1.el8 >> >> One of the problem is after a successful backup, the VM which gets only >> one virtual disk will get 2 identical virtual disks marked as Complete on >> oVirt engine web console, like shown below. >> >> The additional disk might disappear after a while, but before it’s gone >> there could be a lot of problems with the VM. >> >> 1. We cannot perform a new backup, the backup will fail directly. >> >> 2. The VM itself will run into exception, we cannot even power it off, >> and it will also probably get crashed. >> >> 3. If the VM was initially powered off, after backing up, we cannot power >> on the VM. >> >> And there could be some other problems related with this, is this normal >> or there’s something we did wrong? >> >> Thank you in advance for your help! >> > Unfortunately you are hitting this bug: https://bugzilla.redhat.com/1980428 The bug is fixed in 4.4.7 async release, which should be available now according to the bug. Please upgrade to the latest 4.4.7 version. Nir

2 1

Re: RHV and oVirt CBT issue
by luwen.zhang 27 Jul '21

27 Jul '21

1 0

ovirt-engine has been tagged (ovirt-engine-4.4.8.1)
by Tal Nisan 21 Jul '21

21 Jul '21

2 1

Reminder : last week for submitting an abstract for ovirt conference
by Sandro Bonazzola 21 Jul '21

21 Jul '21

Just a reminder that call for papers closes on July 25 and this is the last week! -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>*

1 0

Blog post - Using Ceph only storage for oVirt datacenter
by Sandro Bonazzola 15 Jul '21

15 Jul '21

Hi, I just published the result of the testing of oVirt 4.4.7 with Ceph Pacific and RDO Victoria for deploying an oVirt datacenter using Ceph as storage with a blog post at https://blogs.ovirt.org/2021/07/using-ceph-only-storage-for-ovirt-datacente… Comments and suggestions are welcome. Thanks, -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

4 7

a kind reminder oVirt 2021 CfP closing in 11 days
by Sandro Bonazzola 14 Jul '21

14 Jul '21

Hi, a kind reminder oVirt 2021 CfP closing in 11 days. More details: https://blogs.ovirt.org/ovirt-2021-online-conference/ Looking forward to user stories, developers onboarding journeys, integration stories, new features presentation, old features nobody knows about because nobody presented them before and more you see fit. -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

ovirt-engine has been tagged (ovirt-engine-4.4.8)
by Tal Nisan 14 Jul '21

14 Jul '21

1 0

OST failure, bad comment
by Nir Soffer 12 Jul '21

12 Jul '21

OST failed, and posted unprocessed comment: OST Verified -1 Patch Set 1: Verified-1 https://redir.apps.ovirt.org/dj/job/ds-ost-gating-trigger2/313 : OST https://redir.apps.ovirt.org/dj/job/ds-ost-baremetal_manual/$OST_NUMBER failed with: $OST_MESSAGE https://gerrit.ovirt.org/c/vdsm/+/115642#message-b2993761_adbc1b6a Looking at: https://redir.apps.ovirt.org/dj/job/ds-ost-gating-trigger2/313 It looks like environment issue: ERROR: Build step failed with exception java.lang.NullPointerException: no workspace from node hudson.slaves.DumbSlave[hpe-bl280g6-12...] which is computer hudson.slaves.SlaveComputer@2e2cb321 and has channel null So we have 2 issues: 1. If build fails, error is not handled properly producing bad comment 2. the actual build failure The first issue should be handled by posting a better comment: OST build error: https://redir.apps.ovirt.org/dj/job/ds-ost-gating-trigger2/313 And OST should not mark the patch as verified -1, since it did not run any test. Marking a patch as -1 means some tests have failed, and the author needs to check why. A build error means OST or CI maintainers need to check why the error happened.

2 2

test_cold_incremental_backup_vm2: "Cannot backup VM. The VM is during a backup operation"
by Yedidyah Bar David 08 Jul '21

08 Jul '21

On Sun, Jul 4, 2021 at 8:26 AM <jenkins(a)jenkins.phx.ovirt.org> wrote: > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… > Build Number: 1291 > Build Status: Failure > Triggered By: Started by timer > > ------------------------------------- > Changes Since Last Success: > ------------------------------------- > Changes for Build #1291 > [Marcin Sobczyk] network: Add missing 'ansible_hosts' fixture > > > > > ----------------- > Failed Tests: > ----------------- > 1 tests failed. > FAILED: basic-suite-master.test-scenarios.test_004_basic_sanity.test_cold_incremental_backup_vm2 > > Error Message: > ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Cannot backup VM. The VM is during a backup operation.]". HTTP response code is 409. Something similar now happened to me too: https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/17729/ https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/17729… This job triggered also he-basic, which passed. basic failed as in here. Any idea? Best regards, > > Stack Trace: > engine_api = <ovirtsdk4.Connection object at 0x7f1f943c3208> > get_vm_service_for_vm = <function get_vm_service_for_vm.<locals>.service_for at 0x7f1f944e1268> > > @order_by(_TEST_LIST) > def test_cold_incremental_backup_vm2(engine_api, get_vm_service_for_vm): > _verify_vm_state(engine_api.system_service(), VM2_NAME, types.VmStatus.DOWN) > vm2_backups_service = get_vm_service_for_vm(VM2_NAME).backups_service() > backup.perform_incremental_vm_backup( > > engine_api, vm2_backups_service, DISK2_NAME, "cold_vm_backup") > > basic-suite-master/test-scenarios/test_004_basic_sanity.py:1056: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ost_utils/ost_utils/storage_utils/backup.py:75: in perform_incremental_vm_backup > correlation_id="incremental_" + correlation_id) > ost_utils/ost_utils/storage_utils/backup.py:34: in perform_vm_backup > ), query={'correlation_id': correlation_id} > /usr/lib64/python3.6/site-packages/ovirtsdk4/services.py:34139: in add > return self._internal_add(backup, headers, query, wait) > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:232: in _internal_add > return future.wait() if wait else future > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:55: in wait > return self._code(response) > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:229: in callback > self._check_fault(response) > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:132: in _check_fault > self._raise_error(response, body) > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > response = <ovirtsdk4.http.Response object at 0x7f1f94399c50> > detail = <ovirtsdk4.types.Fault object at 0x7f1f94399f60> > > @staticmethod > def _raise_error(response, detail=None): > """ > Creates and raises an error containing the details of the given HTTP > response and fault. > > This method is intended for internal use by other components of the > SDK. Refrain from using it directly, as backwards compatibility isn't > guaranteed. > """ > fault = detail if isinstance(detail, types.Fault) else None > > msg = '' > if fault: > if fault.reason: > if msg: > msg += ' ' > msg = msg + 'Fault reason is "%s".' % fault.reason > if fault.detail: > if msg: > msg += ' ' > msg = msg + 'Fault detail is "%s".' % fault.detail > if response: > if response.code: > if msg: > msg += ' ' > msg = msg + 'HTTP response code is %s.' % response.code > if response.message: > if msg: > msg += ' ' > msg = msg + 'HTTP response message is "%s".' % response.message > > if isinstance(detail, six.string_types): > if msg: > msg += ' ' > msg = msg + detail + '.' > > class_ = Error > if response is not None: > if response.code in [401, 403]: > class_ = AuthError > elif response.code == 404: > class_ = NotFoundError > > error = class_(msg) > error.code = response.code if response else None > error.fault = fault > > raise error > E ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Cannot backup VM. The VM is during a backup operation.]". HTTP response code is 409. > > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:118: Error_______________________________________________ > Infra mailing list -- infra(a)ovirt.org > To unsubscribe send an email to infra-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/PCZSYD7BHWGVM… -- Didi

4 3

importing fixtures in ovirt-system-tests
by Yedidyah Bar David 08 Jul '21

08 Jul '21

Hi all, We have several different styles of where/how to import fixtures in ovirt-system-tests: - import directly inside the test code - import in conftest.py - import in fixtures modules - For all of above, both 'import *' and importing specific fixtures I think we should try to agree on a specific style and then follow it. One drawback of importing directly in test/fixtures code is that it's then impossible to override them in conftest.py. A drawback of importing '*' and/or doing this in conftest.py is that you might inadvertently import more than you want, or this might happen eventually (after more stuff are added), that this makes it harder to find what uses what, and that it risks unintended collisions in names - as opposed to intended overrides. A related issue is having to update many places if you add/change something. If there is some kind of "best practices" document somewhere that people are happy with, perhaps we should follow it. Otherwise, we should come up with our own. Personally I think I'd like to have a single file with all the imports, of specific fixtures (not '*'), and import this file from conftest.py of all the suites. Didn't actually try this and no idea what complications it might bring. Comments/ideas/opinions/decisions are welcome :-) Best regards, -- Didi

3 2

ppc64le mock environment is broken
by Sandro Bonazzola 07 Jul '21

07 Jul '21

Looking at https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-release_standard… It shows: # /usr/bin/yum --installroot /var/lib/mock/centos-stream-8-ppc64le-6cf77c626743751e199a195f6b050912-bootstrap-10125/root/ --releasever 8 install dnf dnf-plugins-core --setopt=tsflags=nocontexts command being executed and failing with: [2021-07-07T15:18:32.577Z] --> Finished Dependency Resolution [2021-07-07T15:18:32.577Z] Error: Package: python3-dnf-4.7.0-1.el8.noarch (centos-base-el8stream) [2021-07-07T15:18:32.577Z] Requires: python(abi) = 3.6 [2021-07-07T15:18:32.577Z] Error: Package: python3-dnf-4.7.0-1.el8.noarch (centos-base-el8stream) [2021-07-07T15:18:32.577Z] Requires: python3-libdnf >= 0.61.1 [2021-07-07T15:18:32.577Z] Error: Package: 1:python3-dateutil-2.6.1-6.el8.noarch (centos-base-el8stream) [2021-07-07T15:18:32.577Z] Requires: python(abi) = 3.6 [2021-07-07T15:18:32.577Z] Error: Package: python3-dnf-plugins-core-4.0.21-1.el8.noarch (centos-base-el8stream) [2021-07-07T15:18:32.577Z] Requires: python3-dbus [2021-07-07T15:18:32.577Z] Error: Package: python3-dnf-4.7.0-1.el8.noarch (centos-base-el8stream) [2021-07-07T15:18:32.577Z] Requires: python3-libcomps >= 0.1.8 [2021-07-07T15:18:32.577Z] Error: Package: dnf-data-4.7.0-1.el8.noarch (centos-base-el8stream) [2021-07-07T15:18:32.577Z] Requires: libreport-filesystem [2021-07-07T15:18:32.577Z] Error: Package: python3-dnf-4.7.0-1.el8.noarch (centos-base-el8stream) [2021-07-07T15:18:32.577Z] Requires: libmodulemd >= 2.9.3 [2021-07-07T15:18:32.577Z] Error: Package: python3-dnf-plugins-core-4.0.21-1.el8.noarch (centos-base-el8stream) [2021-07-07T15:18:32.577Z] Requires: python(abi) = 3.6 [2021-07-07T15:18:32.577Z] Error: Package: python3-dnf-plugins-core-4.0.21-1.el8.noarch (centos-base-el8stream) [2021-07-07T15:18:32.577Z] Requires: python3-hawkey >= 0.46.1 [2021-07-07T15:18:32.577Z] Error: Package: dnf-4.7.0-1.el8.noarch (centos-base-el8stream) [2021-07-07T15:18:32.577Z] Requires: /bin/sh [2021-07-07T15:18:32.577Z] Error: Package: python3-dnf-4.7.0-1.el8.noarch (centos-base-el8stream) [2021-07-07T15:18:32.577Z] Requires: python3-rpm >= 4.14.2-35 [2021-07-07T15:18:32.577Z] Error: Package: python3-dnf-4.7.0-1.el8.noarch (centos-base-el8stream) [2021-07-07T15:18:32.578Z] Requires: python3-hawkey >= 0.61.1 [2021-07-07T15:18:32.578Z] Error: Package: python3-dnf-4.7.0-1.el8.noarch (centos-base-el8stream) [2021-07-07T15:18:32.578Z] Requires: python3-libdnf [2021-07-07T15:18:32.578Z] Error: Package: python3-dnf-4.7.0-1.el8.noarch (centos-base-el8stream) [2021-07-07T15:18:32.578Z] Requires: /usr/libexec/platform-python [2021-07-07T15:18:32.578Z] Error: Package: python3-six-1.11.0-8.el8.noarch (centos-base-el8stream) [2021-07-07T15:18:32.578Z] Requires: python(abi) = 3.6 [2021-07-07T15:18:32.578Z] Error: Package: python3-dnf-4.7.0-1.el8.noarch (centos-base-el8stream) [2021-07-07T15:18:32.578Z] Requires: python3-gpg [2021-07-07T15:18:32.578Z] You could try using --skip-broken to work around the problem [2021-07-07T15:18:32.578Z] You could try running: rpm -Va --nofiles --nodigest I initially thought it was a repo issue on centos stream side and reported it at https://bugzilla.redhat.com/show_bug.cgi?id=1979601 But it sorted out it's a CI configuration issue. The proof repositories are fine: LC_ALL=C sudo dnf --disablerepo=* --repofrompath=baseos1, http://mirror.centos.org/centos/8/BaseOS/ppc64le/os/ --enablerepo=baseos1 --releasever=8 --installroot=/tmp/test --nogpgcheck install :qdnf dnf-plugins-core --setopt=tsflags=nocontexts --setopt=arch=ppc64le correctly install the needed packages. This is blocking CI on ppc64le architecture. -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 4

Please resync centos-stream-el8stream/base mirror
by Sandro Bonazzola 07 Jul '21

07 Jul '21

Looks like the mirror needs a re-sync: [2021-07-07T06:24:14.859Z] [MIRROR] libpmem-1.6.1-1.el8.x86_64.rpm: Status code: 404 for http://mirrors.phx.ovirt.org/repos/yum/centos-stream-el8stream/base/Package… (IP: 66.187.230.49) [2021-07-07T06:24:14.859Z] [MIRROR] libpmem-1.6.1-1.el8.x86_64.rpm: Status code: 404 for http://mirrors.phx.ovirt.org/repos/yum/centos-stream-el8stream/base/Package… (IP: 66.187.230.49) [2021-07-07T06:24:14.859Z] [MIRROR] libpmem-1.6.1-1.el8.x86_64.rpm: Status code: 404 for http://mirrors.phx.ovirt.org/repos/yum/centos-stream-el8stream/base/Package… (IP: 66.187.230.49) [2021-07-07T06:24:14.859Z] [FAILED] libpmem-1.6.1-1.el8.x86_64.rpm: Status code: 404 for http://mirrors.phx.ovirt.org/repos/yum/centos-stream-el8stream/base/Package… (IP: 66.187.230.49) [2021-07-07T06:24:14.859Z] Error: Error downloading packages: [2021-07-07T06:24:14.859Z] Status code: 404 for http://mirrors.phx.ovirt.org/repos/yum/centos-stream-el8stream/base/Package… (IP: 66.187.230.49) Thanks -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

4 3

Reminder: oVirt conference call for papers closes in 20 days
by Sandro Bonazzola 05 Jul '21

05 Jul '21

A kind reminder that oVirt conference call for papers closes in 20 days! Looking forward to user stories, developers onboarding journeys, integration stories, new features presentation, old features nobody knows about because nobody presented them before and more you see fit. You can find call for paper at https://blogs.ovirt.org/ovirt-2021-online-conference/ Thanks, -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0