- Devel - oVirt List Archives

lago is dead, long live ost!
by Michal Skrivanek 04 Aug '21

04 Aug '21

Hi, we worked for a while on removing our dependency on lago that has been more or less abandoned for a long time now. We got to the minimal feature set that is simple to replace with bunch of virsh commands, and most of the advanced logic is implemented in pytest. We’ve just merged the last patches that freed us up from lago for local and beaker CI runs. There’s a new ost.sh wrapper for the simple operations of running the suite, inspecting the environment and shutting it down. Hopefully it’s self explanatory…. ./ost.sh command [arguments] run <suite> <distro> [<pytest args>...] initializes the workspace with preinstalled distro ost-images, launches VMs and runs the whole suite add extra repos with --custom-repo=url skip check that extra repo is actually used with --skip-custom-repos-check status show environment status, VM details shell <host> [command ...] opens ssh connection console <host> opens virsh console destroy stop and remove the running environment Right now lago and run_suite.sh still works and it is still used by the mock-based jenkins.ovirt.org runs. It will go away in future. Thanks, michal

3 3

Re: RHV and oVirt CBT issue
by Nir Soffer 30 Jul '21

30 Jul '21

On Fri, Jul 30, 2021 at 5:47 AM luwen.zhang <luwen.zhang(a)vinchin.com> wrote: > Sorry I was trying to open a new thread for this issue, but it seems I failed to submit. Here let me explain how the issue is reproduced. > > It’s a regular backup by using CBT+imageip API, after a series of successful backup, at one of the backup session beginning, when we try to obtain the VM config and the snapshot list (obtain snapshot list can determine the VM virtual disk format is RAW or QCOW2) Why do you need the snapshot list when doing incremental backup? What you need is the list of disks in the vms, accessible via: GET /vms/{vm-id}/diskattachments For each disk attachment, get the disk using the diskattachment.disk.id: GET /disks/{disk-id}/ Please check how we do this in backup_vm.py example: https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/backup_v… > by using `GET vms/<vm-id>/snapshots`, but get the following error. > > <?xml version="1.0" encoding="UTF-8" standalone="yes"?> > > <fault> > > <detail>duplicate key acf1edaa-e950-4c4f-94df-1bd6b3da49c1 (attempted merging values org.ovirt.engine.core.common.businessentities.storage.diskimage@5103046c and org.ovirt.engine.core.common.businessentities.storage.diskimage@d973046c)</detail> > > <reason>Operation Failed</reason> > > </fault> We need a much more detailed steps. This is a typical backup flow: 1. Start incremental backup 2. Wait until backup is ready (phase == READY) 3. Start image transfer for incremental backup 4. Wait until image transfer is ready (phase == TRANSFERRING) 5. Download disk incremental data 6. Finalize transfer 7. Wait until transfer is finished (phase == FINISHED_SUCCESS/FINISHED_FAILURE) This is not easy, see this example: https://github.com/oVirt/ovirt-engine-sdk/blob/ac6f05bb5dcd8fdee2a67b2a296a… 8. Finalize backup 9. Wait until backup is finished (phase == FINISHED/FAILED) This is easier, but possible only since 4.4.7: https://github.com/oVirt/ovirt-engine-sdk/blob/ac6f05bb5dcd8fdee2a67b2a296a… 10. Rebase backup image on previous backup (if you store backup as qcow2 layers) Where in this flow you get the snapshot list (and other stuff?) Getting snapshots list is likely not needed for backup, but we need to fix it in case it is broken while running backups or image transfers. Do you run this flow in a loop? Maybe you do not wait until the previous image transfer was finished before starting a new backup? > After this, on oVirt engine web console, the VM show 2 disks (actually it only has 1) , and the disk status always showing “Finalizing”, it’s been more than 30 hours now, and during this, cannot modify VM disk or take snapshots. > > Before upgrading oVirt engine to 4.4.7.7-1.el8 this problem happened frequently, after upgrading, the frequency is reduced. > > Here I’m adding the engine logs and vdsm logs. > Engine logs: https://drive.google.com/file/d/1T3-EOxYYl3oFZOA9VMMBte5WyBoUO48U/view?usp=… > VDSM logs: https://drive.google.com/file/d/1x0B8lGqnKEDrgn666CuN3hqUGwD7fcYv/view?usp=… Thanks, we will check the logs next week. > Thanks & regards! > On 07/29/2021 19:20，Nir Soffer<nsoffer(a)redhat.com> wrote： > > On Thu, Jul 29, 2021 at 10:08 AM luwen.zhang <luwen.zhang(a)vinchin.com> wrote: > > The problem occurred yesterday, but we waited for more than 20 hours, still 2 disks and in Finalizing state. > > > If the image transfer is "finalizing" it means the image transfer is > trying to finalize, but the operation could not complete. > > In this phase the disk remains locked, and it should not be possible > to start a new image transfer > (e.g perform another backup). > > Engine and vdsm logs should explain why the image transfer is stuck in > the finalizing phase. > > Can you add detailed instructions on how to reproduce this issue? >

1 0

This year conference community updates
by Sandro Bonazzola 30 Jul '21

30 Jul '21

Hi, as announced we are going to have oVirt online conference coming in September. I would like to give usual community updates in and as part of that I'd like to report about oVirt downstream products (such as Red Hat Virtualization and Oracle Linux Virtualization Manager) and about companies using oVirt. I could just go googling and guessing by e-mail traffic on oVirt users list but I want to try a different approach this year. So, if you are shipping an oVirt downstream or if you're using oVirt and you'd like to get the company nominated during the community reports please let me know off-list. Also I'd like to remind that oVirt site has a section dedicated to user stories, if you want to share yours you can either push it to https://github.com/oVirt/ovirt-site or get in touch with me off-list to get help getting your story published. Thanks, -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 1

ovirt-provider-ovn git repo got broken
by Sandro Bonazzola 30 Jul '21

30 Jul '21

Hi, not sure what happened but ovirt-provider-ovn repo in gerrit (and its backup on github) now contains a bunch of ovirt-engine tags shipping ovirt-engine content. Can you please have a look? $ git fetch --tags $ git tag --list |grep ovirt-engine ovirt-engine-3.0.0_0001 ovirt-engine-3.1.0 ovirt-engine-3.2.0 ovirt-engine-3.2.0-4 ovirt-engine-3.2.1 ovirt-engine-3.2.1-1 ovirt-engine-3.2.2 ovirt-engine-3.3.0 ovirt-engine-3.3.0.1 ovirt-engine-3.3.1 ovirt-engine-3.3.1_beta ovirt-engine-3.3.1_rc1 ovirt-engine-3.3.2 ovirt-engine-3.3.2_beta1 ovirt-engine-3.3.2_rc1 ovirt-engine-3.3.3_beta1 ovirt-engine-3.3_beta1 ovirt-engine-3.3_rc ovirt-engine-3.3_rc2 ovirt-engine-3.5.2 ovirt-engine-3.5.2.1 ovirt-engine-3.5.4.1 ovirt-engine-3.6.0_alpha1 ovirt-engine-3.6.2.1 ovirt-engine-4.0.2.3 ovirt-engine-4.1.5 ovirt-engine-4.1.7 ovirt-engine-4.1.7.1 ovirt-engine-4.1.7.3 ovirt-engine-4.1.9.2 ovirt-engine-4.2.0 ovirt-engine-4.2.1.5 -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

2 1

Re: RHV and oVirt CBT issue
by Nir Soffer 30 Jul '21

30 Jul '21

On Thu, Jul 29, 2021 at 10:08 AM luwen.zhang <luwen.zhang(a)vinchin.com> wrote: > The problem occurred yesterday, but we waited for more than 20 hours, still 2 disks and in Finalizing state. If the image transfer is "finalizing" it means the image transfer is trying to finalize, but the operation could not complete. In this phase the disk remains locked, and it should not be possible to start a new image transfer (e.g perform another backup). Engine and vdsm logs should explain why the image transfer is stuck in the finalizing phase. Can you add detailed instructions on how to reproduce this issue?

2 1

after a regular backup, VM got duplicated disks and disks' status always in Finalizing
by luwen.zhang＠vinchin.com 30 Jul '21

30 Jul '21

Version: oVirt engine: 4.4.7.7-1.el8 oVirt node: 4.4.7 Steps of reproducing the problem: After a series of regular/successful backup tests by using CBT+imageio API, at one of the backup session beginning, when trying to obtain the VM config and snapshot list (obtain snapshot list can determine the VM disk is in RAW or QCOW2 format) by using `GET vms/<vm-id>/snapshots`, the below error is received. <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <fault> <detail>duplicate key acf1edaa-e950-4c4f-94df-1bd6b3da49c1 (attempted merging values org.ovirt.engine.core.common.businessentities.storage.diskimage@5103046c and org.ovirt.engine.core.common.businessentities.storage.diskimage@d973046c)</detail> <reason>Operation Failed</reason> </fault> After this on the oVirt engine web console, we saw the VM which should only have 1 disk is now showing 2 disks, the disk status always in Finalizing state. It's been more than 30 hours but the VM still got duplicated disks and still showing Finalizing, there's no way to modify the virtual disk or take snapshots for the VM.

1 0

Re: RHV and oVirt CBT issue
by Eyal Shenitzky 29 Jul '21

29 Jul '21

+Benny Zlotnik <bzlotnik(a)redhat.com> +Pavel Bar <pbar(a)redhat.com> On Thu, 29 Jul 2021 at 07:08, luwen.zhang <luwen.zhang(a)vinchin.com> wrote: > > > Dear all, > > FYI, after upgrading oVirt engine to 4.4.7.7-1.el8 the chance of > reproducing this issue is reduced, but it still persists. > Can you please open a bug for it and mention the steps you did to reproduce the problem? Also, please include all the relevant logs in the bug. Benny/Pavel, is there a WA for getting out of this issue besides removing the entry from the DB? > > The oVirt node version is also 4.4.7, the virtual disks are stored in > iSCSI LUN. > > The problem occurred yesterday, but we waited for more than 20 hours, > still 2 disks and in Finalizing state. > > > At this moment, we are unable to check/delete snapshots, or perform > backups. > > Please let us know if we can do anything to help with the debug. > > Thanks & regards! > > Luwen Zhang > On 07/27/2021 18:30，luwen.zhang<luwen.zhang(a)vinchin.com> > <luwen.zhang(a)vinchin.com> wrote： > > > Thank you all for your help, after upgrading oVirt engine the issue is now > fixed. > > Best regards! > Luwen Zhang > On 07/27/2021 18:03，Lev Veyde<lveyde(a)redhat.com> <lveyde(a)redhat.com> > wrote： > > Hi Sandro, > > Yes, it was released as part of the 0-day fix on July 20, including a new > appliance build. > > Thanks in advance, > > On Tue, Jul 27, 2021 at 12:20 PM Sandro Bonazzola <sbonazzo(a)redhat.com> > wrote: > >> >> >> Il giorno mar 27 lug 2021 alle ore 10:53 Eyal Shenitzky < >> eshenitz(a)redhat.com> ha scritto: >> >>> As +Michal Skrivanek <mskrivan(a)redhat.com> already replied, this issue >>> cause by - https://bugzilla.redhat.com/show_bug.cgi?id=1980428. >>> >>> It is already fixed in RHV 4.4.7 and on the master branch. >>> +Sandro Bonazzola <sbonazzo(a)redhat.com>, did we include it on oVirt >>> 4.4.7 also? >>> >> >> $ git tag --contains 6cd8bc532b0a4a6839ed667c67489256ae9ff5b9 >> ovirt-engine-4.4.7.7 >> >> It was shipped on July 20th but I don't see an announcement for it. +Lev >> Veyde <lveyde(a)redhat.com> ? >> >> >> >> >> >>> >>> On Tue, 27 Jul 2021 at 09:44, luwen.zhang <luwen.zhang(a)vinchin.com> >>> wrote: >>> >>>> >>>> Dear team could you please help on the below issue? >>>> >>>> Thanks Sandro for the advice! >>>> On 07/27/2021 14:22，Sandro Bonazzola<sbonazzo(a)redhat.com> >>>> <sbonazzo(a)redhat.com> wrote： >>>> >>>> I would recommend to ask this on devel(a)ovirt.org. >>>> Looping in +Eyal Shenitzky <eshenitz(a)redhat.com> >>>> >>>> Il giorno mar 27 lug 2021 alle ore 08:14 luwen.zhang < >>>> luwen.zhang(a)vinchin.com> ha scritto: >>>> >>>>> >>>>> Dear team, >>>>> >>>>> We had implemented CBT support for RHV and oVirt in our new version, >>>>> but when testing we encountered problems of obtain VM increments and also >>>>> had abnormal with the VM virtual disks. >>>>> >>>>> Our testing environment as below: >>>>> OS Version: RHEL - 8.4.2105.0 - 3.el8 >>>>> OS Description: oVirt Node 4.4.7 >>>>> Kernel Version: 4.18.0 - 315.el8.x86_64 >>>>> KVM Version: 6.0.0 - 19.el8s >>>>> LIBVIRT Version: libvirt-7.4.0-1.el8s >>>>> VDSM Version: vdsm-4.40.70.6-1.el8 >>>>> >>>>> One of the problem is after a successful backup, the VM which gets >>>>> only one virtual disk will get 2 identical virtual disks marked as Complete >>>>> on oVirt engine web console, like shown below. >>>>> >>>>> The additional disk might disappear after a while, but before it’s >>>>> gone there could be a lot of problems with the VM. >>>>> >>>>> 1. We cannot perform a new backup, the backup will fail directly. >>>>> >>>>> 2. The VM itself will run into exception, we cannot even power it off, >>>>> and it will also probably get crashed. >>>>> >>>>> 3. If the VM was initially powered off, after backing up, we cannot >>>>> power on the VM. >>>>> >>>>> And there could be some other problems related with this, is this >>>>> normal or there’s something we did wrong? >>>>> >>>>> Thank you in advance for your help! >>>>> >>>>> >>>>> *Luwen Zhang* | Product Manager >>>>> >>>>> *Mobile*: +8613880424687 | *Skype*: luwen.zhang_cn >>>>> >>>>> *WhatsApp*: +8613880424687 | *WeChat*: +8613880424687 >>>>> >>>>> *Website*: www.vinchin.com >>>>> >>>>> *Address*: F5, Block 8, National Information Security Industry Park, >>>>> >>>>> No.333 Yunhua Road, Hi-tech Zone, Chengdu, China | *Zip*: 610015 >>>>> >>>>> *INNOVATIVE VM BACKUP EXPERT* >>>>> >>>>> VMware Backup >>>>> <https://www.vinchin.com/en/hypervisor/vmware-backup.html> | XenServer/Citrix >>>>> Hypervisor/XCP-ng Backup >>>>> <https://www.vinchin.com/en/hypervisor/xcp-ng-xenserver-backup.html> >>>>> | Hyper-V Backup >>>>> <https://www.vinchin.com/en/hypervisor/hyper-v-backup.html> >>>>> >>>>> RHEV/oVirt Backup >>>>> <https://www.vinchin.com/en/hypervisor/red-hat-virtualization-ovirt-backup.h…> >>>>> | OLVM Backup >>>>> <https://www.vinchin.com/en/hypervisor/oracle-linux-virtualization-manager-b…> >>>>> | Sangfor HCI Backup >>>>> <https://www.vinchin.com/en/hypervisor/sangfor-hci-backup.html> | OpenStack >>>>> Backup <https://www.vinchin.com/en/hypervisor/openstack-backup.html> >>>>> >>>>> >>>>> >>>> >>>> -- >>>> >>>> Sandro Bonazzola >>>> >>>> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV >>>> >>>> Red Hat EMEA <https://www.redhat.com/> >>>> >>>> sbonazzo(a)redhat.com >>>> <https://www.redhat.com/> >>>> >>>> *Red Hat respects your work life balance. Therefore there is no need to >>>> answer this email out of your office hours.* >>>> >>>> >>>> >>> >>> -- >>> Regards, >>> Eyal Shenitzky >>> >> >> >> -- >> >> Sandro Bonazzola >> >> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV >> >> Red Hat EMEA <https://www.redhat.com/> >> >> sbonazzo(a)redhat.com >> <https://www.redhat.com/> >> >> *Red Hat respects your work life balance. Therefore there is no need to >> answer this email out of your office hours. >> <https://mojo.redhat.com/docs/DOC-1199578>* >> >> >> > > -- > > Lev Veyde > > Senior Software Engineer, RHCE | RHCVA | MCITP > > Red Hat Israel > > <https://www.redhat.com> > > lev(a)redhat.com | lveyde(a)redhat.com > <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > > -- Regards, Eyal Shenitzky

3 3

ovirt-engine has been tagged (ovirt-engine-4.4.8.2)
by Tal Nisan 29 Jul '21

29 Jul '21

1 0

Re: RHV and oVirt CBT issue
by luwen.zhang 29 Jul '21

29 Jul '21

1 0

Re: RHV and oVirt CBT issue
by Eyal Shenitzky 27 Jul '21

27 Jul '21

As +Michal Skrivanek <mskrivan(a)redhat.com> already replied, this issue cause by - https://bugzilla.redhat.com/show_bug.cgi?id=1980428. It is already fixed in RHV 4.4.7 and on the master branch. +Sandro Bonazzola <sbonazzo(a)redhat.com>, did we include it on oVirt 4.4.7 also? On Tue, 27 Jul 2021 at 09:44, luwen.zhang <luwen.zhang(a)vinchin.com> wrote: > > Dear team could you please help on the below issue? > > Thanks Sandro for the advice! > On 07/27/2021 14:22，Sandro Bonazzola<sbonazzo(a)redhat.com> > <sbonazzo(a)redhat.com> wrote： > > I would recommend to ask this on devel(a)ovirt.org. > Looping in +Eyal Shenitzky <eshenitz(a)redhat.com> > > Il giorno mar 27 lug 2021 alle ore 08:14 luwen.zhang < > luwen.zhang(a)vinchin.com> ha scritto: > >> >> Dear team, >> >> We had implemented CBT support for RHV and oVirt in our new version, but >> when testing we encountered problems of obtain VM increments and also had >> abnormal with the VM virtual disks. >> >> Our testing environment as below: >> OS Version: RHEL - 8.4.2105.0 - 3.el8 >> OS Description: oVirt Node 4.4.7 >> Kernel Version: 4.18.0 - 315.el8.x86_64 >> KVM Version: 6.0.0 - 19.el8s >> LIBVIRT Version: libvirt-7.4.0-1.el8s >> VDSM Version: vdsm-4.40.70.6-1.el8 >> >> One of the problem is after a successful backup, the VM which gets only >> one virtual disk will get 2 identical virtual disks marked as Complete on >> oVirt engine web console, like shown below. >> >> The additional disk might disappear after a while, but before it’s gone >> there could be a lot of problems with the VM. >> >> 1. We cannot perform a new backup, the backup will fail directly. >> >> 2. The VM itself will run into exception, we cannot even power it off, >> and it will also probably get crashed. >> >> 3. If the VM was initially powered off, after backing up, we cannot power >> on the VM. >> >> And there could be some other problems related with this, is this normal >> or there’s something we did wrong? >> >> Thank you in advance for your help! >> >> >> *Luwen Zhang* | Product Manager >> >> *Mobile*: +8613880424687 | *Skype*: luwen.zhang_cn >> >> *WhatsApp*: +8613880424687 | *WeChat*: +8613880424687 >> >> *Website*: www.vinchin.com >> >> *Address*: F5, Block 8, National Information Security Industry Park, >> >> No.333 Yunhua Road, Hi-tech Zone, Chengdu, China | *Zip*: 610015 >> >> *INNOVATIVE VM BACKUP EXPERT* >> >> VMware Backup <https://www.vinchin.com/en/hypervisor/vmware-backup.html> >> | XenServer/Citrix Hypervisor/XCP-ng Backup >> <https://www.vinchin.com/en/hypervisor/xcp-ng-xenserver-backup.html> | Hyper-V >> Backup <https://www.vinchin.com/en/hypervisor/hyper-v-backup.html> >> >> RHEV/oVirt Backup >> <https://www.vinchin.com/en/hypervisor/red-hat-virtualization-ovirt-backup.h…> >> | OLVM Backup >> <https://www.vinchin.com/en/hypervisor/oracle-linux-virtualization-manager-b…> >> | Sangfor HCI Backup >> <https://www.vinchin.com/en/hypervisor/sangfor-hci-backup.html> | OpenStack >> Backup <https://www.vinchin.com/en/hypervisor/openstack-backup.html> >> >> >> > > -- > > Sandro Bonazzola > > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > > Red Hat EMEA <https://www.redhat.com/> > > sbonazzo(a)redhat.com > <https://www.redhat.com/> > > *Red Hat respects your work life balance. Therefore there is no need to > answer this email out of your office hours.* > > > -- Regards, Eyal Shenitzky

4 4