Error: Adding new Host to ovirt-engine
by Ahmad Khiet
Hi,
Can't add new host to ovirt engine, because the following error:
2019-06-12 12:23:09,664 p=4134 u=engine | TASK [ovirt-host-deploy-facts :
Set facts] *************************************
2019-06-12 12:23:09,684 p=4134 u=engine | ok: [10.35.1.17] => {
"ansible_facts": {
"ansible_python_interpreter": "/usr/bin/python2",
"host_deploy_vdsm_version": "4.40.0"
},
"changed": false
}
2019-06-12 12:23:09,697 p=4134 u=engine | TASK [ovirt-provider-ovn-driver
: Install ovs] *********************************
2019-06-12 12:23:09,726 p=4134 u=engine | fatal: [10.35.1.17]: FAILED! =>
{}
MSG:
The conditional check 'cluster_switch == "ovs" or (ovn_central is defined
and ovn_central | ipaddr and ovn_engine_cluster_version is
version_compare('4.2', '>='))' failed. The error was: The ipaddr filter
requires python's netaddr be installed on the ansible controller
The error appears to be in
'/home/engine/apps/engine/share/ovirt-engine/playbooks/roles/ovirt-provider-ovn-driver/tasks/configure.yml':
line 3, column 5, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
- block:
- name: Install ovs
^ here
2019-06-12 12:23:09,728 p=4134 u=engine | PLAY RECAP
*********************************************************************
2019-06-12 12:23:09,728 p=4134 u=engine | 10.35.1.17 :
ok=3 changed=0 unreachable=0 failed=1 skipped=0 rescued=0
ignored=0
whats missing!?
Thanks
--
Ahmad Khiet
Red Hat <https://www.redhat.com/>
akhiet(a)redhat.com
M: +972-54-6225629
<https://red.ht/sig>
1 year, 4 months
lago is dead, long live ost!
by Michal Skrivanek
Hi,
we worked for a while on removing our dependency on lago that has been more or less abandoned for a long time now. We got to the minimal feature set that is simple to replace with bunch of virsh commands, and most of the advanced logic is implemented in pytest. We’ve just merged the last patches that freed us up from lago for local and beaker CI runs.
There’s a new ost.sh wrapper for the simple operations of running the suite, inspecting the environment and shutting it down.
Hopefully it’s self explanatory….
./ost.sh command [arguments]
run <suite> <distro> [<pytest args>...]
initializes the workspace with preinstalled distro ost-images, launches VMs and runs the whole suite
add extra repos with --custom-repo=url
skip check that extra repo is actually used with --skip-custom-repos-check
status
show environment status, VM details
shell <host> [command ...]
opens ssh connection
console <host>
opens virsh console
destroy
stop and remove the running environment
Right now lago and run_suite.sh still works and it is still used by the mock-based jenkins.ovirt.org runs. It will go away in future.
Thanks,
michal
3 years, 4 months
Re: RHV and oVirt CBT issue
by Nir Soffer
On Fri, Jul 30, 2021 at 5:47 AM luwen.zhang <luwen.zhang(a)vinchin.com> wrote:
> Sorry I was trying to open a new thread for this issue, but it seems I failed to submit. Here let me explain how the issue is reproduced.
>
> It’s a regular backup by using CBT+imageip API, after a series of successful backup, at one of the backup session beginning, when we try to obtain the VM config and the snapshot list (obtain snapshot list can determine the VM virtual disk format is RAW or QCOW2)
Why do you need the snapshot list when doing incremental backup? What you need
is the list of disks in the vms, accessible via:
GET /vms/{vm-id}/diskattachments
For each disk attachment, get the disk using the diskattachment.disk.id:
GET /disks/{disk-id}/
Please check how we do this in backup_vm.py example:
https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/backup...
> by using `GET vms/<vm-id>/snapshots`, but get the following error.
>
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
>
> <fault>
>
> <detail>duplicate key acf1edaa-e950-4c4f-94df-1bd6b3da49c1 (attempted merging values org.ovirt.engine.core.common.businessentities.storage.diskimage@5103046c and org.ovirt.engine.core.common.businessentities.storage.diskimage@d973046c)</detail>
>
> <reason>Operation Failed</reason>
>
> </fault>
We need a much more detailed steps.
This is a typical backup flow:
1. Start incremental backup
2. Wait until backup is ready (phase == READY)
3. Start image transfer for incremental backup
4. Wait until image transfer is ready (phase == TRANSFERRING)
5. Download disk incremental data
6. Finalize transfer
7. Wait until transfer is finished (phase == FINISHED_SUCCESS/FINISHED_FAILURE)
This is not easy, see this example:
https://github.com/oVirt/ovirt-engine-sdk/blob/ac6f05bb5dcd8fdee2a67b2a29...
8. Finalize backup
9. Wait until backup is finished (phase == FINISHED/FAILED)
This is easier, but possible only since 4.4.7:
https://github.com/oVirt/ovirt-engine-sdk/blob/ac6f05bb5dcd8fdee2a67b2a29...
10. Rebase backup image on previous backup (if you store backup as qcow2 layers)
Where in this flow you get the snapshot list (and other stuff?)
Getting snapshots list is likely not needed for backup, but we need to fix it
in case it is broken while running backups or image transfers.
Do you run this flow in a loop? Maybe you do not wait until the previous image
transfer was finished before starting a new backup?
> After this, on oVirt engine web console, the VM show 2 disks (actually it only has 1) , and the disk status always showing “Finalizing”, it’s been more than 30 hours now, and during this, cannot modify VM disk or take snapshots.
>
> Before upgrading oVirt engine to 4.4.7.7-1.el8 this problem happened frequently, after upgrading, the frequency is reduced.
>
> Here I’m adding the engine logs and vdsm logs.
> Engine logs: https://drive.google.com/file/d/1T3-EOxYYl3oFZOA9VMMBte5WyBoUO48U/view?us...
> VDSM logs: https://drive.google.com/file/d/1x0B8lGqnKEDrgn666CuN3hqUGwD7fcYv/view?us...
Thanks, we will check the logs next week.
> Thanks & regards!
> On 07/29/2021 19:20，Nir Soffer<nsoffer(a)redhat.com> wrote：
>
> On Thu, Jul 29, 2021 at 10:08 AM luwen.zhang <luwen.zhang(a)vinchin.com> wrote:
>
> The problem occurred yesterday, but we waited for more than 20 hours, still 2 disks and in Finalizing state.
>
>
> If the image transfer is "finalizing" it means the image transfer is
> trying to finalize, but the operation could not complete.
>
> In this phase the disk remains locked, and it should not be possible
> to start a new image transfer
> (e.g perform another backup).
>
> Engine and vdsm logs should explain why the image transfer is stuck in
> the finalizing phase.
>
> Can you add detailed instructions on how to reproduce this issue?
>
3 years, 4 months
This year conference community updates
by Sandro Bonazzola
Hi,
as announced we are going to have oVirt online conference coming in
September.
I would like to give usual community updates in and as part of that I'd
like to report about oVirt downstream products (such as Red Hat
Virtualization and Oracle Linux Virtualization Manager) and about companies
using oVirt.
I could just go googling and guessing by e-mail traffic on oVirt users list
but I want to try a different approach this year.
So, if you are shipping an oVirt downstream or if you're using oVirt and
you'd like to get the company nominated during the community reports please
let me know off-list.
Also I'd like to remind that oVirt site has a section dedicated to user
stories, if you want to share yours you can either push it to
https://github.com/oVirt/ovirt-site or get in touch with me off-list to get
help getting your story published.
Thanks,
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>
*Red Hat respects your work life balance. Therefore there is no need to
answer this email out of your office hours.*
3 years, 4 months
ovirt-provider-ovn git repo got broken
by Sandro Bonazzola
Hi,
not sure what happened but ovirt-provider-ovn repo in gerrit (and its
backup on github) now contains a bunch of ovirt-engine tags shipping
ovirt-engine content.
Can you please have a look?
$ git fetch --tags
$ git tag --list |grep ovirt-engine
ovirt-engine-3.0.0_0001
ovirt-engine-3.1.0
ovirt-engine-3.2.0
ovirt-engine-3.2.0-4
ovirt-engine-3.2.1
ovirt-engine-3.2.1-1
ovirt-engine-3.2.2
ovirt-engine-3.3.0
ovirt-engine-3.3.0.1
ovirt-engine-3.3.1
ovirt-engine-3.3.1_beta
ovirt-engine-3.3.1_rc1
ovirt-engine-3.3.2
ovirt-engine-3.3.2_beta1
ovirt-engine-3.3.2_rc1
ovirt-engine-3.3.3_beta1
ovirt-engine-3.3_beta1
ovirt-engine-3.3_rc
ovirt-engine-3.3_rc2
ovirt-engine-3.5.2
ovirt-engine-3.5.2.1
ovirt-engine-3.5.4.1
ovirt-engine-3.6.0_alpha1
ovirt-engine-3.6.2.1
ovirt-engine-4.0.2.3
ovirt-engine-4.1.5
ovirt-engine-4.1.7
ovirt-engine-4.1.7.1
ovirt-engine-4.1.7.3
ovirt-engine-4.1.9.2
ovirt-engine-4.2.0
ovirt-engine-4.2.1.5
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>
*Red Hat respects your work life balance. Therefore there is no need to
answer this email out of your office hours.*
3 years, 4 months
Re: RHV and oVirt CBT issue
by Nir Soffer
On Thu, Jul 29, 2021 at 10:08 AM luwen.zhang <luwen.zhang(a)vinchin.com> wrote:
> The problem occurred yesterday, but we waited for more than 20 hours, still 2 disks and in Finalizing state.
If the image transfer is "finalizing" it means the image transfer is
trying to finalize, but the operation could not complete.
In this phase the disk remains locked, and it should not be possible
to start a new image transfer
(e.g perform another backup).
Engine and vdsm logs should explain why the image transfer is stuck in
the finalizing phase.
Can you add detailed instructions on how to reproduce this issue?
3 years, 4 months
after a regular backup, VM got duplicated disks and disks' status always in Finalizing
by luwen.zhang＠vinchin.com
Version:
oVirt engine: 4.4.7.7-1.el8
oVirt node: 4.4.7
Steps of reproducing the problem:
After a series of regular/successful backup tests by using CBT+imageio API, at one of the backup session beginning, when trying to obtain the VM config and snapshot list (obtain snapshot list can determine the VM disk is in RAW or QCOW2 format) by using `GET vms/<vm-id>/snapshots`, the below error is received.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<fault>
<detail>duplicate key acf1edaa-e950-4c4f-94df-1bd6b3da49c1 (attempted merging values org.ovirt.engine.core.common.businessentities.storage.diskimage@5103046c and org.ovirt.engine.core.common.businessentities.storage.diskimage@d973046c)</detail>
<reason>Operation Failed</reason>
</fault>
After this on the oVirt engine web console, we saw the VM which should only have 1 disk is now showing 2 disks, the disk status always in Finalizing state.
It's been more than 30 hours but the VM still got duplicated disks and still showing Finalizing, there's no way to modify the virtual disk or take snapshots for the VM.
3 years, 4 months
Re: RHV and oVirt CBT issue
by Eyal Shenitzky
+Benny Zlotnik <bzlotnik(a)redhat.com> +Pavel Bar <pbar(a)redhat.com>
On Thu, 29 Jul 2021 at 07:08, luwen.zhang <luwen.zhang(a)vinchin.com> wrote:
>
>
> Dear all,
>
> FYI, after upgrading oVirt engine to 4.4.7.7-1.el8 the chance of
> reproducing this issue is reduced, but it still persists.
>
Can you please open a bug for it and mention the steps you did to reproduce
the problem?
Also, please include all the relevant logs in the bug.
Benny/Pavel, is there a WA for getting out of this issue besides removing
the entry from the DB?
>
> The oVirt node version is also 4.4.7, the virtual disks are stored in
> iSCSI LUN.
>
> The problem occurred yesterday, but we waited for more than 20 hours,
> still 2 disks and in Finalizing state.
>
>
> At this moment, we are unable to check/delete snapshots, or perform
> backups.
>
> Please let us know if we can do anything to help with the debug.
>
> Thanks & regards!
>
> Luwen Zhang
> On 07/27/2021 18:30，luwen.zhang<luwen.zhang(a)vinchin.com>
> <luwen.zhang(a)vinchin.com> wrote：
>
>
> Thank you all for your help, after upgrading oVirt engine the issue is now
> fixed.
>
> Best regards!
> Luwen Zhang
> On 07/27/2021 18:03，Lev Veyde<lveyde(a)redhat.com> <lveyde(a)redhat.com>
> wrote：
>
> Hi Sandro,
>
> Yes, it was released as part of the 0-day fix on July 20, including a new
> appliance build.
>
> Thanks in advance,
>
> On Tue, Jul 27, 2021 at 12:20 PM Sandro Bonazzola <sbonazzo(a)redhat.com>
> wrote:
>
>>
>>
>> Il giorno mar 27 lug 2021 alle ore 10:53 Eyal Shenitzky <
>> eshenitz(a)redhat.com> ha scritto:
>>
>>> As +Michal Skrivanek <mskrivan(a)redhat.com> already replied, this issue
>>> cause by - https://bugzilla.redhat.com/show_bug.cgi?id=1980428.
>>>
>>> It is already fixed in RHV 4.4.7 and on the master branch.
>>> +Sandro Bonazzola <sbonazzo(a)redhat.com>, did we include it on oVirt
>>> 4.4.7 also?
>>>
>>
>> $ git tag --contains 6cd8bc532b0a4a6839ed667c67489256ae9ff5b9
>> ovirt-engine-4.4.7.7
>>
>> It was shipped on July 20th but I don't see an announcement for it. +Lev
>> Veyde <lveyde(a)redhat.com> ?
>>
>>
>>
>>
>>
>>>
>>> On Tue, 27 Jul 2021 at 09:44, luwen.zhang <luwen.zhang(a)vinchin.com>
>>> wrote:
>>>
>>>>
>>>> Dear team could you please help on the below issue?
>>>>
>>>> Thanks Sandro for the advice!
>>>> On 07/27/2021 14:22，Sandro Bonazzola<sbonazzo(a)redhat.com>
>>>> <sbonazzo(a)redhat.com> wrote：
>>>>
>>>> I would recommend to ask this on devel(a)ovirt.org.
>>>> Looping in +Eyal Shenitzky <eshenitz(a)redhat.com>
>>>>
>>>> Il giorno mar 27 lug 2021 alle ore 08:14 luwen.zhang <
>>>> luwen.zhang(a)vinchin.com> ha scritto:
>>>>
>>>>>
>>>>> Dear team,
>>>>>
>>>>> We had implemented CBT support for RHV and oVirt in our new version,
>>>>> but when testing we encountered problems of obtain VM increments and also
>>>>> had abnormal with the VM virtual disks.
>>>>>
>>>>> Our testing environment as below:
>>>>> OS Version: RHEL - 8.4.2105.0 - 3.el8
>>>>> OS Description: oVirt Node 4.4.7
>>>>> Kernel Version: 4.18.0 - 315.el8.x86_64
>>>>> KVM Version: 6.0.0 - 19.el8s
>>>>> LIBVIRT Version: libvirt-7.4.0-1.el8s
>>>>> VDSM Version: vdsm-4.40.70.6-1.el8
>>>>>
>>>>> One of the problem is after a successful backup, the VM which gets
>>>>> only one virtual disk will get 2 identical virtual disks marked as Complete
>>>>> on oVirt engine web console, like shown below.
>>>>>
>>>>> The additional disk might disappear after a while, but before it’s
>>>>> gone there could be a lot of problems with the VM.
>>>>>
>>>>> 1. We cannot perform a new backup, the backup will fail directly.
>>>>>
>>>>> 2. The VM itself will run into exception, we cannot even power it off,
>>>>> and it will also probably get crashed.
>>>>>
>>>>> 3. If the VM was initially powered off, after backing up, we cannot
>>>>> power on the VM.
>>>>>
>>>>> And there could be some other problems related with this, is this
>>>>> normal or there’s something we did wrong?
>>>>>
>>>>> Thank you in advance for your help!
>>>>>
>>>>>
>>>>> *Luwen Zhang* | Product Manager
>>>>>
>>>>> *Mobile*: +8613880424687 | *Skype*: luwen.zhang_cn
>>>>>
>>>>> *WhatsApp*: +8613880424687 | *WeChat*: +8613880424687
>>>>>
>>>>> *Website*: www.vinchin.com
>>>>>
>>>>> *Address*: F5, Block 8, National Information Security Industry Park,
>>>>>
>>>>> No.333 Yunhua Road, Hi-tech Zone, Chengdu, China | *Zip*: 610015
>>>>>
>>>>> *INNOVATIVE VM BACKUP EXPERT*
>>>>>
>>>>> VMware Backup
>>>>> <https://www.vinchin.com/en/hypervisor/vmware-backup.html> | XenServer/Citrix
>>>>> Hypervisor/XCP-ng Backup
>>>>> <https://www.vinchin.com/en/hypervisor/xcp-ng-xenserver-backup.html>
>>>>> | Hyper-V Backup
>>>>> <https://www.vinchin.com/en/hypervisor/hyper-v-backup.html>
>>>>>
>>>>> RHEV/oVirt Backup
>>>>> <https://www.vinchin.com/en/hypervisor/red-hat-virtualization-ovirt-backup...>
>>>>> | OLVM Backup
>>>>> <https://www.vinchin.com/en/hypervisor/oracle-linux-virtualization-manager...>
>>>>> | Sangfor HCI Backup
>>>>> <https://www.vinchin.com/en/hypervisor/sangfor-hci-backup.html> | OpenStack
>>>>> Backup <https://www.vinchin.com/en/hypervisor/openstack-backup.html>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Sandro Bonazzola
>>>>
>>>> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
>>>>
>>>> Red Hat EMEA <https://www.redhat.com/>
>>>>
>>>> sbonazzo(a)redhat.com
>>>> <https://www.redhat.com/>
>>>>
>>>> *Red Hat respects your work life balance. Therefore there is no need to
>>>> answer this email out of your office hours.*
>>>>
>>>>
>>>>
>>>
>>> --
>>> Regards,
>>> Eyal Shenitzky
>>>
>>
>>
>> --
>>
>> Sandro Bonazzola
>>
>> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
>>
>> Red Hat EMEA <https://www.redhat.com/>
>>
>> sbonazzo(a)redhat.com
>> <https://www.redhat.com/>
>>
>> *Red Hat respects your work life balance. Therefore there is no need to
>> answer this email out of your office hours.
>> <https://mojo.redhat.com/docs/DOC-1199578>*
>>
>>
>>
>
> --
>
> Lev Veyde
>
> Senior Software Engineer, RHCE | RHCVA | MCITP
>
> Red Hat Israel
>
> <https://www.redhat.com>
>
> lev(a)redhat.com | lveyde(a)redhat.com
> <https://red.ht/sig>
> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>
>
--
Regards,
Eyal Shenitzky
3 years, 4 months
