- Devel - oVirt List Archives

Error during deployment of self hosted engine oVirt 4.4
by i iordanov 04 Aug '20

04 Aug '20

Hi guys, I am trying to install oVirt 4.4 for testing of the aSPICE and Opaque Android clients and tried to follow this slightly outdated doc: https://www.ovirt.org/documentation/installing_ovirt_as_a_self-hosted_engin… to deploy an all-in-one self-hosted engine using the command-line. I started with a clean CentOS 8 installation, set up an NFS server and tested that mounts work from the local host and other hosts, opened all ports with firewalld to my LAN and localhost (but left firewalld enabled). During the run of hosted-engine --deploy I got the following error: 2020-08-03 23:31:51,426-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:103 TASK [ovirt.hosted_engine_setup : debug] 2020-08-03 23:31:51,827-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:103 server_cpu_dict: {'Intel Nehalem Family': 'Nehalem', 'Secure Intel Nehalem Family': 'Nehalem,+sp ec-ctrl,+ssbd,+md-clear', 'Intel Westmere Family': 'Westmere', 'Secure Intel Westmere Family': 'Westmere,+pcid,+spec-ctrl,+ssbd,+md-clear', 'Intel SandyBridge Family': 'SandyBridge', 'Secure Intel SandyBridge Fa mily': 'SandyBridge,+pcid,+spec-ctrl,+ssbd,+md-clear', 'Intel IvyBridge Family': 'IvyBridge', 'Secure Intel IvyBridge Family': 'IvyBridge,+pcid,+spec-ctrl,+ssbd,+md-clear', 'Intel Haswell Family': 'Haswell-noTSX ', 'Secure Intel Haswell Family': 'Haswell-noTSX,+spec-ctrl,+ssbd,+md-clear', 'Intel Broadwell Family': 'Broadwell-noTSX', 'Secure Intel Broadwell Family': 'Broadwell-noTSX,+spec-ctrl,+ssbd,+md-clear', 'Intel Sk ylake Client Family': 'Skylake-Client,-hle,-rtm', 'Secure Intel Skylake Client Family': 'Skylake-Client,+spec-ctrl,+ssbd,+md-clear,-hle,-rtm', 'Intel Skylake Server Family': 'Skylake-Server,-hle,-rtm', 'Secure I ntel Skylake Server Family': 'Skylake-Server,+spec-ctrl,+ssbd,+md-clear,-hle,-rtm', 'Intel Cascadelake Server Family': 'Cascadelake-Server,-hle,-rtm,+arch-capabilities', 'Secure Intel Cascadelake Server Family': 'Cascadelake-Server,+md-clear,+mds-no,-hle,-rtm,+tsx-ctrl,+arch-capabilities', 'AMD Opteron G4': 'Opteron_G4', 'AMD Opteron G5': 'Opteron_G5', 'AMD EPYC': 'EPYC', 'Secure AMD EPYC': 'EPYC,+ibpb,+virt-ssbd', 'IB M POWER8': 'POWER8', 'IBM POWER9': 'POWER9', 'IBM z114, z196': 'z196-base', 'IBM zBC12, zEC12': 'zEC12-base', 'IBM z13s, z13': 'z13-base', 'IBM z14': 'z14-base'} 2020-08-03 23:31:52,128-0400 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup : Convert CPU model name] 2020-08-03 23:31:52,530-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:103 {'msg': "The task includes an option with an undefined variable. The error was: 'dict object' ha s no attribute ''\n\nThe error appears to be in '/usr/share/ansible/roles/ovirt.hosted_engine_setup/tasks/create_target_vm/01_create_target_hosted_engine_vm.yml': line 105, column 15, but may\nbe elsewhere in th e file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n - debug: var=server_cpu_dict\n ^ here\n\nThere appears to be both 'k=v' shorthand syntax and YAML in this task . Only one syntax may be used.\n", '_ansible_no_log': False} 2020-08-03 23:31:52,630-0400 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:107 fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute ''\n\nThe error appears to be in '/usr/share/ansible/roles/ovirt.hosted_engine_setup/tasks/create_target_vm/01_create_target_hosted_engine_vm.yml': line 105, column 15, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n - debug: var=server_cpu_dict\n ^ here\n\nThere appears to be both 'k=v' shortha nd syntax and YAML in this task. Only one syntax may be used.\n"} 2020-08-03 23:31:52,931-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:103 PLAY RECAP [localhost] : ok: 35 changed: 5 unreachable: 0 skipped: 5 failed: 1 2020-08-03 23:31:53,032-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:215 ansible-playbook rc: 2 2020-08-03 23:31:53,032-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:222 ansible-playbook stdout: 2020-08-03 23:31:53,032-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:225 ansible-playbook stderr: 2020-08-03 23:31:53,033-0400 DEBUG otopi.context context._executeMethod:145 method exception Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/otopi/context.py", line 132, in _executeMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/target_vm.py", line 238, in _closeup r = ah.run() File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/ansible_utils.py", line 229, in run raise RuntimeError(_('Failed executing ansible-playbook')) RuntimeError: Failed executing ansible-playbook 2020-08-03 23:31:53,034-0400 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Closing up': Failed executing ansible-playbook On line 105 of the Ansible config file indicated, there is the following task: - debug: var=server_cpu_dict However, here is a bit more of the file: - name: Parse server CPU list set_fact: server_cpu_dict: "{{ server_cpu_dict | combine({item.split(':')[1]: item.split(':')[3]}) }}" with_items: >- {{ server_cpu_list.json['values']['system_option_value'][0]['value'].split('; ')|list|difference(['']) }} - debug: var=server_cpu_dict - name: Convert CPU model name set_fact: cluster_cpu_model: "{{ server_cpu_dict[cluster_cpu.type] }}" - debug: var=cluster_cpu_model - name: Parse emulated_machine set_fact: emulated_machine: emulated_machine_list.json['values']['system_option_value'][0]['value'].replace( '[','').replace(']','').split(', ')|first Exact version of the package that contains the file in question: ovirt-ansible-hosted-engine-setup-1.1.7-1.el8.noarch Many thanks for your attention! Please cc me when you reply, as I may not be monitoring the mailing list actively. Happy to provide any other information. In case it's relevant, the CPU of the machine: # cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 23 model name : Intel(R) Xeon(R) CPU X5460 @ 3.16GHz stepping : 6 microcode : 0x60f cpu MHz : 2703.129 cache size : 6144 KB physical id : 0 siblings : 4 core id : 0 cpu cores : 4 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm pti tpr_shadow vnmi flexpriority dtherm bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit bogomips : 6318.15 clflush size : 64 cache_alignment : 64 address sizes : 38 bits physical, 48 bits virtual power management: Sincerely, iordan -- The conscious mind has only one thread of execution.

2 2

[VDSM] Trouble merging patch, related to jenkins whitelist or new RHEL CI?
by Nir Soffer 03 Aug '20

03 Aug '20

I could not merge: https://gerrit.ovirt.org/c/109402/ Although it was approved and verified, and got +1 from Continuous Integration. There is a new "RHEL-Continuous-Integration", which does not run with this change, even when I trigger the tests manually with "ci test". I tried to add +1 for "RHEL-Continuous-Integration" but this is not possible, I see: RHEL-Continuous-Integration You don't have permission to edit this label. So finally I downloaded the patch and pushed it manually. I think this patch will fix the problem: https://gerrit.ovirt.org/#/c/110576/ But I need CI experts to review this. Since we have 50 projects that need this, this probably should be fixed elsewhere, and inherited by all projects. I know that Germano was not able to trigger tests because he was missing in the jenkins whitelist, but this was fixed last week. Germano, maybe just to check that everything works for you, you can post some trivial patch? Nir

4 9

please i'm facing a with making el8
by Amit Chaudhary 30 Jul '20

30 Jul '20

ovirt.zip <https://drive.google.com/file/d/1I1Kcbnmi3IIE1taUsWEe2uQaWFWvERyE/view?usp=…> i have attached my ovirt zip. the issue im facing is that the kickstart is not generated for el8 and im not able to figure out how to use .j2 file that is being generated. i've manage to frankenstein a kickstart file that is in the zip with all the settings but its also giving error while compressing (dracut module 'livenet' cannot be found or installed) while if i search installed modules of dracut i can see livenet is installed. see see my zip it have logs of last compilation also. sir/mam it would be a great help for my college final year project project on hci and my research paper. please sir/mam please help me Amit Chaudhary +919650596355 -- *CONFIDENTIALITY NOTICE* This communication (including any attachment/s) is intended only for the use of the addressee(s) and contains information that is PRIVILEGED AND CONFIDENTIAL. Unauthorized reading, dissemination, distribution, or copying of this communication is prohibited. * DISCLAIMER* Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the University. WARNING:- This communication, including any attachments, may not be free of viruses, interceptions or interference, and may not be compatible with your systems. You should carry out your own virus checks before opening any attachment to this e-mail. The sender of this e-mail and Galgotias University shall not be liable for any damage that you may sustain as a result of viruses, incompleteness of this message, a delay in receipt of this message or computer problems experienced.

2 1

OST switching back to CirrOS
by Michal Skrivanek 30 Jul '20

30 Jul '20

Hi, FYI we’ve just switched back from CentOS to CirrOS VMs in OST basic suite, saving 900MB of data transferred all the time (and copied back and forth during test) as well as lower memory requirements (down to 128MB from 384MB, times 3 VMs). For a good measure a memory hot unplug test was added too. Let me know if you see anything unusual, other than faster run times. Thanks, michal

2 2

Error while executing action New NFS Storage Domain: Unexpected exception
by Dana Elfassy 30 Jul '20

30 Jul '20

Hi, When I'm trying to add a storage domain via the webadmin I'm getting the error message that's on the subject. in the vdsm.log I see this error: 2020-07-29 04:53:48,045-0400 ERROR (check/loop) [storage.Monitor] Error checking path /rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata (monitor:525) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 523, in _pathChecked delay = result.delay() File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line 391, in delay raise exception.MiscFileReadException(self.path, self.rc, self.err) vdsm.storage.exception.MiscFileReadException: Internal file read failure: ('/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata', 1, bytearray(b"/usr/bin/dd: failed to open \'/rhev/data-center/mnt/vserver-spider.eng.lab.tlv.redhat.com:_pub_delfassy_nfs__42/17093147-8e4b-4161-9ae3-4d4606b140df/dom_md/metadata\': No such file or directory\n")) when looking at the directory I do see that metadata was created under this directory, with the following content: [delfassy@delfassy mnt]$ cd new_nfs/cddff7bb-bb0a-4fc8-8ba4-023be91d822d/dom_md/ [delfassy@delfassy dom_md]$ less metadata CLASS=Data DESCRIPTION=spider IOOPTIMEOUTSEC=10 LEASERETRIES=3 LEASETIMESEC=60 LOCKPOLICY= LOCKRENEWALINTERVALSEC=5 POOL_UUID= REMOTE_PATH=vserver-spider.eng.lab.tlv.redhat.com:/pub/delfassy/new_nfs ROLE=Regular SDUUID=cddff7bb-bb0a-4fc8-8ba4-023be91d822d TYPE=NFS VERSION=4 _SHA_CKSUM=f4953016c94f205127b00a5b10f0e7d5d9cf24f7 The permissions on the directory are: drwxrwxrwx. 3 vdsm kvm 4096 Jul 29 11:49 new_mnt Are there any other configurations I'm missing? Any idea how this issue can be resolved? Thanks, Dana

2 3

Re: Issue with ovirt 4.4 after doing some incremental backups.
by Eyal Shenitzky 29 Jul '20

29 Jul '20

According to the vdsm.log it seems that the host reports checkpoint 'ef0dfe55-c08c-4d9e-ad32-d6b6d5cbdac6' as defined checkpoint for the VM - 7-28 16:24:06,488+0200 INFO (jsonrpc/3) [api.virt] FINISH list_checkpoints return={'result': ['cd078706-84c0-4370-a6ec-654ccd6a21aa', '906fe1fd-4e81-473c-9582-de693985f462', 'd20de661-fcc1-43ff-86a3-6294a0a94e1b', '805b447e-d033-4e b1-905a-d287ca50c85a', '9ae68264-a0c9-4f75-9ffa-20c40323abe9', '8664da63-6d92-44b8-a690-04e17afb483a', 'fb6fa799-3b6f-44fa-a60e-c66ce8c76e35', 'f9f53ee7-2a8c-4dfc-a669-2dd788361eae', '7a0f37db-02bc-49c4-bfea-a03d3474f1f8', '95d8d5c7-55fc- 4c39-8a72-fc3aaec72c39', 'aa71a6e9-174b-4d8d-a2da-1a12d5e2ab2e', 'ac833f22-c8fa-4925-becc-dcf4dac81d3f', '7fc9afd1-076f-4985-b412-4ea029fd2af7', 'ae299a18-c4b0-46c4-81e1-f9583e6954f6', 'cee97ef9-6e7c-4518-91b1-dfa3b467f5c4', '51b3bf23-3cc 3-4a16-b6d9-942c6e4f7573', '0f376950-a25e-477a-93b8-d1d0ffbff716', 'ef0dfe55-c08c-4d9e-ad32-d6b6d5cbdac6'], 'status': {'code': 0, 'message': 'Done'}} from=::ffff:10.40.1.66,56474, flow_id=32ba9d9d-32c2-4652-ac01-895e92123323, vmId=116aa6e b-31a1-43db-9b1e-ad6e32fb9260 (api:54) It means as Nir said that the host assumes the corresponding bitmap is also exists in QEMU. The logs only contain a short period of time so I cannot tell if some bitmap destructive operation occurs on the VM (Live merge, cold snapshot creation etc..) Peter, is there any other log that may assist? On Wed, 29 Jul 2020 at 11:28, Łukasz Kołaciński <l.kolacinski(a)storware.eu> wrote: > Hello, > I am sending logs from vdsm, maybe they will help. I cannot find useful > libvirt logs and file libvirt.log doesn't exists. > > Best Regards, > > Łukasz Kołaciński > > Junior Java Developer > > e-mail: l.kolacinski(a)storware.eu > <m.helbert(a)storware.eu> > > > > > *[image: STORWARE]* <http://www.storware.eu/> > > > > *ul. Leszno 8/44 01-192 Warszawa www.storware.eu > <https://www.storware.eu/>* > > *[image: facebook]* <https://www.facebook.com/storware> > > *[image: twitter]* <https://twitter.com/storware> > > *[image: linkedin]* <https://www.linkedin.com/company/storware> > > *[image: Storware_Stopka_09]* > <https://www.youtube.com/channel/UCKvLitYPyAplBctXibFWrkw> > > > > *Storware Spółka z o.o. nr wpisu do ewidencji KRS dla M.St. Warszawa > 000510131* *, NIP 5213672602.** Wiadomość ta jest przeznaczona jedynie > dla osoby lub podmiotu, który jest jej adresatem i może zawierać poufne > i/lub uprzywilejowane informacje. Zakazane jest jakiekolwiek przeglądanie, > przesyłanie, rozpowszechnianie lub inne wykorzystanie tych informacji lub > podjęcie jakichkolwiek działań odnośnie tych informacji przez osoby lub > podmioty inne niż zamierzony adresat. Jeżeli Państwo otrzymali przez > pomyłkę tę informację prosimy o poinformowanie o tym nadawcy i usunięcie > tej wiadomości z wszelkich komputerów. **This message is intended only > for the person or entity to which it is addressed and may contain > confidential and/or privileged material. Any review, retransmission, > dissemination or other use of, or taking of any action in reliance upon, > this information by persons or entities other than the intended recipient > is prohibited. If you have received this message in error, please contact > the sender and remove the material from all of your computer systems.* > > ------------------------------ > *Od:* Nir Soffer <nsoffer(a)redhat.com> > *Wysłane:* środa, 29 lipca 2020 00:31 > *Do:* Łukasz Kołaciński <l.kolacinski(a)storware.eu> > *DW:* Eyal Shenitzky <eshenitz(a)redhat.com>; Mateusz Maziarz < > m.maziarz(a)storware.eu>; Marcin Kubacki <m.kubacki(a)storware.eu>; Peter > Krempa <pkrempa(a)redhat.com>; devel <devel(a)ovirt.org> > *Temat:* Re: Issue with ovirt 4.4 after doing some incremental backups. > > On Tue, Jul 28, 2020 at 4:05 PM Łukasz Kołaciński < > l.kolacinski(a)storware.eu> wrote: > > Hello, > > > Hi Łukasz > > I moved the discussion to devel(a)ovirt.org since it is more appropriate > for this issue. > > After doing a few vm backups, something breaks and I am unable to perform > any operations. I cannot do incremental backups and even full backups > doesn't work. I have this issue third time. I don't know how to fix this so > I am currently making new vms for testing purposes > > VDSM ovirt44-h2.storware.local command StartVmBackupVDS failed: Backup > Error: {'vm_id': '116aa6eb-31a1-43db-9b1e-ad6e32fb9260', 'backup': > <vdsm.virt.backup.BackupConfig object at 0x7f42602bba20>, 'reason': "Error > starting backup: internal error: unable to execute QEMU command > 'transaction': Dirty bitmap 'ef0dfe55-c08c-4d9e-ad32-d6b6d5cbdac6' not > found"} > > This means that libvirt cannot find the dirty bitmap when starting the > backup. > > When we start a backup, we get the list of checkpoints from libvirt and we > redefine all checkpoints. We assume that all redefined checkpoints have > a bitmap in qemu at the time of the redefine. > > Peter, is this assumption correct? > > If libvirt and engine agree on the existing checkpoints, we start the > backup. > In this case one of the bitmaps was missing, so the backup failed. > > We know about some flows that may cause loss of the bitmaps: > - copying disks (bitmaps are not copied yet) > - live storage migration (it copy the disks) > - deleting snapshots > - live migration may cause this, not tested yet > - unclean shutdown of the vm > - storage is not accessible when vm is terminated > > Did you do any of these operations on the tested vm? > > Some of the issues can never be fixed, like unclean shutdown or storage > issue when qemu try to write the bitmaps to disk. So your backup > application > must be able to recover from this error. > > However oVirt does not provide a useful error that enables recovery. We > have > a special error when full backup is required, but it seem that this error > is > not returned in this case, and instead we return internal error. > > Also since you cannot do a full backup after this error, I guess that > engine > did not delete the checkpoint with the missing bitmap. This is not > surprising > since the error returned from vdsm is a generic error (BackupError), so > engine > cannot tell what is the reason for the failure. > > Did you check the backup events? What was the backup completion event? > > See this example how to get backup events: > > https://github.com/oVirt/ovirt-engine-sdk/blob/4a143351fcd3cdb0df8c50888931… > > https://github.com/oVirt/ovirt-engine-sdk/blob/4a143351fcd3cdb0df8c50888931… > > Eyal, looking at the API docs: > http://ovirt.github.io/ovirt-engine-api-model/master/#types/event > > Event code is an integer. This is not usable for detecting errors since > the value > is not part of the API. We need an enum like: > > http://ovirt.github.io/ovirt-engine-api-model/4.4/#types/image_transfer_pha… > > Peter, do we have a specific error code in libvirt about missing bitmap? > we need > this to pass useful error to engine, and engine needs this error to pass > useful > error to the user. > > The error seen here is generated by: > > flags = libvirt.VIR_DOMAIN_BACKUP_BEGIN_REUSE_EXTERNAL > try: > dom.backupBegin(backup_xml, checkpoint_xml, flags=flags) > except libvirt.libvirtError as e: > raise exception.BackupError( > reason="Error starting backup: {}".format(e), > vm_id=vm.id, > backup=backup_cfg) > > Looking in the documentation: > https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBackupBegin > > There are no specified errors so we cannot detect the reason for the > failure > and return meaningful error to our caller. > > Peter, how do you suggest to recover from internal errors? how can we > tell if this temporary error that can succeed in the next attempt, or an > error > that requires starting from full backup? > > Nir > -- Regards, Eyal Shenitzky

2 1

Re: Issue with ovirt 4.4 after doing some incremental backups.
by Eyal Shenitzky 29 Jul '20

29 Jul '20

Libvirt logs need to be activated on the host in order to collect them, please try: - systemctl stop vdsmd - systemctl stop libvirtd - Under /etc/libvirt/libvirtd.conf add: - log_filters="1:qemu 1:libvirt 4:object 4:json 4:event 1:util" - log_outputs="1:file:/var/log/libvirt/libvirtd.log" - systemctl start libvirtd - systemctl start vdsmd On Wed, 29 Jul 2020 at 11:28, Łukasz Kołaciński <l.kolacinski(a)storware.eu> wrote: > Hello, > I am sending logs from vdsm, maybe they will help. I cannot find useful > libvirt logs and file libvirt.log doesn't exists. > > Best Regards, > > Łukasz Kołaciński > > Junior Java Developer > > e-mail: l.kolacinski(a)storware.eu > <m.helbert(a)storware.eu> > > > > > *[image: STORWARE]* <http://www.storware.eu/> > > > > *ul. Leszno 8/44 01-192 Warszawa www.storware.eu > <https://www.storware.eu/>* > > *[image: facebook]* <https://www.facebook.com/storware> > > *[image: twitter]* <https://twitter.com/storware> > > *[image: linkedin]* <https://www.linkedin.com/company/storware> > > *[image: Storware_Stopka_09]* > <https://www.youtube.com/channel/UCKvLitYPyAplBctXibFWrkw> > > > > *Storware Spółka z o.o. nr wpisu do ewidencji KRS dla M.St. Warszawa > 000510131* *, NIP 5213672602.** Wiadomość ta jest przeznaczona jedynie > dla osoby lub podmiotu, który jest jej adresatem i może zawierać poufne > i/lub uprzywilejowane informacje. Zakazane jest jakiekolwiek przeglądanie, > przesyłanie, rozpowszechnianie lub inne wykorzystanie tych informacji lub > podjęcie jakichkolwiek działań odnośnie tych informacji przez osoby lub > podmioty inne niż zamierzony adresat. Jeżeli Państwo otrzymali przez > pomyłkę tę informację prosimy o poinformowanie o tym nadawcy i usunięcie > tej wiadomości z wszelkich komputerów. **This message is intended only > for the person or entity to which it is addressed and may contain > confidential and/or privileged material. Any review, retransmission, > dissemination or other use of, or taking of any action in reliance upon, > this information by persons or entities other than the intended recipient > is prohibited. If you have received this message in error, please contact > the sender and remove the material from all of your computer systems.* > > ------------------------------ > *Od:* Nir Soffer <nsoffer(a)redhat.com> > *Wysłane:* środa, 29 lipca 2020 00:31 > *Do:* Łukasz Kołaciński <l.kolacinski(a)storware.eu> > *DW:* Eyal Shenitzky <eshenitz(a)redhat.com>; Mateusz Maziarz < > m.maziarz(a)storware.eu>; Marcin Kubacki <m.kubacki(a)storware.eu>; Peter > Krempa <pkrempa(a)redhat.com>; devel <devel(a)ovirt.org> > *Temat:* Re: Issue with ovirt 4.4 after doing some incremental backups. > > On Tue, Jul 28, 2020 at 4:05 PM Łukasz Kołaciński < > l.kolacinski(a)storware.eu> wrote: > > Hello, > > > Hi Łukasz > > I moved the discussion to devel(a)ovirt.org since it is more appropriate > for this issue. > > After doing a few vm backups, something breaks and I am unable to perform > any operations. I cannot do incremental backups and even full backups > doesn't work. I have this issue third time. I don't know how to fix this so > I am currently making new vms for testing purposes > > VDSM ovirt44-h2.storware.local command StartVmBackupVDS failed: Backup > Error: {'vm_id': '116aa6eb-31a1-43db-9b1e-ad6e32fb9260', 'backup': > <vdsm.virt.backup.BackupConfig object at 0x7f42602bba20>, 'reason': "Error > starting backup: internal error: unable to execute QEMU command > 'transaction': Dirty bitmap 'ef0dfe55-c08c-4d9e-ad32-d6b6d5cbdac6' not > found"} > > This means that libvirt cannot find the dirty bitmap when starting the > backup. > > When we start a backup, we get the list of checkpoints from libvirt and we > redefine all checkpoints. We assume that all redefined checkpoints have > a bitmap in qemu at the time of the redefine. > > Peter, is this assumption correct? > > If libvirt and engine agree on the existing checkpoints, we start the > backup. > In this case one of the bitmaps was missing, so the backup failed. > > We know about some flows that may cause loss of the bitmaps: > - copying disks (bitmaps are not copied yet) > - live storage migration (it copy the disks) > - deleting snapshots > - live migration may cause this, not tested yet > - unclean shutdown of the vm > - storage is not accessible when vm is terminated > > Did you do any of these operations on the tested vm? > > Some of the issues can never be fixed, like unclean shutdown or storage > issue when qemu try to write the bitmaps to disk. So your backup > application > must be able to recover from this error. > > However oVirt does not provide a useful error that enables recovery. We > have > a special error when full backup is required, but it seem that this error > is > not returned in this case, and instead we return internal error. > > Also since you cannot do a full backup after this error, I guess that > engine > did not delete the checkpoint with the missing bitmap. This is not > surprising > since the error returned from vdsm is a generic error (BackupError), so > engine > cannot tell what is the reason for the failure. > > Did you check the backup events? What was the backup completion event? > > See this example how to get backup events: > > https://github.com/oVirt/ovirt-engine-sdk/blob/4a143351fcd3cdb0df8c50888931… > > https://github.com/oVirt/ovirt-engine-sdk/blob/4a143351fcd3cdb0df8c50888931… > > Eyal, looking at the API docs: > http://ovirt.github.io/ovirt-engine-api-model/master/#types/event > > Event code is an integer. This is not usable for detecting errors since > the value > is not part of the API. We need an enum like: > > http://ovirt.github.io/ovirt-engine-api-model/4.4/#types/image_transfer_pha… > > Peter, do we have a specific error code in libvirt about missing bitmap? > we need > this to pass useful error to engine, and engine needs this error to pass > useful > error to the user. > > The error seen here is generated by: > > flags = libvirt.VIR_DOMAIN_BACKUP_BEGIN_REUSE_EXTERNAL > try: > dom.backupBegin(backup_xml, checkpoint_xml, flags=flags) > except libvirt.libvirtError as e: > raise exception.BackupError( > reason="Error starting backup: {}".format(e), > vm_id=vm.id, > backup=backup_cfg) > > Looking in the documentation: > https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBackupBegin > > There are no specified errors so we cannot detect the reason for the > failure > and return meaningful error to our caller. > > Peter, how do you suggest to recover from internal errors? how can we > tell if this temporary error that can succeed in the next attempt, or an > error > that requires starting from full backup? > > Nir > -- Regards, Eyal Shenitzky

2 1

Re: Issue with ovirt 4.4 after doing some incremental backups.
by Nir Soffer 29 Jul '20

29 Jul '20

On Tue, Jul 28, 2020 at 4:05 PM Łukasz Kołaciński <l.kolacinski(a)storware.eu> wrote: > Hello, > Hi Łukasz I moved the discussion to devel(a)ovirt.org since it is more appropriate for this issue. After doing a few vm backups, something breaks and I am unable to perform > any operations. I cannot do incremental backups and even full backups > doesn't work. I have this issue third time. I don't know how to fix this so > I am currently making new vms for testing purposes > > VDSM ovirt44-h2.storware.local command StartVmBackupVDS failed: Backup > Error: {'vm_id': '116aa6eb-31a1-43db-9b1e-ad6e32fb9260', 'backup': > <vdsm.virt.backup.BackupConfig object at 0x7f42602bba20>, 'reason': "Error > starting backup: internal error: unable to execute QEMU command > 'transaction': Dirty bitmap 'ef0dfe55-c08c-4d9e-ad32-d6b6d5cbdac6' not > found"} > > This means that libvirt cannot find the dirty bitmap when starting the backup. When we start a backup, we get the list of checkpoints from libvirt and we redefine all checkpoints. We assume that all redefined checkpoints have a bitmap in qemu at the time of the redefine. Peter, is this assumption correct? If libvirt and engine agree on the existing checkpoints, we start the backup. In this case one of the bitmaps was missing, so the backup failed. We know about some flows that may cause loss of the bitmaps: - copying disks (bitmaps are not copied yet) - live storage migration (it copy the disks) - deleting snapshots - live migration may cause this, not tested yet - unclean shutdown of the vm - storage is not accessible when vm is terminated Did you do any of these operations on the tested vm? Some of the issues can never be fixed, like unclean shutdown or storage issue when qemu try to write the bitmaps to disk. So your backup application must be able to recover from this error. However oVirt does not provide a useful error that enables recovery. We have a special error when full backup is required, but it seem that this error is not returned in this case, and instead we return internal error. Also since you cannot do a full backup after this error, I guess that engine did not delete the checkpoint with the missing bitmap. This is not surprising since the error returned from vdsm is a generic error (BackupError), so engine cannot tell what is the reason for the failure. Did you check the backup events? What was the backup completion event? See this example how to get backup events: https://github.com/oVirt/ovirt-engine-sdk/blob/4a143351fcd3cdb0df8c50888931… https://github.com/oVirt/ovirt-engine-sdk/blob/4a143351fcd3cdb0df8c50888931… Eyal, looking at the API docs: http://ovirt.github.io/ovirt-engine-api-model/master/#types/event Event code is an integer. This is not usable for detecting errors since the value is not part of the API. We need an enum like: http://ovirt.github.io/ovirt-engine-api-model/4.4/#types/image_transfer_pha… Peter, do we have a specific error code in libvirt about missing bitmap? we need this to pass useful error to engine, and engine needs this error to pass useful error to the user. The error seen here is generated by: flags = libvirt.VIR_DOMAIN_BACKUP_BEGIN_REUSE_EXTERNAL try: dom.backupBegin(backup_xml, checkpoint_xml, flags=flags) except libvirt.libvirtError as e: raise exception.BackupError( reason="Error starting backup: {}".format(e), vm_id=vm.id, backup=backup_cfg) Looking in the documentation: https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBackupBegin There are no specified errors so we cannot detect the reason for the failure and return meaningful error to our caller. Peter, how do you suggest to recover from internal errors? how can we tell if this temporary error that can succeed in the next attempt, or an error that requires starting from full backup? Nir

3 2

ovirt-engine has been tagged (ovirt-engine-4.4.2)
by Tal Nisan 29 Jul '20

29 Jul '20

3 2

ovirt-engine has been tagged (ovirt-engine-4.3.11.3)
by Tal Nisan 28 Jul '20

28 Jul '20

1 0