Bongiorno Gianluca, I am entering the data in bugzilla right now... Logs: There is a problem, I have not found any log evidence of this export and import, I don't even know which component is actually doing this. For 'foreign' imports like VMware and other *.vmdk producing hypervisors, VDSM seems to be doing the job and leaves logs in /var/log/vdsm/import, describing the format conversion using virt-v2v. For these 'domestic' imports, there is no such log while the imports seem to complete, but with little or no disk data. I am just validating that at least the transfer between farms or the export/import via the deprecated export domain still works. It's quite messy and unintuitive as buttons to detach an export domain don't enable, until you have found a way to put them into maintenance, which for some reason has to be done from the datacenter etc. It's still importing on the target as I write this... Yes, I guess some validation by others would be lovely, just make sure you don't delete any VM you might want back later ;-)

On Thu, Jul 30, 2020 at 5:10 PM <thomas(a)hoberg.net&gt; wrote: OVA export/import[1] is not a backup solution. This is a way to move a VM from one environment to another. For this we have a much better way, detach data domain and import it in another environment. If 4.4 we have a real backup API (tech preview). This will be used by backup vendors to provide complete backup solution, but we have basic backup capabilities included in the sdk examples. See the devconf 2020 talk[4] for more info. The examples deal only with backing up and restoring disks. Getting the vm configuration and creating a vm from the configuration is not included yet. Here is how you can do full back of a running vm disks using backup_vm.py[2] example: $ ./backup_vm.py full --engine-url https://engine3 \ --username admin@internal \ --password-file engine3-password \ --cafile engine3.pem \ --backup-dir backups \ ed2e2c59-36d3-41e2-ac7e-f4d33eb69ad4 [ 0.0 ] Starting full backup for VM ed2e2c59-36d3-41e2-ac7e-f4d33eb69ad4 [ 1.7 ] Waiting until backup 4ac0ae37-b1dd-4a8d-ad13-1ddb51e70d3e is ready [ 3.8 ] Created checkpoint '3648484c-61dc-4680-a7b4-0256544fa21d' (to use in --from-checkpoint-uuid for the next incremental backup) [ 3.8 ] Creating image transfer for disk 419b83e7-a6b5-4445-9324-a85e27b134d2 [ 5.1 ] Image transfer c659437e-6242-42b9-9293-8ccf10321261 is ready Formatting 'backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008011652.full.qcow2', fmt=qcow2 size=107374182400 cluster_size=65536 lazy_refcounts=off refcount_bits=16 [ 100.00% ] 100.00 GiB, 0.39 seconds, 258.06 GiB/s [ 5.5 ] Finalizing image transfer [ 7.5 ] Creating image transfer for disk 58daea80-1229-4c6b-b33c-1a4e568c8ad7 [ 8.7 ] Image transfer 78254cef-5f22-4161-8dbd-010420730a7f is ready Formatting 'backups/58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008011652.full.qcow2', fmt=qcow2 size=6442450944 cluster_size=65536 lazy_refcounts=off refcount_bits=16 [ 100.00% ] 6.00 GiB, 10.33 seconds, 594.54 MiB/s [ 19.0 ] Finalizing image transfer [ 23.3 ] Full backup completed successfully Note that we did not create any snapshot, backup provides a consistent view on all disks at the time of the backup, but you must have qemu-guest-agent installed in the VM. This is the same consistency you get from snapshot without memory. $ ls -lh backups/ total 2.3G -rw-r--r--. 1 nsoffer nsoffer 52M Aug 1 16:52 419b83e7-a6b5-4445-9324-a85e27b134d2.202008011652.full.qcow2 -rw-r--r--. 1 nsoffer nsoffer 2.2G Aug 1 16:52 58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008011652.full.qcow2 In this example we detected that the 100 GiB disk 419b83e7-a6b5-4445-9324-a85e27b134d2.202008011652 is mostly empty (mounted file system, no data written yet) so we could download only the allocated blocks. $ qemu-img info backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008011652.full.qcow2 image: backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008011652.full.qcow2 file format: qcow2 virtual size: 100 GiB (107374182400 bytes) disk size: 51 MiB cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false The second disk is a Fedora 32 OS disk, so we downloaded only the allocated blocks, about 2.2 GiB. $ qemu-img info backups/58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008011652.full.qcow2 image: backups/58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008011652.full.qcow2 file format: qcow2 virtual size: 6 GiB (6442450944 bytes) disk size: 2.17 GiB cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false To restore the disks you can use upload_disk.py[3] example: $ ./upload_disk.py --engine-url https://engine3 \ --username admin@internal \ --password-file engine3-password \ --cafile engine3.pem \ --sd-name nfs1 \ --disk-format raw \ --disk-sparse \ backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008011652.full.qcow2 Checking image... Image format: qcow2 Disk format: raw Disk content type: data Disk provisioned size: 107374182400 Disk initial size: 107374182400 Disk name: 419b83e7-a6b5-4445-9324-a85e27b134d2.202008011652.full.raw Disk backup: False Connecting... Creating disk... Disk ID: 5c9465d6-6437-4ece-bcaa-eec6b2d00a9d Creating image transfer... Transfer ID: dfcce09b-1f65-45e6-8ea5-41dff9368b15 Transfer host name: host4 Uploading image... [ 100.00% ] 100.00 GiB, 0.75 seconds, 133.31 GiB/s Finalizing image transfer... Upload completed successfully Again we upload only the allocated blocks, so the upload is very quick. Note that we upload the qcow2 disk to raw sparse disk. We support on-the-fly image format conversion. $ ./upload_disk.py --engine-url https://engine3 \ --username admin@internal \ --password-file engine3-password \ --cafile engine3.pem \ --sd-name nfs1 \ --disk-format qcow2 \ --disk-sparse \ backups/58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008011652.full.qcow2 Checking image... Image format: qcow2 Disk format: cow Disk content type: data Disk provisioned size: 6442450944 Disk initial size: 2332491776 Disk name: 58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008011652.full.qcow2 Disk backup: False Connecting... Creating disk... Disk ID: 5cf45aa6-6825-4aa5-adf1-4eb70809c37f Creating image transfer... Transfer ID: ed4046c3-63f9-445b-895b-e964dbcfc134 Transfer host name: host4 Uploading image... [ 100.00% ] 6.00 GiB, 5.62 seconds, 1.07 GiB/s Finalizing image transfer... Upload completed successfully This time we uploaded to a qcow2 format. You can create a new vm from these disks with the same configuration of the previous vm. I hope that future versions will backup and restore complete vms. This also supports incremental backups, but it is not suitable for production, there are too many rough edges that need to be smoothed. We also support uploading disks from ova file. Lets create one with compressed qcow2 format. This creates smaller ova files optimized for transfering to other machines. $ mkdir ova $ qemu-img convert -f qcow2 -O qcow2 -c backups/58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008011652.full.qcow2 ova/disk1.qcow2 $ qemu-img convert -f qcow2 -O qcow2 -c backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008011652.full.qcow2 ova/disk2.qcow2 $ ls -lh ova total 1.3G -rw-r--r--. 1 nsoffer nsoffer 1.3G Aug 1 17:20 disk1.qcow2 -rw-r--r--. 1 nsoffer nsoffer 576K Aug 1 17:20 disk2.qcow2 $ echo "this is not a real ovf" > ova/vm.ovf $ tar cvf test.ova -C ova vm.ovf disk1.qcow2 disk2.qcow2 vm.ovf disk1.qcow2 disk2.qcow2 $ ls -lhs test.ova 1.3G -rw-rw-r--. 1 nsoffer nsoffer 1.3G Aug 1 17:24 test.ova This is not a valid ovf, but good enough to show the upload capability: $ for disk in disk1.qcow2 disk2.qcow2; do ./upload_from_ova.py --engine-url https://engine3 \ --username admin@internal \ --password-file engine3-password \ --cafile engine3.pem \ --sd-name nfs1 \ --disk-format raw \ --disk-sparse \ --ova-disk-name $disk \ test.ova done Checking image... Image format: qcow2 Disk name: disk1.raw Disk format: raw Disk provisioned size: 6442450944 Disk initial size: None Disk backup: None Creating disk... Disk ID: acb7bc75-f6e9-49c8-a158-d7d581c108b4 Creating image transfer... Transfer ID: fca2d537-0f91-4033-8b35-ef0084191b4e Transfer host name: host4 Uploading disk disk1.qcow2 from test.ova... [ 100.00% ] 6.00 GiB, 6.88 seconds, 893.04 MiB/s Finalizing image transfer... Upload completed successfully Checking image... Image format: qcow2 Disk name: disk2.raw Disk format: raw Disk provisioned size: 107374182400 Disk initial size: None Disk backup: None Creating disk... Disk ID: ec57e31a-a531-48c2-8447-6468087bb4ab Creating image transfer... Transfer ID: 3aa23287-6ec4-4571-8623-7724c4a36341 Transfer host name: host4 Uploading disk disk2.qcow2 from test.ova... [ 100.00% ] 100.00 GiB, 0.49 seconds, 204.34 GiB/s Finalizing image transfer... Upload completed successfully To try this you need to install python3-ovirt-enigne-sdk4. The examples scripts are in: /usr/share/doc/python3-ovirt-engine-sdk4/examples/ We have automated tests using the upload/download/backup example scripts, so they should work. Nir [1] https://www.youtube.com/watch?v=VtuwWL6SMYk [2] https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/backup... [3] https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/upload... [4] https://www.youtube.com/watch?v=foyi1UyadEc

On Tue, Aug 4, 2020 at 7:37 PM <thomas(a)hoberg.net&gt; wrote: This is good, you should not use export domain at this point. We have better replacement (see my other mail). I think you already found that OVA are not what you think they are. They work for exporting VMs from the same hypervisor and back, unless you have a tool that know how to convert OVA from one hypervisor to another like virt-v2v. True, ease of use is important. But if you are going to do this a lost, scripting the operation is important, and oVirt has a very powerful API/SDK. What you say is basically that having a backup is useful :-) How are you going to use the OVA on a bare metal machine? Why do you need the desktop hypervisor? I would like to hear more about this use case. And if you need one, why not use something based on KVM (like virt-manager) so disks from oVirt can work without any change? This will make it easy to move from oVit to desktop hypervisor and back with relatively little effort. Thanks Thomas, this is very useful feedback! Nir

Here is the ticket: https://bugzilla.redhat.com/show_bug.cgi?id=1862115

Wow, it seems a good find for a potential reason! I'd totally subscribe to the cause, if it only affected my Atom HCI clusters, who have had timing issues so bad, even the initial 3 node HCI setup had to be run on a 5 GHz Kaby Lake to succeed, because my J5005 Atoms evidently were too slow. But I see the very same effect also on another farm, not using quite the newest generation of hardware, but even a 28 core Broadwell Xeon, so if timing is the issue, somebody from QA needs a serious performance review. Did you get a chance to test the OVA export/import cycle?

I confirm, I get "qemu-img: /dev/loop0: error while converting qcow2: Could not open device: Permission denied", too. You've nailed it and Nir has pasted your log into the bug report 1862115.

HI Gianluca, I have had a look at the change and it's a single line of code added on the hosted-engine. I'll verify that it's working 4.3 and will make a note of checking it with engine upgrade, which for now seems the less troublesome approach. Hopefully it will get integrated/backported also with 4.3 or I'll have a chance to validate 4.4.

Keep in mind that oVirt relies on OS components like qemu and if they do fail it is not oVirt's fault. Also, OVA export/import is not the most popular approach for backup/restore in Virtualization Environments, so this bug should not be a reason to 'drop it like a hot potato' . Yet, I agree that QA tests should have catched it in the first place, but here comes the community part - to assist the devs with finding the test cases we all need. Best Regards, Strahil Nikolov На 1 август 2020 г. 12:51:37 GMT+03:00, thomas(a)hoberg.net написа: >Unfortunately that means OVA export/import doesn't seem to be a QA test >case for oVirt releases... which would make me want to drop it like a >hot potato, if there was any alternative at this point. > >I guess my biggest mistake was to assume that oVirt was like CentOS. >_______________________________________________ >Users mailing list -- users(a)ovirt.org >To unsubscribe send an email to users-leave(a)ovirt.org >Privacy Statement: https://www.ovirt.org/privacy-policy.html >oVirt Code of Conduct: >https://www.ovirt.org/community/about/community-guidelines/ >List Archives: >https://lists.ovirt.org/archives/list/users@ovirt.org/message/ODJEYA6RW7NHFQ2YO3GW2KPL2SUMIHQS/