On Thu, Jul 30, 2020 at 2:37 PM <thomas(a)hoberg.net&gt; wrote: I think the best way to get appropriate attention is to open a bugzilla for it, providing information and logs.... In the meantime I can test in a 4.3.10 environment of mine if I have the same problem. HIH, Gianluca

Here is the ticket: https://bugzilla.redhat.com/show_bug.cgi?id=1862115

On Thu, Jul 30, 2020 at 4:04 PM <thomas(a)hoberg.net&gt; wrote: could it be this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1813028 ? It seems only verified as a status. Can you check /var/log/ovirt-engine/ova/ (on engine I presume) if you find similar permission errors inside it? Gianluca

On Sat, Aug 1, 2020 at 12:36 AM <thomas(a)hoberg.net&gt; wrote: Yes, but did you check the files at the path I gave you? Right now. Environment with oVirt 4.3.10. I have a CentOS 8 VM with a 20Gb preallocated disk (and inside the VM 4Gb actually used). I shutdown the VM and then export as OVA. The process completes, apparently successfully, and on the engine in /var/log/ovirt-engine/ova 3 files are created (ov301 is the host chosen for the export): ovirt-export-ova-validate-ansible-20200801011543-ov301.mydomain-bc243185-2c3b-4f22-ac8c-8a59ecab5610.log ovirt-image-measure-ansible-20200801011608-ov301.mydomain-476aff3d.log ovirt-export-ova-ansible-20200801011610-ov301.mydomain-476aff3d.log and as described inside the bug I referred I see in the third file: " converting disk: /rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/ff10a405- cc61-4d00-a83f-3ee04b19f381/146d7e09-a6df-41d1-9a88-fb5c0174b099, offset 13312 qemu-img: /dev/loop0: error while converting qcow2: Could not open device: Permission denied STDERR: Shared connection to ov301 closed. " On ov301 apparently the file has been created: [g.cecchi@ov301 ~]$ ll /tmp/c8client.ova -rw-------. 1 root root 21478389760 Aug 1 01:16 /tmp/c8client.ova [g.cecchi@ov301 ~]$ But: [g.cecchi@ov301 ~]$ du -sh /tmp/c8client.ova 20K /tmp/c8client.ova [g.cecchi@ov301 ~]$ So I think this is a severe problem, much more because it doesn't give any error and gives false sensation of a good backup (until you try to import it anywhere... ) I already asked in the bugzilla to backport to 4.3. Otherwise if there is no will/time/resources to solve it, at least to disable the functionality. Post your comments to it too. Let's see and monitor the bug... Gianluca

Unfortunately that means OVA export/import doesn't seem to be a QA test case for oVirt releases... which would make me want to drop it like a hot potato, if there was any alternative at this point. I guess my biggest mistake was to assume that oVirt was like CentOS.

On Thu, Jul 30, 2020 at 5:10 PM <thomas(a)hoberg.net&gt; wrote: OVA export/import[1] is not a backup solution. This is a way to move a VM from one environment to another. For this we have a much better way, detach data domain and import it in another environment. If 4.4 we have a real backup API (tech preview). This will be used by backup vendors to provide complete backup solution, but we have basic backup capabilities included in the sdk examples. See the devconf 2020 talk[4] for more info. The examples deal only with backing up and restoring disks. Getting the vm configuration and creating a vm from the configuration is not included yet. Here is how you can do full back of a running vm disks using backup_vm.py[2] example: $ ./backup_vm.py full --engine-url https://engine3 \ --username admin@internal \ --password-file engine3-password \ --cafile engine3.pem \ --backup-dir backups \ ed2e2c59-36d3-41e2-ac7e-f4d33eb69ad4 [ 0.0 ] Starting full backup for VM ed2e2c59-36d3-41e2-ac7e-f4d33eb69ad4 [ 1.7 ] Waiting until backup 4ac0ae37-b1dd-4a8d-ad13-1ddb51e70d3e is ready [ 3.8 ] Created checkpoint '3648484c-61dc-4680-a7b4-0256544fa21d' (to use in --from-checkpoint-uuid for the next incremental backup) [ 3.8 ] Creating image transfer for disk 419b83e7-a6b5-4445-9324-a85e27b134d2 [ 5.1 ] Image transfer c659437e-6242-42b9-9293-8ccf10321261 is ready Formatting 'backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008011652.full.qcow2', fmt=qcow2 size=107374182400 cluster_size=65536 lazy_refcounts=off refcount_bits=16 [ 100.00% ] 100.00 GiB, 0.39 seconds, 258.06 GiB/s [ 5.5 ] Finalizing image transfer [ 7.5 ] Creating image transfer for disk 58daea80-1229-4c6b-b33c-1a4e568c8ad7 [ 8.7 ] Image transfer 78254cef-5f22-4161-8dbd-010420730a7f is ready Formatting 'backups/58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008011652.full.qcow2', fmt=qcow2 size=6442450944 cluster_size=65536 lazy_refcounts=off refcount_bits=16 [ 100.00% ] 6.00 GiB, 10.33 seconds, 594.54 MiB/s [ 19.0 ] Finalizing image transfer [ 23.3 ] Full backup completed successfully Note that we did not create any snapshot, backup provides a consistent view on all disks at the time of the backup, but you must have qemu-guest-agent installed in the VM. This is the same consistency you get from snapshot without memory. $ ls -lh backups/ total 2.3G -rw-r--r--. 1 nsoffer nsoffer 52M Aug 1 16:52 419b83e7-a6b5-4445-9324-a85e27b134d2.202008011652.full.qcow2 -rw-r--r--. 1 nsoffer nsoffer 2.2G Aug 1 16:52 58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008011652.full.qcow2 In this example we detected that the 100 GiB disk 419b83e7-a6b5-4445-9324-a85e27b134d2.202008011652 is mostly empty (mounted file system, no data written yet) so we could download only the allocated blocks. $ qemu-img info backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008011652.full.qcow2 image: backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008011652.full.qcow2 file format: qcow2 virtual size: 100 GiB (107374182400 bytes) disk size: 51 MiB cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false The second disk is a Fedora 32 OS disk, so we downloaded only the allocated blocks, about 2.2 GiB. $ qemu-img info backups/58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008011652.full.qcow2 image: backups/58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008011652.full.qcow2 file format: qcow2 virtual size: 6 GiB (6442450944 bytes) disk size: 2.17 GiB cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false To restore the disks you can use upload_disk.py[3] example: $ ./upload_disk.py --engine-url https://engine3 \ --username admin@internal \ --password-file engine3-password \ --cafile engine3.pem \ --sd-name nfs1 \ --disk-format raw \ --disk-sparse \ backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008011652.full.qcow2 Checking image... Image format: qcow2 Disk format: raw Disk content type: data Disk provisioned size: 107374182400 Disk initial size: 107374182400 Disk name: 419b83e7-a6b5-4445-9324-a85e27b134d2.202008011652.full.raw Disk backup: False Connecting... Creating disk... Disk ID: 5c9465d6-6437-4ece-bcaa-eec6b2d00a9d Creating image transfer... Transfer ID: dfcce09b-1f65-45e6-8ea5-41dff9368b15 Transfer host name: host4 Uploading image... [ 100.00% ] 100.00 GiB, 0.75 seconds, 133.31 GiB/s Finalizing image transfer... Upload completed successfully Again we upload only the allocated blocks, so the upload is very quick. Note that we upload the qcow2 disk to raw sparse disk. We support on-the-fly image format conversion. $ ./upload_disk.py --engine-url https://engine3 \ --username admin@internal \ --password-file engine3-password \ --cafile engine3.pem \ --sd-name nfs1 \ --disk-format qcow2 \ --disk-sparse \ backups/58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008011652.full.qcow2 Checking image... Image format: qcow2 Disk format: cow Disk content type: data Disk provisioned size: 6442450944 Disk initial size: 2332491776 Disk name: 58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008011652.full.qcow2 Disk backup: False Connecting... Creating disk... Disk ID: 5cf45aa6-6825-4aa5-adf1-4eb70809c37f Creating image transfer... Transfer ID: ed4046c3-63f9-445b-895b-e964dbcfc134 Transfer host name: host4 Uploading image... [ 100.00% ] 6.00 GiB, 5.62 seconds, 1.07 GiB/s Finalizing image transfer... Upload completed successfully This time we uploaded to a qcow2 format. You can create a new vm from these disks with the same configuration of the previous vm. I hope that future versions will backup and restore complete vms. This also supports incremental backups, but it is not suitable for production, there are too many rough edges that need to be smoothed. We also support uploading disks from ova file. Lets create one with compressed qcow2 format. This creates smaller ova files optimized for transfering to other machines. $ mkdir ova $ qemu-img convert -f qcow2 -O qcow2 -c backups/58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008011652.full.qcow2 ova/disk1.qcow2 $ qemu-img convert -f qcow2 -O qcow2 -c backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008011652.full.qcow2 ova/disk2.qcow2 $ ls -lh ova total 1.3G -rw-r--r--. 1 nsoffer nsoffer 1.3G Aug 1 17:20 disk1.qcow2 -rw-r--r--. 1 nsoffer nsoffer 576K Aug 1 17:20 disk2.qcow2 $ echo "this is not a real ovf" > ova/vm.ovf $ tar cvf test.ova -C ova vm.ovf disk1.qcow2 disk2.qcow2 vm.ovf disk1.qcow2 disk2.qcow2 $ ls -lhs test.ova 1.3G -rw-rw-r--. 1 nsoffer nsoffer 1.3G Aug 1 17:24 test.ova This is not a valid ovf, but good enough to show the upload capability: $ for disk in disk1.qcow2 disk2.qcow2; do ./upload_from_ova.py --engine-url https://engine3 \ --username admin@internal \ --password-file engine3-password \ --cafile engine3.pem \ --sd-name nfs1 \ --disk-format raw \ --disk-sparse \ --ova-disk-name $disk \ test.ova done Checking image... Image format: qcow2 Disk name: disk1.raw Disk format: raw Disk provisioned size: 6442450944 Disk initial size: None Disk backup: None Creating disk... Disk ID: acb7bc75-f6e9-49c8-a158-d7d581c108b4 Creating image transfer... Transfer ID: fca2d537-0f91-4033-8b35-ef0084191b4e Transfer host name: host4 Uploading disk disk1.qcow2 from test.ova... [ 100.00% ] 6.00 GiB, 6.88 seconds, 893.04 MiB/s Finalizing image transfer... Upload completed successfully Checking image... Image format: qcow2 Disk name: disk2.raw Disk format: raw Disk provisioned size: 107374182400 Disk initial size: None Disk backup: None Creating disk... Disk ID: ec57e31a-a531-48c2-8447-6468087bb4ab Creating image transfer... Transfer ID: 3aa23287-6ec4-4571-8623-7724c4a36341 Transfer host name: host4 Uploading disk disk2.qcow2 from test.ova... [ 100.00% ] 100.00 GiB, 0.49 seconds, 204.34 GiB/s Finalizing image transfer... Upload completed successfully To try this you need to install python3-ovirt-enigne-sdk4. The examples scripts are in: /usr/share/doc/python3-ovirt-engine-sdk4/examples/ We have automated tests using the upload/download/backup example scripts, so they should work. Nir [1] https://www.youtube.com/watch?v=VtuwWL6SMYk [2] https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/backup... [3] https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/upload... [4] https://www.youtube.com/watch?v=foyi1UyadEc

Nir, first of all, thanks a lot for the detailed description and the quick fix in 4.4! I guess I'll be able to paste that single line fix into the 4.3. variant myself, but I'd rather see that included in the next 4.3 release, too: How would that work? --- "OVA is not a backup solution": From time to time, try to put youself into a user's shoes. The fist thing you read about Export Domains, is that they are deprecated: That doesn't give you the warm fuzzy feeling that you are learning something useful when you start using them, especially in the context of a migration to the next release. OVA on the other hand, stands for a maximum of interoperability and when given a choice between something proprietary and deprecated and a file format that will port pretty much everywhere, any normal user (who doesn't have the code behind the scene in his mind), will jump for OVA export/import. Also it's just two buttons, no hassle, while it took me a while to get an Export domain defined, filled, detached, re-attached, and tested. Again from a user's perspective: HCI gluster storage in oVirt is black magic, especially since disk images are all chunked up. For a user it will probably take many years of continous oVirt operation until he's confident that he'l recover VM storage in the case of a serious hickup and that whatever might have gone wrong or bitrot might have occured, won't carry over to an export domain. OVA files seem like a nice bet to recover your VM on whatever platform you can get back running in a minor disaster. In many cases, it doesn't even matter you have to shut down the machine to do the export, because the machines are application level redundant or simply it's ok to have them down for a couple of minutes, if you know you can get them back up no matter what in a comparable time frame, oVirt farm dead or alive, e.g. on a bare metal machine. And then my case, many of the images are just meant to move between an oVirt farm and a desktop hypervisor. tl;dr (working) OVA export and import IMHO are elemental and crucial functionality, without which oVirt can't be labelled a product. I completely appreciate the new backup API, especially with the consistency enabled for running VMs; perhaps a little less that I'd have to purchase an extra product to do a fundamental operation with a similar ease as the OVA export/import buttons, but at least it's there. That doesn't mean OVA in/ex isn't important or that in fact a shared import/export domain would be nice, too. Thanks for your time!

HI Gianluca, I have had a look at the change and it's a single line of code added on the hosted-engine. I'll verify that it's working 4.3 and will make a note of checking it with engine upgrade, which for now seems the less troublesome approach. Hopefully it will get integrated/backported also with 4.3 or I'll have a chance to validate 4.4.

I agree that one way should be enough, but it needs to work and ideally improve on usability generation on generation. I am used to deal with VMs very much like LEGO bricks. I have moved them between hardware and hypervisors ever since Mendel Rosenblum & friends managed to trick the 80486SL into running a hypervisor with the system management mode intended by Intel only to make DOS not eat batteries on luggable hardware. Windows systems have made huge improvements, moving p2v, v2v, p2p, even v2p, Linux requires some precautions and constraints I have learned to observe. The the more sensible (not as polite...) question is to ask: Should I actually move these VMs? And you're right, that I should better not, use infrastructure as code etc. but that leads right to containers rather than VMs. I've used OpenVZ in production for more than a decade for that and even nested them with Docker to get both scale-in and scale-out benefits and less friction between ops and devs. But as your very own Roy Goland blogged in January '19, every cloud needs an anchor, contro place or service VMs, for which oVirt is just right... but also the outer-most loop, where infra as code delivers the lowest benefits. Still the lab infra I support as a side job is so small, and my coding has become so rusty (wrote a Linux emulator for a distributed µ-kernel as part of my thesis when Linus still didn't know that task state segments are too klunky for task switching), that I'd just prefer to use an appliance now, but which I know to have proper REST/Python APIs in case it grows into something bigger. That doesn't change that any hypervisor should just support the elemental save/store/export/import operations in addition to start/stop/suspend/clone/migrate etc. VMware, XenServer, VirtualBox each probably aren't just minor players compared to oVirt/RHV and when you want to gain market share, bi-directional interoperability can lower barriers... even if they are a tad costly to maintain. In any case I currently blame the others for not understanding the OVAs QEMU is producing, at least until I can prove that wrong. Yes, when you promise your team that things will be easier and better with oVirt, it's nice when you can avoid them losing everything. Not the typical emergency use case actually, even if I have done v2p on occasion. More typical would be that I'd just put (actually have) a VirtualBox on a normal Docker/Kubernetes compute node and just have that run a control plane/service VM that I had exported to cover this near disaster scenario of a failed cluster. Actually again for LEGO spirit and flexibility I was going to put oVirt, VirtualBox and Docker on our big GPU ML compute nodes and then put policies in place to control how and if workloads would ever migrate there, because each would be blind to one another. Alas, while Docker and VirtualBox get along fine, oVirt doesn't like to play with any of the other two, let alone both. Nothing magic, really just that developers sometimes like to build prototypes on their corporate Windows mobile workstations to work on them at home or even demo them to customers. VirtualBox isn't too bad but VMware workstation can make it very easy to just build an infrastructure out of LEGO VMs with a network included. Far less sophisticated than OVN and oVirt, but *everybody* can use it pretty much out of the box. Bare KVM is just not the same experience and I was actually very sad to notice, that VirtualBox and oVirt refuse to co-exist on a single host, even if both use KVM underneath! I got VirtualBox and VMware on my Windows machines without issues, and it's only HyperV that is getting increasingly exclusive, after a while when I could even run all three on a single machine. Not that anyone *should* do that, but then for type 2 hypervisors there is no (technical) reason not to play well and I don't want politics on *my* machines. And don't get me started on wanting to use nesting for the real LEGO experience! You know, when somebody gives you a button, you tend to assume it's even easier than moving disks :-) I wouldn't actually assume that OVA export does much more than export disk images and add a few tags for the common subset of settings between hypervisors. And I was likewise quite surprised at the amount of work done by OVA import, when VirtualBox/VMware images were recognized. It's nice to see when it works, but I wasn't really expecting that. Linux is so hypervisor enabled and comes with built-in guest additions, that I haven't seen the need when moving VMs between VB/VMware/Hyper-V. And with Windows VMs, starting with W7/W2012 it's become almost a no-op, just pop it in and it will adjust with little more than a reboot, unless you want those virtio drivers for speed. I'll be happy to write up a long review of my experience if you would like that. I can type pretty fast, but few people like reading any more.. Thank you for your time and patience!