On Wed, Aug 19, 2020 at 1:04 PM Kevin Doyle <kevin.doyle(a)manchester.ac.uk&gt; wrote: Full backup is supported for a while and can be used now. Incremental backup can be used now with on CentOS 8.2, but there are many rough edges that may break your incremental backup, like creating and deleting snapshots or moving disks. Restoring backups requires manual work, since we don't really have a backup solution but a backup API. The example backup scripts only do the minimal work to show how to use the API, and require some scripting to build a simple solution. What you run on the VM does not matter, since the backup is in the disk level, not in the guest filesystem level. However, to get consistent backups, you must run qemu-guest-agent in the guest. With these notes, here is an example backup for a vm. The VM has 2 disks: - 6g qcow2 - os disk - 100g qcow2 - data disk (empty file system) qcow2 is important, incremental backup is possible only for qcow2 disks. To enable incremental backup when creating a disk, you need to check this: [ ] Enable incremental backup This will use qcow2 format for the disk, regardless of the allocation policy (thin or preallocated). I'm running this on RHEL AV 8.3 nightly build, but this used to work on RHEL AV 8.2 few weeks ago, so CentOS 8.2 should work. If not you will have to wait for the next CentOS updated since we don't test CentOS. Install packages on the host running the backup script: dnf install python3-ovirt-engine-sdk4 ovirt-imageio-client If you run this on a oVirt host, the packages are already installed, and the backup will be faster. Creating a full backup: $ python3 /usr/share/doc/python3-ovirt-engine-sdk4/examples/backup_vm.py full \ --engine-url https://engine3 \ --username admin@internal \ --password-file password \ --cafile engine3.pem \ --backup-dir /var/tmp/backups \ ed2e2c59-36d3-41e2-ac7e-f4d33eb69ad4 [ 0.0 ] Starting full backup for VM ed2e2c59-36d3-41e2-ac7e-f4d33eb69ad4 [ 0.2 ] Waiting until backup e0de71d6-146a-40f6-80f6-7c6b2969aa3f is ready [ 1.2 ] Created checkpoint 'c13420d7-ab33-4545-acad-4fe1417cc61f' (to use in --from-checkpoint-uuid for the next incremental backup) [ 1.2 ] Creating image transfer for disk 419b83e7-a6b5-4445-9324-a85e27b134d2 [ 2.3 ] Image transfer 6fc1944c-ce4c-40e7-8119-61208f6489fe is ready Formatting '/var/tmp/backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008192037.full.qcow2', fmt=qcow2 cluster_size=65536 compression_type=zlib size=107374182400 lazy_refcounts=off refcount_bits=16 [ 100.00% ] 100.00 GiB, 0.79 seconds, 126.17 GiB/s [ 3.1 ] Finalizing image transfer [ 5.2 ] Creating image transfer for disk 58daea80-1229-4c6b-b33c-1a4e568c8ad7 [ 6.2 ] Image transfer 63e1f899-7add-46a8-8d68-4ef3e6a1385b is ready Formatting '/var/tmp/backups/58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008192037.full.qcow2', fmt=qcow2 cluster_size=65536 compression_type=zlib size=6442450944 lazy_refcounts=off refcount_bits=16 [ 100.00% ] 6.00 GiB, 8.39 seconds, 731.93 MiB/s [ 14.6 ] Finalizing image transfer [ 19.8 ] Full backup completed successfully The backup created these files: $ qemu-img info /var/tmp/backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008192037.full.qcow2 image: /var/tmp/backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008192037.full.qcow2 file format: qcow2 virtual size: 100 GiB (107374182400 bytes) disk size: 51 MiB ... $ qemu-img info /var/tmp/backups/58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008192037.full.qcow2 image: /var/tmp/backups/58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008192037.full.qcow2 file format: qcow2 virtual size: 6 GiB (6442450944 bytes) disk size: 2.2 GiB ... Now lets create an incremental backup - we need to use the checkpoint id created during the previous backup: [ 1.2 ] Created checkpoint 'c13420d7-ab33-4545-acad-4fe1417cc61f' (to use in --from-checkpoint-uuid for the next incremental backup) And the "incremental" sub command: $ python3 /usr/share/doc/python3-ovirt-engine-sdk4/examples/backup_vm.py incremental \ --engine-url https://engine3 \ --username admin@internal \ --password-file password \ --cafile engine3.pem \ --backup-dir /var/tmp/backups \ --from-checkpoint-uuid c13420d7-ab33-4545-acad-4fe1417cc61f \ ed2e2c59-36d3-41e2-ac7e-f4d33eb69ad4 [ 0.0 ] Starting incremental backup for VM ed2e2c59-36d3-41e2-ac7e-f4d33eb69ad4 [ 0.2 ] Waiting until backup 76fbea79-b92a-4970-a1bf-f60904dabc6a is ready [ 1.2 ] Created checkpoint '68936483-6c5c-41e8-9f0c-b00ff963edfc' (to use in --from-checkpoint-uuid for the next incremental backup) [ 1.2 ] Creating image transfer for disk 419b83e7-a6b5-4445-9324-a85e27b134d2 [ 2.3 ] Image transfer d173c1e6-0f97-4876-9fb9-a7abcff6ea03 is ready Formatting '/var/tmp/backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008192049.incremental.qcow2', fmt=qcow2 cluster_size=65536 compression_type=zlib size=107374182400 lazy_refcounts=off refcount_bits=16 [ 100.00% ] 100.00 GiB, 0.07 seconds, 1.47 TiB/s [ 2.4 ] Finalizing image transfer [ 4.4 ] Creating image transfer for disk 58daea80-1229-4c6b-b33c-1a4e568c8ad7 [ 5.5 ] Image transfer 43eeb0a2-71d0-4edb-ab9d-0a63f944f5f2 is ready Formatting '/var/tmp/backups/58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008192049.incremental.qcow2', fmt=qcow2 cluster_size=65536 compression_type=zlib size=6442450944 lazy_refcounts=off refcount_bits=16 [ 100.00% ] 6.00 GiB, 0.66 seconds, 9.05 GiB/s [ 6.2 ] Finalizing image transfer [ 8.3 ] Incremental backup completed successfully Because not much was changed in the vm in the last 5 minutes, the data disk backup is actually empty: $ qemu-img info /var/tmp/backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008192049.incremental.qcow2 image: /var/tmp/backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008192049.incremental.qcow2 file format: qcow2 virtual size: 100 GiB (107374182400 bytes) disk size: 196 KiB ... The os disk has some changes, probably logs: $ qemu-img info /var/tmp/backups/58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008192049.incremental.qcow2 image: /var/tmp/backups/58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008192049.incremental.qcow2 file format: qcow2 virtual size: 6 GiB (6442450944 bytes) disk size: 112 MiB ... Now let's write something in the data disk inside the guest: $ ssh root(a)192.168.122.178 # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 1024M 0 rom vda 252:0 0 6G 0 disk ├─vda1 252:1 0 1M 0 part ├─vda2 252:2 0 1G 0 part /boot ├─vda3 252:3 0 615M 0 part [SWAP] └─vda4 252:4 0 4.4G 0 part / vdb 252:16 0 100G 0 disk └─vdb1 252:17 0 100G 0 part /data # dd if=/dev/urandom bs=1M count=1024 of=/data/test oflag=direct 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 33.0801 s, 32.5 MB/s # exit logout Connection to 192.168.122.178 closed. Now let's make a new incremental backup, since the last incremental backup. Again, we need to use the checkpoint id: [ 1.2 ] Created checkpoint '68936483-6c5c-41e8-9f0c-b00ff963edfc' (to use in --from-checkpoint-uuid for the next incremental backup) $ python3 /usr/share/doc/python3-ovirt-engine-sdk4/examples/backup_vm.py incremental \ --engine-url https://engine3 \ --username admin@internal \ --password-file password \ --cafile engine3.pem \ --backup-dir /var/tmp/backups \ --from-checkpoint-uuid 68936483-6c5c-41e8-9f0c-b00ff963edfc \ ed2e2c59-36d3-41e2-ac7e-f4d33eb69ad4 [ 0.0 ] Starting incremental backup for VM ed2e2c59-36d3-41e2-ac7e-f4d33eb69ad4 [ 0.2 ] Waiting until backup 5e3d89b2-9031-4e9d-acc0-325cccaf0fc2 is ready [ 1.3 ] Created checkpoint '06af1c50-d242-46bf-9e1f-2b1b2b303a99' (to use in --from-checkpoint-uuid for the next incremental backup) [ 1.3 ] Creating image transfer for disk 419b83e7-a6b5-4445-9324-a85e27b134d2 [ 2.3 ] Image transfer 2f93354e-8d12-4348-8715-c39f57e973af is ready Formatting '/var/tmp/backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008192104.incremental.qcow2', fmt=qcow2 cluster_size=65536 compression_type=zlib size=107374182400 lazy_refcounts=off refcount_bits=16 [ 100.00% ] 100.00 GiB, 3.28 seconds, 30.52 GiB/s [ 5.6 ] Finalizing image transfer [ 8.7 ] Creating image transfer for disk 58daea80-1229-4c6b-b33c-1a4e568c8ad7 [ 9.8 ] Image transfer 6a36db7e-149f-46b6-82d3-79ddd2de5254 is ready Formatting '/var/tmp/backups/58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008192104.incremental.qcow2', fmt=qcow2 cluster_size=65536 compression_type=zlib size=6442450944 lazy_refcounts=off refcount_bits=16 [ 100.00% ] 6.00 GiB, 0.55 seconds, 10.84 GiB/s [ 10.4 ] Finalizing image transfer [ 11.5 ] Incremental backup completed successfully This time we copied about 1 GiB from the data disk: $ qemu-img info /var/tmp/backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008192104.incremental.qcow2 image: /var/tmp/backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008192104.incremental.qcow2 file format: qcow2 virtual size: 100 GiB (107374182400 bytes) disk size: 1 GiB ... But fewer changes from the os disk: $ qemu-img info /var/tmp/backups/58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008192104.incremental.qcow2 image: /var/tmp/backups/58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008192104.incremental.qcow2 file format: qcow2 virtual size: 6 GiB (6442450944 bytes) disk size: 12.8 MiB ... So we have 3 backups - we can restore this vm to any of these backups. Some manual work is needed to restore, rebasing every backup on top of the previous backup: $ ls -lh /var/tmp/backups/ total 3.4G -rw-r--r--. 1 nsoffer nsoffer 52M Aug 19 20:37 419b83e7-a6b5-4445-9324-a85e27b134d2.202008192037.full.qcow2 -rw-r--r--. 1 nsoffer nsoffer 194K Aug 19 20:49 419b83e7-a6b5-4445-9324-a85e27b134d2.202008192049.incremental.qcow2 -rw-r--r--. 1 nsoffer nsoffer 1.1G Aug 19 21:04 419b83e7-a6b5-4445-9324-a85e27b134d2.202008192104.incremental.qcow2 -rw-r--r--. 1 nsoffer nsoffer 2.2G Aug 19 20:38 58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008192037.full.qcow2 -rw-r--r--. 1 nsoffer nsoffer 112M Aug 19 20:50 58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008192049.incremental.qcow2 -rw-r--r--. 1 nsoffer nsoffer 13M Aug 19 21:04 58daea80-1229-4c6b-b33c-1a4e568c8ad7.202008192104.incremental.qcow2 This steps is something that the backup script should do automatically but it does not do this yet. $ cd /var/tmp/backups $ qemu-img rebase -u \ -b 419b83e7-a6b5-4445-9324-a85e27b134d2.202008192049.incremental.qcow2 \ -F qcow2 \ 419b83e7-a6b5-4445-9324-a85e27b134d2.202008192104.incremental.qcow2 $ qemu-img rebase -u \ -b 419b83e7-a6b5-4445-9324-a85e27b134d2.202008192037.full.qcow2 \ -F qcow2 \ 419b83e7-a6b5-4445-9324-a85e27b134d2.202008192049.incremental.qcow2 After the rebase, we have a complete chain: $ qemu-img info --backing-chain 419b83e7-a6b5-4445-9324-a85e27b134d2.202008192104.incremental.qcow2 image: 419b83e7-a6b5-4445-9324-a85e27b134d2.202008192104.incremental.qcow2 file format: qcow2 virtual size: 100 GiB (107374182400 bytes) disk size: 1 GiB backing file: 419b83e7-a6b5-4445-9324-a85e27b134d2.202008192049.incremental.qcow2 backing file format: qcow2 ... image: 419b83e7-a6b5-4445-9324-a85e27b134d2.202008192049.incremental.qcow2 file format: qcow2 virtual size: 100 GiB (107374182400 bytes) disk size: 196 KiB backing file: 419b83e7-a6b5-4445-9324-a85e27b134d2.202008192037.full.qcow2 backing file format: qcow2 ... image: 419b83e7-a6b5-4445-9324-a85e27b134d2.202008192037.full.qcow2 file format: qcow2 virtual size: 100 GiB (107374182400 bytes) disk size: 51 MiB ... Same rebase should be done for the other disk. To restore this disk to its latest state, upload the disk contents from the entire chain to some storage domain: $ python3 /usr/share/doc/python3-ovirt-engine-sdk4/examples/upload_disk.py \ --engine-url https://engine3 \ --username admin@internal \ --password-file password \ --cafile engine3.pem \ --sd-name iscsi1 \ --disk-sparse \ /var/tmp/backups/419b83e7-a6b5-4445-9324-a85e27b134d2.202008192104.incremental.qcow2 Checking image... Image format: qcow2 Disk format: cow Disk content type: data Disk provisioned size: 107374182400 Disk initial size: 1143341056 Disk name: 419b83e7-a6b5-4445-9324-a85e27b134d2.202008192104.incremental.qcow2 Disk backup: False Connecting... Creating disk... Disk ID: d5f88bdb-2b77-4c4e-a876-ef607d5a88a0 Creating image transfer... Transfer ID: d2dc3718-f8ee-4c13-9b7f-84fca902a35b Transfer host name: host4 Uploading image... [ 100.00% ] 100.00 GiB, 4.73 seconds, 21.15 GiB/s Finalizing image transfer... Upload completed successfully After rebasing the uploading the other disk you can create a new vm from the uploaded disks, and if needed remove the old vm. This probably looks complicated, because it is. Several backup vendors are working on integration with the new backup APIs, and you should have some easy to use solutions in the future. Nir

Vprotect can do some form of incremental backup of ovirt vms. At least on 4.3 I’m not sure where they’re at for 4.4 support. Worth checking out, free for 10 vms On Wed, Aug 19, 2020 at 7:03 AM Kevin Doyle <kevin.doyle(a)manchester.ac.uk&gt; wrote:

I have tried protect and found it slow, thanks anyway