[ovirt-devel] Re: [EXTERNAL] [BACKUP] Estimating qcow2 disk image size before upload

Hi Redhat Team, Thanks for the information on QCOW2 On ISCSI storage domain, if am creating a floating disk with “raw” + “sparse”. Then it gives me an error that this is not a valid format for this storage domain type. <disk> <storage_domains> <storage_domain id="834df3ca-b9c2-45d3-ade9-6384dc1517da"/> </storage_domains> <name>mydisk</name> <provisioned_size>400</provisioned_size> <format>raw</format> <sparse>true</sparse> </disk> Response <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <fault> <detail>[Cannot add Virtual Disk. Disk configuration (RAW Sparse) is incompatible with the storage domain type.]</detail> <reason>Operation Failed</reason> </fault> On further digging I found a already opened BZ for it https://bugzilla.redhat.com/show_bug.cgi?id=1600547 Is this going to be fixed in the near future ? Right now, it creates only COW disk if thin provisioning is chosen. From: Nir Soffer <nsoffer(a)redhat.com&gt; Date: Thursday, November 15, 2018 at 10:25 PM To: devel <devel(a)ovirt.org&gt; Cc: Suchitra Herwadkar <Suchitra.Herwadkar(a)veritas.com&gt;, Mahesh Falmari <Mahesh.Falmari(a)veritas.com&gt;, Daniel Erez <derez(a)redhat.com&gt;, "Nisan, Tal" <tnisan(a)redhat.com&gt;, Pavan Chavva <pchavva(a)redhat.com&gt;, Raj Asher <Raj.Asher(a)veritas.com&gt;, Himani Vaidya <Himani.Vaidya(a)veritas.com&gt; Subject: [EXTERNAL] [BACKUP] Estimating qcow2 disk image size before upload When uploading to qcow2 disks on block storage you must set the disk initial_size correctly so the system allocates big enough disk. If the initial size is too small, the upload will fail when trying to write after the end of the device. This info can be useful for people working on a backup solution for oVirt. The easiest case is upload of existing qcow2 image using the SDK. In this case we create a new disk with: initial_size=image_size, provisioned_size=virtual_size, image_size is the size of the file: os.stat('image').st_size Note that "qemu-img info" return "actual-size" - this is the allocated size on storage which is not helpful for uploading images. Example: https://github.com/oVirt/ovirt-engine-sdk/blob/78c3d5bd14eeb93ef72ec31d77... A more tricky case is when a backup system keeps raw guest data, but know how to generate qcow2 image stream, without creating a temporary image. In this case the required size can be calculated by counting the number of clusters that need to be allocated in the final image. This depends on the location of the data in the image. For example this creates 1G image with only one cluster: $ python -c 'with open("one-cluster.raw", "wb") as f: f.truncate(1024**3) f.write("x")' $ ls -lhs one-cluster.raw 4.0K -rw-rw-r--. 1 nsoffer nsoffer 1.0G Nov 15 18:24 one-cluster.raw $ qemu-img measure -f raw -O qcow2 one-cluster.raw required size: 458752 fully allocated size: 1074135040 $ qemu-img convert -f raw -O qcow2 one-cluster.raw one-cluster.qcow2 $ ls -lhs one-cluster.qcow2 324K -rw-r--r--. 1 nsoffer nsoffer 384K Nov 15 18:25 one-cluster.qcow2 But this creates a fully allocated 1G image: $ python -c 'with open("fully-allocated.raw", "wb") as f: f.truncate(1024**3) for i in range(0, 1024**3, 64 * 1024): f.seek(i) f.write("x")' $ ls -lhs fully-allocated.raw 64M -rw-rw-r--. 1 nsoffer nsoffer 1.0G Nov 15 18:30 fully-allocated.raw $ qemu-img measure -f raw -O qcow2 fully-allocated.raw required size: 1074135040 fully allocated size: 1074135040 $ qemu-img convert -f raw -O qcow2 fully-allocated.raw fully-allocated.qcow2 $ ls -lhs fully-allocated.qcow2 1.1G -rw-r--r--. 1 nsoffer nsoffer 1.1G Nov 15 18:31 fully-allocated.qcow2 We had code in vdsm that does exactly this, and it was removed since qemu-img support a new "measure" command in RHEL 7.5 providing this info. But this works only for existing images. You can find the code in this vdsm commit: https://github.com/oVirt/vdsm/tree/4eee97d6aa532b6f3ecdfde0333d17c27c412f86 The module implementing estimation: https://github.com/oVirt/vdsm/blob/4eee97d6aa532b6f3ecdfde0333d17c27c412f... The tests for this module: https://github.com/oVirt/vdsm/blob/4eee97d6aa532b6f3ecdfde0333d17c27c412f... If you know the data ranges that will be written to the qcow2 image, you can count the clusters like this: https://github.com/oVirt/vdsm/blob/4eee97d6aa532b6f3ecdfde0333d17c27c412f... Then you can use the cluster count to estimate qcow2 file size: https://github.com/oVirt/vdsm/blob/4eee97d6aa532b6f3ecdfde0333d17c27c412f... Nir