Re: [ovirt-users] import thin provisioned disks with image upload

On Thu, Dec 7, 2017 at 10:15 AM Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: Trying harder... It is not. file size is what you get from stat: $ truncate -s 1g empty $ stat empty File: 'empty' Size: 1073741824 Blocks: 0 IO Block: 4096 regular file ... $ qemu-img info empty image: empty file format: raw virtual size: 1.0G (1073741824 bytes) disk size: 0 The value "disk size" used by qemu-img is confusing and not useful when you want to transfer the file to another host. I don't know why qemu-img display this value instead of the actual file size, adding qemu-block mailing list in case someone can explain this. When you upload or download this file, you will transfer 1g of zeros. Yes this is raw sparse file. That bug is not related. It was about converting raw sparse format to qcow2. In 4.2 we use qemu-img map and the same algorithm used in qemu to estimate the required size needed to convert raw volume to qcow2 volume. For transferring images over http, there is no way to avoid sending unused blocks, except compression. You can use rsync to transfer images if you like, but we support only http. I tested creating qcow2 files with virt manager, and it seems to use the preallocation=metadata by default (behind your back), I guess for improving performance in some cases. oVirt doe not use this option so we don't have this issue when downloading files. Here example of image created by virt-manager: # ls -lsh fedora26-2.img 3.4M -rw-------. 1 root root 21G Dec 7 21:36 fedora26-2.img # qemu-img info fedora26-2.img image: fedora26-2.img file format: qcow2 virtual size: 20G (21474836480 bytes) disk size: 3.3M cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: true refcount bits: 16 corrupt: false And here is same image as ovirt create images: # qemu-img create -f qcow2 test.qcow2 20g Formatting 'test.qcow2', fmt=qcow2 size=21474836480 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 # qemu-img info test.qcow2 image: test.qcow2 file format: qcow2 virtual size: 20G (21474836480 bytes) disk size: 196K cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false # ls -lhs test.qcow2 196K -rw-r--r--. 1 root root 193K Dec 7 21:40 test.qcow2 So to download this image you need to transfer only 196k of data, not 20g. We can get the same results as virt-manager if we use preallocation=metadata: # qemu-img create -f qcow2 -o preallocation=metadata test-preallocation-metadata.qcow2 20g Formatting 'test-preallocation-metadata.qcow2', fmt=qcow2 size=21474836480 encryption=off cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16 # qemu-img info test-preallocation-metadata.qcow2 image: test-preallocation-metadata.qcow2 file format: qcow2 virtual size: 20G (21474836480 bytes) disk size: 3.3M cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false # ls -lhs test-preallocation-metadata.qcow2 3.4M -rw-r--r--. 1 root root 21G Dec 7 21:42 test-preallocation-metadata.qcow2 First message at beginning of upload: To optimize the import of images with unneeded preallocation, you can convert them to unallocated and compressed images as I showed in the previous mail. Here is an example with the preallocated images, like the one from virt-manager. I'll create file with some data - I'm using data that will compress well to make the example more convincing :-) # dd if=/dev/zero bs=8M count=128 | tr "\0" "\1" > image.raw 128+0 records in 128+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.88171 s, 571 MB/s Add some empty space: # truncate -s 10g image.raw # ls -lhs image.raw 1.1G -rw-r--r--. 1 root root 10G Dec 7 22:01 image.raw Now lest create a qcow2 preallocated image like virt-manger does: # qemu-img convert -f raw -O qcow2 -o preallocation=metadata image.raw image-preallocated-metadata.qcow2 # qemu-img info image-preallocated-metadata.qcow2 image: image-preallocated-metadata.qcow2 file format: qcow2 virtual size: 10G (10737418240 bytes) disk size: 1.0G cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false # ls -lhs image* 1.1G -rw-r--r--. 1 root root 11G Dec 7 22:02 image-preallocated-metadata.qcow2 1.1G -rw-r--r--. 1 root root 10G Dec 7 22:01 image.raw Finally, create a compressed qcow2, optimized for upload: # qemu-img convert -c -f qcow2 -O qcow2 image-preallocated-metadata.qcow2 image-preallocated-metadata-optimized.qcow2 # qemu-img info image-preallocated-metadata-optimized.qcow2 image: image-preallocated-metadata-optimized.qcow2 file format: qcow2 virtual size: 10G (10737418240 bytes) disk size: 1.6M cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false # ls -lhs image* 1.6M -rw-r--r--. 1 root root 1.7M Dec 7 22:04 image-preallocated-metadata-optimized.qcow2 1.1G -rw-r--r--. 1 root root 11G Dec 7 22:02 image-preallocated-metadata.qcow2 1.1G -rw-r--r--. 1 root root 10G Dec 7 22:01 image.raw All the images contain exactly the same data. Now, how can we optimize your image on upload? we can't. When uploading from the UI, our code run in the browser and we don't have access to your file system. When using the oVirt SDK, you write the code for uploading so you can do the conversion if want to save upload time and bandwidth and can spend the time to do the conversion. We can provide an upload command line tool that will convert and compress images before upload, and will use the oVirt SDK to upload the images. But I think it will be simpler and easier to maintain to support compression when uploading or downloading data. This is already supported by browsers and common tools like curl, and it works with any file format without conversion. I hope I convinced you this time ;-) Nir