On Thu, Dec 7, 2017 at 10:15 AM Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: Trying harder... It is not. file size is what you get from stat: $ truncate -s 1g empty $ stat empty File: 'empty' Size: 1073741824 Blocks: 0 IO Block: 4096 regular file ... $ qemu-img info empty image: empty file format: raw virtual size: 1.0G (1073741824 bytes) disk size: 0 The value "disk size" used by qemu-img is confusing and not useful when you want to transfer the file to another host. I don't know why qemu-img display this value instead of the actual file size, adding qemu-block mailing list in case someone can explain this. When you upload or download this file, you will transfer 1g of zeros. Yes this is raw sparse file. That bug is not related. It was about converting raw sparse format to qcow2. In 4.2 we use qemu-img map and the same algorithm used in qemu to estimate the required size needed to convert raw volume to qcow2 volume. For transferring images over http, there is no way to avoid sending unused blocks, except compression. You can use rsync to transfer images if you like, but we support only http. I tested creating qcow2 files with virt manager, and it seems to use the preallocation=metadata by default (behind your back), I guess for improving performance in some cases. oVirt doe not use this option so we don't have this issue when downloading files. Here example of image created by virt-manager: # ls -lsh fedora26-2.img 3.4M -rw-------. 1 root root 21G Dec 7 21:36 fedora26-2.img # qemu-img info fedora26-2.img image: fedora26-2.img file format: qcow2 virtual size: 20G (21474836480 bytes) disk size: 3.3M cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: true refcount bits: 16 corrupt: false And here is same image as ovirt create images: # qemu-img create -f qcow2 test.qcow2 20g Formatting 'test.qcow2', fmt=qcow2 size=21474836480 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 # qemu-img info test.qcow2 image: test.qcow2 file format: qcow2 virtual size: 20G (21474836480 bytes) disk size: 196K cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false # ls -lhs test.qcow2 196K -rw-r--r--. 1 root root 193K Dec 7 21:40 test.qcow2 So to download this image you need to transfer only 196k of data, not 20g. We can get the same results as virt-manager if we use preallocation=metadata: # qemu-img create -f qcow2 -o preallocation=metadata test-preallocation-metadata.qcow2 20g Formatting 'test-preallocation-metadata.qcow2', fmt=qcow2 size=21474836480 encryption=off cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16 # qemu-img info test-preallocation-metadata.qcow2 image: test-preallocation-metadata.qcow2 file format: qcow2 virtual size: 20G (21474836480 bytes) disk size: 3.3M cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false # ls -lhs test-preallocation-metadata.qcow2 3.4M -rw-r--r--. 1 root root 21G Dec 7 21:42 test-preallocation-metadata.qcow2 First message at beginning of upload: To optimize the import of images with unneeded preallocation, you can convert them to unallocated and compressed images as I showed in the previous mail. Here is an example with the preallocated images, like the one from virt-manager. I'll create file with some data - I'm using data that will compress well to make the example more convincing :-) # dd if=/dev/zero bs=8M count=128 | tr "\0" "\1" > image.raw 128+0 records in 128+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.88171 s, 571 MB/s Add some empty space: # truncate -s 10g image.raw # ls -lhs image.raw 1.1G -rw-r--r--. 1 root root 10G Dec 7 22:01 image.raw Now lest create a qcow2 preallocated image like virt-manger does: # qemu-img convert -f raw -O qcow2 -o preallocation=metadata image.raw image-preallocated-metadata.qcow2 # qemu-img info image-preallocated-metadata.qcow2 image: image-preallocated-metadata.qcow2 file format: qcow2 virtual size: 10G (10737418240 bytes) disk size: 1.0G cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false # ls -lhs image* 1.1G -rw-r--r--. 1 root root 11G Dec 7 22:02 image-preallocated-metadata.qcow2 1.1G -rw-r--r--. 1 root root 10G Dec 7 22:01 image.raw Finally, create a compressed qcow2, optimized for upload: # qemu-img convert -c -f qcow2 -O qcow2 image-preallocated-metadata.qcow2 image-preallocated-metadata-optimized.qcow2 # qemu-img info image-preallocated-metadata-optimized.qcow2 image: image-preallocated-metadata-optimized.qcow2 file format: qcow2 virtual size: 10G (10737418240 bytes) disk size: 1.6M cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false # ls -lhs image* 1.6M -rw-r--r--. 1 root root 1.7M Dec 7 22:04 image-preallocated-metadata-optimized.qcow2 1.1G -rw-r--r--. 1 root root 11G Dec 7 22:02 image-preallocated-metadata.qcow2 1.1G -rw-r--r--. 1 root root 10G Dec 7 22:01 image.raw All the images contain exactly the same data. Now, how can we optimize your image on upload? we can't. When uploading from the UI, our code run in the browser and we don't have access to your file system. When using the oVirt SDK, you write the code for uploading so you can do the conversion if want to save upload time and bandwidth and can spend the time to do the conversion. We can provide an upload command line tool that will convert and compress images before upload, and will use the oVirt SDK to upload the images. But I think it will be simpler and easier to maintain to support compression when uploading or downloading data. This is already supported by browsers and common tools like curl, and it works with any file format without conversion. I hope I convinced you this time ;-) Nir

On Thu, Dec 7, 2017 at 9:33 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: OK, understood Ah, ok. I didn't know about this preallocation option of qemu-img Does your example mean that by default oVirt uses preallocation=off when creating thin provisoned disks? Yes :- ] and thanks for the deep an detailed explanation, Nir Thanks also to Eric for his comments in the other reply Gianluca

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --bhR347vL55SA6RBIfMovKtx9QhoMUo0mw Content-Type: multipart/mixed; boundary="Rex9njBKm0vVDcsej8W9ig7Xt3vd8hFbN"; protected-headers="v1" From: Max Reitz <mreitz(a)redhat.com&gt; To: Nir Soffer <nsoffer(a)redhat.com&gt;, Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt;, &quot;qemu-block(a)nongnu.org&quot; <qemu-block(a)nongnu.org&gt; Cc: Paolo Margara <paolo.margara(a)polito.it&gt;, users <users(a)ovirt.org&gt; Message-ID: <5c36d206-1e28-9376-dd68-3675793af55f(a)redhat.com&gt; Subject: Re: [Qemu-block] import thin provisioned disks with image upload References: <CAG2kNCy9Ok=rqbAi4i+XfWGoF=pSjugsAv=iP=whgf7QPtJVsA(a)mail.gmail.com&gt; <CAMRbyyu4rchkT8WYE-QWwkFZFKGaKtFzoLPX1_jkwYDhNBn0Mg(a)mail.gmail.com&gt; In-Reply-To: <CAMRbyyu4rchkT8WYE-QWwkFZFKGaKtFzoLPX1_jkwYDhNBn0Mg(a)mail.gmail.com&gt; --Rex9njBKm0vVDcsej8W9ig7Xt3vd8hFbN Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 2017-12-07 21:33, Nir Soffer wrote: [...] lock: 4096=C2=A0 =C2=A0regular file Because I still think it's ambiguous. $ qemu-img create -f raw empty 1G Formatting 'empty', fmt=3Draw size=3D1073741824 $ qemu-img info empty image: empty file format: raw virtual size: 1.0G (1073741824 bytes) disk size: 0 $ LANG=3DC stat empty File: empty Size: 1073741824 Blocks: 0 IO Block: 4096 regular file [...] $ ls -s empty 0 empty $ du -h empty 0 empty $ ls -l empty -rw-r--r--. 1 maxx maxx 1073741824 8. Dez 16:20 empty What "stat" reports as "size" I'd call the length (what qemu-img info calls "virtual size" for raw images). What I (and qemu-img info) call "size" is how much disk space is actually used. And both ls -s and du agree that it's 0 bytes in this case. By the way, yes, "stat" has a different definition of "size", so that seems wrong. But I argue that the "ls" option is called "--size", so there's a conflict already and qemu-img info is not the first tool to disagree with stat on this. Max --Rex9njBKm0vVDcsej8W9ig7Xt3vd8hFbN-- --bhR347vL55SA6RBIfMovKtx9QhoMUo0mw Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" --bhR347vL55SA6RBIfMovKtx9QhoMUo0mw--

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --4JgC3IunLbJ0G8fkgNsQeg3PIJ43XvNLO Content-Type: multipart/mixed; boundary="o3shsonQasX9drPhV9NfPorFSsboNCKwB"; protected-headers="v1" From: Max Reitz <mreitz(a)redhat.com&gt; To: Nir Soffer <nsoffer(a)redhat.com&gt; Cc: Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt;, &quot;qemu-block(a)nongnu.org&quot; <qemu-block(a)nongnu.org&gt;, Paolo Margara <paolo.margara(a)polito.it&gt;, users <users(a)ovirt.org&gt; Message-ID: <546b8b49-4523-3a55-6665-b7b2db85f7e6(a)redhat.com&gt; Subject: Re: [Qemu-block] import thin provisioned disks with image upload References: <CAG2kNCy9Ok=rqbAi4i+XfWGoF=pSjugsAv=iP=whgf7QPtJVsA(a)mail.gmail.com&gt; <CAMRbyyu4rchkT8WYE-QWwkFZFKGaKtFzoLPX1_jkwYDhNBn0Mg(a)mail.gmail.com&gt; <5c36d206-1e28-9376-dd68-3675793af55f(a)redhat.com&gt; <CAMRbyyt7P+Du1MS6q5FHzuyrBvMZhf5y2CRJ=Fa3dV_co5VBDw(a)mail.gmail.com&gt; In-Reply-To: <CAMRbyyt7P+Du1MS6q5FHzuyrBvMZhf5y2CRJ=Fa3dV_co5VBDw(a)mail.gmail.com&gt; --o3shsonQasX9drPhV9NfPorFSsboNCKwB Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 2017-12-08 21:56, Nir Soffer wrote: [...] fo I personally don't quite see how the file length is important for other image types. At least the qcow2 driver doesn't really try to keep the file length small because it generally assumes that it's just a number that doesn't really matter. (That was my impression, at least.) s and du at ze", so to at First of all, we cannot easily change the name from "size". In the QAPI definition of ImageInfo it's called "actual-size", so we cannot change it without breaking compatibility. We could change the human-readable output, though; maybe to something like "host disk usage". Documentation is a different issue. There is some for the actual-size field of ImageInfo, but even that is indeed lacking. e" Not impossible, although I personally don't really see the point. To be honest, what I fear is that people see the file length and complain why it's so large when it's actually just some number that shouldn't matter whatsoever. If people really want to find out, I don't think ls -l is that hard to use. OK, so let me sum up: First, I agree that "file size" is confusing. (I disagree that it would be wrong. I think it is simply ambiguous.) So we should rename it at least in the human-readable output. Secondly, we should have documentation about this in the qemu-img man page, and better documentation for ImageInfo.actual-size. Thirdly, I don't see the point of displaying the file length in qemu-img info. But since you think it useful and it probably wouldn't be too hard, we can add it. My only fear about this is that I consider it an arbitrary and useless number that may confuse people. It hopefully shouldn't, as long as they can see the actual disk usage at the same time, though. (I just don't know how seeing the actual image file length in qemu-img info would have helped in your case. The important thing would have been to know that image files usually do contain large holes and you need to enable hole detection when transferring image files anywhere (as far as I have seen).) Max --o3shsonQasX9drPhV9NfPorFSsboNCKwB-- --4JgC3IunLbJ0G8fkgNsQeg3PIJ43XvNLO Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" --4JgC3IunLbJ0G8fkgNsQeg3PIJ43XvNLO--