Re: [ovirt-users] [Qemu-block] import thin provisioned disks with image upload

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --ldHd2lCWj7jKEETVuI81oTog5UxEokqf6 Content-Type: multipart/mixed; boundary="Rcw9BITPPwikllx1VkJV7F1EW1x5wWi88"; protected-headers="v1" From: Eric Blake <eblake(a)redhat.com&gt; To: Nir Soffer <nsoffer(a)redhat.com&gt;, Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt;, &quot;qemu-block(a)nongnu.org&quot; <qemu-block(a)nongnu.org&gt; Cc: Paolo Margara <paolo.margara(a)polito.it&gt;, users <users(a)ovirt.org&gt; Message-ID: <9ae28d4c-35e7-c7f6-bbae-6f826a7007ef(a)redhat.com&gt; Subject: Re: [Qemu-block] import thin provisioned disks with image upload References: <CAG2kNCy9Ok=rqbAi4i+XfWGoF=pSjugsAv=iP=whgf7QPtJVsA(a)mail.gmail.com&gt; <CAMRbyyu4rchkT8WYE-QWwkFZFKGaKtFzoLPX1_jkwYDhNBn0Mg(a)mail.gmail.com&gt; In-Reply-To: <CAMRbyyu4rchkT8WYE-QWwkFZFKGaKtFzoLPX1_jkwYDhNBn0Mg(a)mail.gmail.com&gt; --Rcw9BITPPwikllx1VkJV7F1EW1x5wWi88 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 12/07/2017 02:33 PM, Nir Soffer wrote: For regular files, qemu is reporting the stat Blocks value (a completely sparse file occupies 0 blocks, and therefore uses 0 bytes of the host filesystem). Well, depending on whether your transfer mechanism has optimizations for transferring holes efficiently, you may not have to send 1G of zeroes over the wire; but yes, the receiving end must reconstruct a file containing 1G of zeroes (even if that reconstruction is also sparse, and occupies 0 blocks of the file system). d390a125daa6 d390a125daa6 You have to understand something about the way file systems work. Just because the filesystem reports 1.3G used, does NOT mean that 1.3G is contiguous; depending on how scatter-shot its allocation patterns have been, and the minimum granularity for holes in the host filesystem, there could very well be gaps that the guest is reporting as unused but which still contribute towards the in-use block count reported by the hos= t. Furthermore, when you delete a file, most filesystems do not immediately rewrite the underlying storage back to 0, but just update metadata in the filesystem to state that the storage can be reused later for a new file. The TRIM operation on filesystems can help reclaim some of that (now-unused) space, and more and more filesystems are starting to support the Linux ioctls for punching holes in the middle of existing files to quit occupying blocks. But the fact that the host filesystem claims 8.6G of used blocks is a typical symptom of not having TRIM'ed the guest filesystem recently, or a case where guest TRIMs do not reach through all the layers into a host hole-punching operation. Disk size of a raw file is how many host blocks are currently in use; but you are correct that it does not necessarily tell you how much data the guest filesystem is currently reporting as used - on the other hand, it is a good conservative upper bound for how much storage you need to reproduce the same bit pattern on the destination (even if many of the bits don't need to be reproduced, because the guest filesystem didn't care about them) - but only insofar as you can preserve sparseness of the portions of the file that are holes in the source. You may also be interested in the virt-sparsify tool from libguestfs, which is very good at squeezing out unneeded space from an image in order to get a smaller version that still produces the same content that the guest cares about. --=20 Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org --Rcw9BITPPwikllx1VkJV7F1EW1x5wWi88-- --ldHd2lCWj7jKEETVuI81oTog5UxEokqf6 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" --ldHd2lCWj7jKEETVuI81oTog5UxEokqf6--