On Thu, Dec 7, 2017 at 10:15 AM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Thu, Dec 7, 2017 at 1:28 AM, Nir Soffer <nsoffer@redhat.com> wrote:


On Thu, Dec 7, 2017 at 1:22 AM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Wed, Dec 6, 2017 at 11:42 PM, Nir Soffer <nsoffer@redhat.com> wrote:



BTW: I notice that the disk seems preallocated even if original qcow2 is thin... is this expected?
This obviously also impacts the time to upload (20Gb virtual disk with actual 1.2Gb occupied needs the time equivalent for full 20Gb...)

We upload exactly the file you provided, there is no way we can upload 20G from 1.2G file :-)

But the upload process at a medium rate of 40-50MB/s has last about 9 minutes that confirms the 20Gb size
The disk at source has been created as virtio type and format qcow2 from virt-manager and then only installed a CentOS 7.2 OS with infrastructure server configuration.
Apart from qemu-img also ls:
# ls -lhs c7lab1.qcow2 
1.3G -rw------- 1 root root 21G Dec  6 23:05 c7lab1.qcow2

The fiile size is 21G - matching what you see. This is the size we upload.

I changed the subject to better address and track the thread..
You are not convincing me...

Trying harder...
 
The term "file size" is ambiguous in this context...

It is not. file size is what you get from stat:

$ truncate -s 1g empty

$ stat empty 
  File: 'empty'
  Size: 1073741824 Blocks: 0          IO Block: 4096   regular file
  ...

$ qemu-img info empty
image: empty
file format: raw
virtual size: 1.0G (1073741824 bytes)
disk size: 0

The value "disk size" used by qemu-img is confusing and not useful
when you want to transfer the file to another host.

I don't know why qemu-img display this value instead of the actual
file size, adding qemu-block mailing list in case someone can explain
this.

When you upload or download this file, you will transfer 1g of zeros.
 

If I take a disk that was born on this oVirt environment itself I have this:

that shows that is thin provisioned
Its id shown here:

If I go and check on my exported file system I get this

[root@ovirt01 NFS_DOMAIN]# find . -name "3c68d43f-0f28-4564-b557-d390a125daa6"
./572eabe7-15d0-42c2-8fa9-0bd773e22e2e/images/3c68d43f-0f28-4564-b557-d390a125daa6
[root@ovirt01 NFS_DOMAIN]# ls -lsh ./572eabe7-15d0-42c2-8fa9-0bd773e22e2e/images/3c68d43f-0f28-4564-b557-d390a125daa6
total 8.6G
8.6G -rw-rw----. 1 vdsm kvm  10G Dec  7 08:42 09ad8e53-0b22-4fe3-b718-d14352b8290a
1.0M -rw-rw----. 1 vdsm kvm 1.0M May  1  2016 09ad8e53-0b22-4fe3-b718-d14352b8290a.lease
4.0K -rw-r--r--. 1 vdsm kvm  317 Jun 21 10:41 09ad8e53-0b22-4fe3-b718-d14352b8290a.meta

[root@ovirt01 NFS_DOMAIN]# qemu-img info ./572eabe7-15d0-42c2-8fa9-0bd773e22e2e/images/3c68d43f-0f28-4564-b557-d390a125daa6/09ad8e53-0b22-4fe3-b718-d14352b8290a
image: ./572eabe7-15d0-42c2-8fa9-0bd773e22e2e/images/3c68d43f-0f28-4564-b557-d390a125daa6/09ad8e53-0b22-4fe3-b718-d14352b8290a
file format: raw
virtual size: 10G (10737418240 bytes)
disk size: 8.5G
[root@ovirt01 NFS_DOMAIN]#

So it seems it is raw/sparse

Yes this is raw sparse file.
 
1.3G is the used size on the file system, we cannot upload only used blocks.
qemu-img info "Disk size" is not the file size the the used size, not useful 
for upload.

Why not? You can detect the format of source and use a similar strategy of the bugzilla I'm referring (see below)
Or couldn't you use something like 

That bug is not related. It was about  converting raw sparse format
to qcow2.

In 4.2 we use qemu-img map and the same algorithm used in qemu to 
estimate the required size needed to convert raw volume to qcow2 volume.

For transferring images over http, there is no way to avoid sending 
unused blocks, except compression.
 

 rsync -av --sparse source target

You can use rsync to transfer images if you like, but we support
only http.
 

if the target is NFS?


Maybe this file was crated with preallocation=full?

In virt-manager there is not this logic by default.
The default format is qcow2/sparse
When you create/add a disk you can choose "Select or create custom storage":

And then choose a format between:
raw, qcow, qcow2, qed, vmdk, vpc, vdi

In my case it was the default one, so qcow2, as qemu-img correctly detects.

I tested creating qcow2 files with virt manager, and it seems to use
the preallocation=metadata by default (behind your back), I guess 
for improving performance in some cases. oVirt doe not use this
option so we don't have this issue when downloading files.

Here example of image created by virt-manager:

# ls -lsh fedora26-2.img 
3.4M -rw-------. 1 root root 21G Dec  7 21:36 fedora26-2.img

# qemu-img info fedora26-2.img 
image: fedora26-2.img
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 3.3M
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: true
    refcount bits: 16
    corrupt: false

And here is same image as ovirt create images:

# qemu-img create -f qcow2 test.qcow2 20g
Formatting 'test.qcow2', fmt=qcow2 size=21474836480 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

# qemu-img info test.qcow2 
image: test.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 196K
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

# ls -lhs test.qcow2 
196K -rw-r--r--. 1 root root 193K Dec  7 21:40 test.qcow2

So to download this image you need to transfer only 196k of data, not 20g.

We can get the same results as virt-manager if we use preallocation=metadata:
 
# qemu-img create -f qcow2 -o preallocation=metadata test-preallocation-metadata.qcow2 20g
Formatting 'test-preallocation-metadata.qcow2', fmt=qcow2 size=21474836480 encryption=off cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16

# qemu-img info test-preallocation-metadata.qcow2 
image: test-preallocation-metadata.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 3.3M
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

# ls -lhs test-preallocation-metadata.qcow2 
3.4M -rw-r--r--. 1 root root 21G Dec  7 21:42 test-preallocation-metadata.qcow2


First message at beginning of upload:

2017-12-06 23:09:50,183+0100 INFO  (jsonrpc/4) [vdsm.api] START createVolume(sdUUID=u'572eabe7-15d0-42c2-8fa9-0bd773e22e2e', spUUID=u'00000001-0001-0001-0001-000000000343', imgUUID=u'251063f6-5570-4bdc-b28f-21e82aa5e185', size=u'22548578304', volFormat=4, preallocate=2, diskType=2, volUUID=u'77aacfb3-9e67-4ad2-96f6-242b5ba4d9e0', desc=u'{"DiskAlias":"c7lab1","DiskDescription":""}', srcImgUUID=u'00000000-0000-0000-0000-000000000000', srcVolUUID=u'00000000-0000-0000-0000-000000000000', initialSize=None) from=192.168.1.212,56846, flow_id=18c6bd3b-76ab-45f9-b8c7-09c727f44c91, task_id=e7cc67e6-4b61-4bb3-81b1-6bc687ea5ee9 (api:46)


In this bugzilla, that is related to export from NFS to block:

it seems to understand, from a comment of Maor, that 
preallocate=2 --> sparse

what are instead the possible values of volFormat and their decodification?
In my case it detects 
volFormat=4,

Could it be something similar to the bugzilla above? Can it change anything if exporting with NFS 4.2 or using oVirt 4.2?

I think that if one has big qcow2 disk files that he/she desire to migrate into oVirt, the process could be optimized

To optimize the import of images with unneeded preallocation, you can
convert them to unallocated and compressed images as I showed
in the previous mail.

Here is an example with the preallocated images, like the one from virt-manager.

I'll create file with some data - I'm using data that will compress well
to make the example more convincing :-)

# dd if=/dev/zero bs=8M count=128 | tr "\0" "\1" > image.raw
128+0 records in
128+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.88171 s, 571 MB/s

Add some empty space:

# truncate -s 10g image.raw 

# ls -lhs image.raw 
1.1G -rw-r--r--. 1 root root 10G Dec  7 22:01 image.raw

Now lest create a qcow2 preallocated image like virt-manger does:

# qemu-img convert -f raw -O qcow2 -o preallocation=metadata image.raw image-preallocated-metadata.qcow2

# qemu-img info image-preallocated-metadata.qcow2 
image: image-preallocated-metadata.qcow2
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 1.0G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

# ls -lhs image*
1.1G -rw-r--r--. 1 root root 11G Dec  7 22:02 image-preallocated-metadata.qcow2
1.1G -rw-r--r--. 1 root root 10G Dec  7 22:01 image.raw

Finally, create a compressed qcow2, optimized for upload:

# qemu-img convert -c -f qcow2 -O qcow2 image-preallocated-metadata.qcow2 image-preallocated-metadata-optimized.qcow2 

# qemu-img info image-preallocated-metadata-optimized.qcow2 
image: image-preallocated-metadata-optimized.qcow2
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 1.6M
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

# ls -lhs image*
1.6M -rw-r--r--. 1 root root 1.7M Dec  7 22:04 image-preallocated-metadata-optimized.qcow2
1.1G -rw-r--r--. 1 root root  11G Dec  7 22:02 image-preallocated-metadata.qcow2
1.1G -rw-r--r--. 1 root root  10G Dec  7 22:01 image.raw

All the images contain exactly the same data.

Now, how can we optimize your image on upload? we can't.

When uploading from the UI, our code run in the browser
and we don't have access to your file system.

When using the oVirt SDK, you write the code for uploading so you can
do the conversion if want to save upload time and bandwidth and  can
spend the time to do the conversion.

We can provide an upload command line tool that will convert
and compress images before upload, and will use the oVirt SDK
to upload the images.

But I think it will be simpler and easier to maintain to support
compression when uploading or downloading data. This is
already supported by browsers and common tools like curl,
and it works with any file format without conversion.

I hope I convinced you this time ;-)

Nir
 

Thanks,
Gianluca