On 05/29/2018 04:11 PM, Nir Soffer wrote:

>> I think real streaming is unlikely to happen because most image formats
>> that QEMU supports aren't made that way. If there is a compelling
>> reason, we can consider it, but it would work only with very few target
>> formats and as such would have to be separate from existing commands.
>>
>
> Real streaming is exactly what we want, and we need it only for qcow2
> format,
> because it is our preferred way to pack images in ova.
>
> We have 2 possible use cases:
>
> Exporting images or ova files:
>
> image in any format -> qemu-img -> [qcow2 byte stream] -> imageio http
> server -> http client

image in any format -> qemu-img measure (to learn how large to size qcow2)
then create destination qcow2 file that large

and serve it over NBD as
raw (perhaps using an nbdkit plugin for this part)
then qemu-img convert to destination format qcow2 as NBD client

So, as long as your NBD server (via nbdkit plugin) can talk to imageio
http server -> http client, and sized things properly according to
qemu-img measure, then qemu-img can write qcow2 (rather than it's more
usual raw) over the NBD connection, and when the process is complete,
the http client will have a fully-populated qcow2 file with no temporary
files created in the meantime.

>
> Importing images or ova files:
>
> http client -> imageio http server -> [qcow2 byte stream] -> qemu-img ->
> image in any format

Same sort of thing - as long as the NBD server is serving a qcow2 file
as raw data, then the NBD client is interpreting that data as qcow2,
then qemu-img convert should be able to convert that qcow2 stream into
any format.

Or, put another way, usually you do the conversion from qcow2 to raw at
the server, then the client sees raw bytes:

qemu-nbd -f qcow2 file.qcow2 # expose only the guest-visible bytes... 

qemu-img convert -f raw nbd://host output # and write those bytes 

but in this case, you'd be serving raw bytes at the server, and letting
the client do qcow2 conversion:

qemu-nbd -f raw file.qcow2 # expose the full qcow2 file...
qemu-img convert -f qcow2 nbd://host output # and extract the guest view

where using nbdkit instead of qemu-nbd as your point of contact with the
imageio http server may make more sense.

>
> If we will have this, we don't need to create temporary storage space and we
> can avoid several image copies in the process.
>
> This will also improve the user experience, avoiding the wait until
> a ova is created before the user can start downloading it.
>
> As for OVA files, I think it might be useful to have a tar block driver
>> instead which would allow you to open a file inside a tar archive (you
>> could then also directly run an OVA without extracting it first). We
>> probably wouldn't be able to support resizing images there, but that
>> should be okay.
>>
>> If you can create a tar file that reserves space for the image file
>> without actually writing it, a possible workaround today would be using
>> the offset/size runtime options of the raw driver to convert directly
>> into a region inside the tar archive.
>>
>
> What are the offset/size runtime options? I cannot find anything about
> them in man qemu-img.

##
# @BlockdevOptionsRaw:
#
# Driver specific block device options for the raw driver.
#
# @offset:      position where the block device starts
# @size:        the assumed size of the device
#
# Since: 2.9
##
{ 'struct': 'BlockdevOptionsRaw',
   'base': 'BlockdevOptionsGenericFormat',
   'data': { '*offset': 'int', '*size': 'int' } }


Yeah, it's a pity that "qemu-img create -o help -f raw" has forgotten to
document them, the way "qemu-img create -o help -f qcow2" does for its
options, so we should fix that.