On 05/29/2018 04:11 PM, Nir Soffer wrote:
I think real streaming is unlikely to happen because most image formats that QEMU supports aren't made that way. If there is a compelling reason, we can consider it, but it would work only with very few target formats and as such would have to be separate from existing commands.
Real streaming is exactly what we want, and we need it only for qcow2 format, because it is our preferred way to pack images in ova.
We have 2 possible use cases:
Exporting images or ova files:
image in any format -> qemu-img -> [qcow2 byte stream] -> imageio http server -> http client
image in any format -> qemu-img measure (to learn how large to size qcow2) then create destination qcow2 file that large and serve it over NBD as raw (perhaps using an nbdkit plugin for this part) then qemu-img convert to destination format qcow2 as NBD client So, as long as your NBD server (via nbdkit plugin) can talk to imageio http server -> http client, and sized things properly according to qemu-img measure, then qemu-img can write qcow2 (rather than it's more usual raw) over the NBD connection, and when the process is complete, the http client will have a fully-populated qcow2 file with no temporary files created in the meantime.
Importing images or ova files:
http client -> imageio http server -> [qcow2 byte stream] -> qemu-img -> image in any format
Same sort of thing - as long as the NBD server is serving a qcow2 file as raw data, then the NBD client is interpreting that data as qcow2, then qemu-img convert should be able to convert that qcow2 stream into any format. Or, put another way, usually you do the conversion from qcow2 to raw at the server, then the client sees raw bytes: qemu-nbd -f qcow2 file.qcow2 # expose only the guest-visible bytes... qemu-img convert -f raw nbd://host output # and write those bytes but in this case, you'd be serving raw bytes at the server, and letting the client do qcow2 conversion: qemu-nbd -f raw file.qcow2 # expose the full qcow2 file... qemu-img convert -f qcow2 nbd://host output # and extract the guest view where using nbdkit instead of qemu-nbd as your point of contact with the imageio http server may make more sense.
If we will have this, we don't need to create temporary storage space and we can avoid several image copies in the process.
This will also improve the user experience, avoiding the wait until a ova is created before the user can start downloading it.
As for OVA files, I think it might be useful to have a tar block driver
instead which would allow you to open a file inside a tar archive (you could then also directly run an OVA without extracting it first). We probably wouldn't be able to support resizing images there, but that should be okay.
If you can create a tar file that reserves space for the image file without actually writing it, a possible workaround today would be using the offset/size runtime options of the raw driver to convert directly into a region inside the tar archive.
What are the offset/size runtime options? I cannot find anything about them in man qemu-img.
## # @BlockdevOptionsRaw: # # Driver specific block device options for the raw driver. # # @offset: position where the block device starts # @size: the assumed size of the device # # Since: 2.9 ## { 'struct': 'BlockdevOptionsRaw', 'base': 'BlockdevOptionsGenericFormat', 'data': { '*offset': 'int', '*size': 'int' } } Yeah, it's a pity that "qemu-img create -o help -f raw" has forgotten to document them, the way "qemu-img create -o help -f qcow2" does for its options, so we should fix that. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org