On Mon, May 28, 2018 at 2:38 PM Kevin Wolf <kwolf@redhat.com> wrote:
Am 28.05.2018 um 12:27 hat Arik Hadas geschrieben:
> On Mon, May 28, 2018 at 11:25 AM, Kevin Wolf <kwolf@redhat.com> wrote:
>
> > [ Adding qemu-block ]
> >
> > Am 27.05.2018 um 10:36 hat Arik Hadas geschrieben:
> > > On Thu, May 24, 2018 at 6:13 PM, Nir Soffer <nsoffer@redhat.com> wrote:
> > >
> > > > On Thu, May 24, 2018 at 6:06 PM Vrgotic, Marko <
> > M.Vrgotic@activevideo.com>
> > > > wrote:
> > > >
> > > >> Dear Nir,
> > > >>
> > > >> Thank you for quick reply.
> > > >>
> > > >> Ok, why it will not work?
> > > >>
> > > >
> > > > Because the image has a backing file which is not accessible to oVirt.
> > > >
> > > >
> > > >> I used qemu+tcp connection, via import method through engine admin UI.
> > > >>
> > > >> Images was imported and converted according logs, still “backing file”
> > > >> invalid entry remained.
> > > >>
> > > >> Also, I did use same method before, connecting to plain “libvirt kvm”
> > > >> host, import and conversion went smooth, no backend file.
> > > >>
> > > >> Image format is qcow(2) which is supported by oVirt.
> > > >>
> > > >> What am I missing? Should I use different method?
> > > >>
> > > >
> > > > I guess this is not a problem on your side, but a bug in our side.
> > > >
> > > > Either we should block the operation that cannot work, or fix the
> > process
> > > > so we don't refer to non-existing image.
> > > >
> > > > When importing we have 2 options:
> > > >
> > > > - import the entire chain,  importing all images in the chain,
> > converting
> > > >  each image to oVirt volume, and updating the backing file of each
> > layer
> > > > to point to the oVirt image.
> > > >
> > > > - import the current state of the image into a new image, using either
> > raw
> > > > or qcow2, but without any backing file.
> > > >
> > > > Arik, do you know why we create qcow2 file with invalid backing file?
> > > >
> > >
> > > It seems to be a result of a bit naive behavior of the kvm2ovirt module
> > > that tries to download only the top-level volume the VM uses, assuming
> > each
> > > of the disks to be imported is comprised of a single volume.
> > >
> > > Maybe it's time to finally asking QEMU guys to provide a way to consume
> > the
> > > 'collapsed' form of a chain of volumes as a stream if that's not
> > available
> > > yet? ;) It can also boost the recently added process of exporting VMs as
> > > OVAs...
> >
> > Not sure which operation we're talking about on the QEMU level, but
> > generally the "collapsed" view is the normal thing because that's what
> > guests see.
> >
> > For example, if you use 'qemu-img convert', you have to pass options to
> > specifically disable it and convert only a single layer if you want to
> > keep using backing files instead of getting a standalone image that
> > contains everything.
> >
>
> Yeah, some context was missing. Sorry about that.
>
> Let me demonstrate briefly the flow for OVA:
> Let's say that we have a VM that is based on a template and has one disk
> and one snapshot, so its volume-chain would be:
> T -> S -> V
> (V is the volume the VM writes to, S is the backing file of V and T is the
> backing file of S).
> When exporting that VM to an OVA file we want the produced tar file to be
> comprised of:
> (1) OVF configuration
> (2) single disk volume (preferably qcow).
>
> So we need to collapse T, S, V into a single volume.
> Sure, we can do 'qemu-img convert'. That's what we do now in oVirt 4.2:
> (a) qemu-img convert produces a 'temporary' collapsed volume
> (b) make a tar file of the OVf configuration and that 'temporary' volume
> (c) delete the temporary volume
>
> But the fact that we produce that 'temporary' volume obviously slows down
> the entire operation.
> It would be much better if we could "open" a stream that we can read from
> the 'collapsed' form of that chain and stream it directly into the
> appropriate tar file entry, without extra writes to the storage device.
>
> Few months ago people from the oVirt-storage team checked the qemu toolset
> and replied that this capability is not yet provided, therefore we
> implemented the workaround described above. Apparently, the desired ability
> can also be useful for the flow discussed in this thread so it worth asking
> for it again :)

I think real streaming is unlikely to happen because most image formats
that QEMU supports aren't made that way. If there is a compelling
reason, we can consider it, but it would work only with very few target
formats and as such would have to be separate from existing commands.

Real streaming is exactly what we want, and we need it only for qcow2 format,
because it is our preferred way to pack images in ova.

We have 2 possible use cases:

Exporting images or ova files:

image in any format -> qemu-img -> [qcow2 byte stream] -> imageio http server -> http client

Importing images or ova files:

http client -> imageio http server -> [qcow2 byte stream] -> qemu-img -> image in any format

If we will have this, we don't need to create temporary storage space and we
can avoid several image copies in the process.

This will also improve the user experience, avoiding the wait until
a ova is created before the user can start downloading it.

As for OVA files, I think it might be useful to have a tar block driver
instead which would allow you to open a file inside a tar archive (you
could then also directly run an OVA without extracting it first). We
probably wouldn't be able to support resizing images there, but that
should be okay.

If you can create a tar file that reserves space for the image file
without actually writing it, a possible workaround today would be using
the offset/size runtime options of the raw driver to convert directly
into a region inside the tar archive.

What are the offset/size runtime options? I cannot find anything about
them in man qemu-img.

Nir