10:05 a.m.
On 05/30/2018 09:16 AM, Kevin Wolf wrote:
Am 30.05.2018 um 15:44 hat Eric Blake geschrieben:
> On 05/29/2018 04:18 PM, Nir Soffer wrote:
>>> You CAN get a logically collapsed view of storage (that is, what the
>>> guest would see), by using an NBD export of volume V. Reading from that
>>> volume will then pull sectors from whichever portion of the chain you
>>> need. You can use either qemu-nbd (if no guest is writing to the
>>> chain), or within a running qemu, you can use nbd-server-start and
>>> nbd-server-add (over QMP) to get such an NBD server running.
>>
>>
>> NBD expose the guest data, but we want the qcow2 stream - without
>> creating a new image.
>
> NBD can do both. You choose whether it exposes the guest data or the qcow2
> data, by whether the client or the server is interpreting qcow2 data.
But if I understand correctly, it doesn't result in the image Nir wants.
You would only export an existing qcow2 file, i.e. a single layer in the
backing chain, this way. The question was about a collapsed image, i.e.
the disk content as the guest sees it.
The problem is that qcow2 just isn't made to be streamable. Importing a
qcow2 stream without saving it into a temporary file (or a memory buffer
as large as the image file) simply isn't possible in the general case.
If I understood the question, we start with a local:
T (any format) <- S (qcow2) <- V (qcow2)
and want to create a remote tar file:
dest.tar == | header ... | qcow2 image |
where we write a single collapsed view of the T<-S<-V chain as a qcow2
image in the subset of the remote tar file.
So, first use qemu-img to learn how big to size the collapsed qcow2
image, and by extension, the overall tar image
$ qemu-img measure -f qcow2 -O qcow2 V
then pre-create a large enough tar file on the destination
$ create header
$ truncate --size=XXX dest.qcow2
$ tar cf dest.tar header dest.qcow2
(note that I explicitly did NOT use tar --sparse; dest.qcow2 is sparse
and occupies practically no disk space, but dest.tar must NOT be sparse
because neither tar nor NBD work well with after-the-fact resizing)
then set up an NBD server on the destination that can write to the
subset of the tar file:
$ learn the offset of dest.qcow2 within dest.tar (probably a multiple of
10240, given default GNU tar options)
$ qemu-nbd --image-opts
driver=raw,offset=YYY,size=XXX,file.driver=file,file.filename=dest.tar
(I'm not sure if I got the --image-opts syntax exactly correct. nbdkit
has more examples of learning offsets within a tar file, and may be a
better option as a server than qemu-nbd - but the point remains: serve
up the subset of the dest.tar file as raw bytes)
finally set up qemu as an NBD client on the source:
$ qemu-img convert -f qcow2 V -O qcow2 nbd://remote
(now the client collapses the qcow2 chain onto the source, and writes
that into a qcow2 subset of the tar file on the destination, where the
destination was already sized large enough to hold the qcow2 image, and
where no other temporary storage was needed other than the sparse
dest.qcow2 used in creating a large enough tar file)
Exporting to a stream is possible if we're allowed to make two
passes
over the source, but the existing QEMU code is useless for that because
it inherently requires seeking. I think if I had to get something like
this, I'd probably implement such an exporter as a script external to
QEMU.
Wait. What are we trying to stream? A qcow2 file, or what the guest
would see? If you stream just what the guest sees, then 'qemu-img map'
tells you which portions of which source files to read in order to
reconstruct data in the order it would be seen by the guest. But yeah,
an external exporter that takes a raw file, learns its size and where
the holes are, and then writes a trivial qcow2 header and appends
L1/L2/refcount tables on the end to convert the raw file into a
slightly-larger qcow2 file, might be a valid way to create a qcow2 file
from a two-pass read.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org