2:54 p.m.
This is a multi-part message in MIME format.
--------------8112B4A3C10F96F410CF2A58
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
I apparently was unable to connect the dots when I was working on this
yesterday.
So, just to test I now manually changed the size value in the meta file
67108864 --> 73924608
And after that I was able to import the vm.
So perhaps the real problem is in the export ?
Rgds Jonas
On 23/10/16 20:57, Jonas Israelsson wrote:
On 23/10/16 20:06, Nir Soffer wrote:
> On Sun, Oct 23, 2016 at 5:34 PM, Jonas Israelsson
> <jonas.israelsson(a)elementary.se> wrote:
>> Greetings.
>>
>> We are in the process of migrating from oVirt 3.6 to 4.0. To
>> properly test
>> 4.0 we have setup a parallel 4.0 environment.
>>
>> For the non critical vm:s we thought we try the "export vms --> move
>> storage
>> domain to the other DC --> import vms" method.
>>
>> While many imports are successful quite a few fails with 'low level
>> Image
>> copy failed'
>>
>> One of these vm impossible to import have the following disk layout.
>>
>> * Disk 1 - 100GB (Thin)
>>
>> * Disk2 - 32GB (Preallocated)
> According to the volume .meta file bellow, this is COW/SPARSE,
> not preallocated.
It's because I'm an idiot and gave you information about the wrong
disk. My apologizes..
$ /usr/bin/qemu-img.org info
/rhev/data-center/9d200b26-359e-48b6-972a-90da179e4829/61842ad9-42da-40a9-8ec8-dd7807a82916/images/9eb60288-27b6-4fb1-aef1-4246455d588e/ddf8b402-514c-4a3c-9683-26810a7c41c0
image:
/rhev/data-center/9d200b26-359e-48b6-972a-90da179e4829/61842ad9-42da-40a9-8ec8-dd7807a82916/images/9eb60288-27b6-4fb1-aef1-4246455d588e/ddf8b402-514c-4a3c-9683-26810a7c41c0
file format: raw
virtual size: 35G (37849399296 bytes)
disk size: 35G
[root@patty tmp]# cat
/rhev/data-center/9d200b26-359e-48b6-972a-90da179e4829/61842ad9-42da-40a9-8ec8-dd7807a82916/images/9eb60288-27b6-4fb1-aef1-4246455d588e/ddf8b402-514c-4a3c-9683-26810a7c41c0.meta
DOMAIN=61842ad9-42da-40a9-8ec8-dd7807a82916
VOLTYPE=LEAF
CTIME=1476880543
FORMAT=RAW
IMAGE=9eb60288-27b6-4fb1-aef1-4246455d588e
DISKTYPE=2
PUUID=00000000-0000-0000-0000-000000000000
LEGALITY=LEGAL
MTIME=0
POOL_UUID=
SIZE=67108864
TYPE=PREALLOCATED
DESCRIPTION=
EOF
>
> Can you share the original vm disk metadata before the export?
Could you please instruct me how to ? It's on a FC-LUN so it's then
hiding on a lv somewhere. I could perhaps just move it to an nfs data
domain .. ?
> Looking at the metadata before the export, after the export, and after
> the import, we can understand what is the root cause.
>
> It will be hard to find the metadata after the failed copy since vdsm
> try
> hard to clean up after errors, but the information should be available
> in vdsm log.
Yes I noticed, hence the qemu-img wrapper
>> * Disk3 - 32GB (Thin)
>>
>> Where the two thin disk (1 & 3) are successfully imported but disk2,
>> the
>> preallocated always fail.
>>
> ...
>> and from vdsm.log
>>
> ...
>> CopyImageError: low level Image copy failed: ('ecode=1, stdout=,
>> stderr=qemu-img: error while writing sector 73912303: No space left on
>> device\n, message=None',)
> We need log from the entire flow, starting at "Run and protect:
> copyImage..."
>
> ...
>> The first checking the size of the image (37849399296) , and the
>> second the
>> size of logical volume (34359738368) just created to hold this image.
>> And as you can see the volume is smaller in size than the image it
>> should
>> hold, whereas we are under the impression something made an incorrect
>> decision when creating that volume.
> The destination image size depend on the destination format. If the
> destination
> is preallocated, the logical volume size *must* be the virtual size
> (32G). If it is
> sparse, the logical volume should be the file size on the export
> domain (35G).
>
> According to your findings, we created a destination image for a
> preallocated
> disk (32G), and then tried to run "qemu-img convert" with qcow2
> format as
> both source and destination. However this is only a guess, since I
> don't have
> the log showing the actual qemu-img command.
12:37:15 685557156 --- Identifier: 51635 , Arguments: convert -p
-t none -T none -f raw
/rhev/data-center/9d200b26-359e-48b6-972a-90da179e4829/61842ad9-42da-40a9-8ec8-dd7807a82916/images/9eb60288-27b6-4fb1-aef1-4246455d588e/ddf8b402-514c-4a3c-9683-26810a7c41c0
-O raw
/rhev/data-center/mnt/blockSD/cb64e1fc-98b6-4b8c-916e-418d05bcd467/images/a1d70c22-cace-48d2-9809-caadc70b77e7/71f5fe82-81dd-47e9-aa3f-1a66622db4cb
> Please share complete engine and vdsm logs showing the entire flow.
http://whs1.elementary.se/logs.tar.gz
In vdsm.log search for 12:37:15
--------------8112B4A3C10F96F410CF2A58
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>I apparently was unable to connect the dots when I was working on
this yesterday.</p>
<p>So, just to test I now manually changed the size value in the
meta file<br>
</p>
<p><span style="color: rgb(38, 50, 56); font-family: Roboto, Arial,
sans-serif; font-size: 13px; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing: normal;
line-height: 16px; orphans: auto; text-align: left; text-indent:
0px; text-transform: none; white-space: normal; widows: 1;
word-spacing: 0px; -webkit-text-stroke-width: 0px; display:
inline !important; float: none;">67108864 --> 73924608<br>
</span></p>
And after that I was able to import the vm.<br>
<br>
So perhaps the real problem is in the export ?<br>
<br>
Rgds Jonas<br>
<br>
<div class="moz-cite-prefix"><br>
On 23/10/16 20:57, Jonas Israelsson wrote:<br>
</div>
<blockquote
cite="mid:872832cc-5330-92ab-1733-e7fbc1774cd9@elementary.se"
type="cite">On 23/10/16 20:06, Nir Soffer wrote:
<br>
<br>
<blockquote type="cite">On Sun, Oct 23, 2016 at 5:34 PM, Jonas
Israelsson
<br>
<a class="moz-txt-link-rfc2396E"
href="mailto:jonas.israelsson@elementary.se"><jonas.israelsson@elementary.se></a>
wrote:
<br>
<blockquote type="cite">Greetings.
<br>
<br>
We are in the process of migrating from oVirt 3.6 to 4.0. To
properly test
<br>
4.0 we have setup a parallel 4.0 environment.
<br>
<br>
For the non critical vm:s we thought we try the "export vms
--> move storage
<br>
domain to the other DC --> import vms" method.
<br>
<br>
While many imports are successful quite a few fails with 'low
level Image
<br>
copy failed'
<br>
<br>
One of these vm impossible to import have the following disk
layout.
<br>
<br>
* Disk 1 - 100GB (Thin)
<br>
<br>
* Disk2 - 32GB (Preallocated)
<br>
</blockquote>
According to the volume .meta file bellow, this is COW/SPARSE,
<br>
not preallocated.
<br>
</blockquote>
It's because I'm an idiot and gave you information about the wrong
disk. My apologizes..
<br>
<br>
$ /usr/bin/qemu-img.org info
/rhev/data-center/9d200b26-359e-48b6-972a-90da179e4829/61842ad9-42da-40a9-8ec8-dd7807a82916/images/9eb60288-27b6-4fb1-aef1-4246455d588e/ddf8b402-514c-4a3c-9683-26810a7c41c0<br>
<br>
image:
/rhev/data-center/9d200b26-359e-48b6-972a-90da179e4829/61842ad9-42da-40a9-8ec8-dd7807a82916/images/9eb60288-27b6-4fb1-aef1-4246455d588e/ddf8b402-514c-4a3c-9683-26810a7c41c0<br>
file format: raw
<br>
virtual size: 35G (37849399296 bytes)
<br>
disk size: 35G
<br>
<br>
<br>
[root@patty tmp]# cat
/rhev/data-center/9d200b26-359e-48b6-972a-90da179e4829/61842ad9-42da-40a9-8ec8-dd7807a82916/images/9eb60288-27b6-4fb1-aef1-4246455d588e/ddf8b402-514c-4a3c-9683-26810a7c41c0.meta
<br>
DOMAIN=61842ad9-42da-40a9-8ec8-dd7807a82916
<br>
VOLTYPE=LEAF
<br>
CTIME=1476880543
<br>
FORMAT=RAW
<br>
IMAGE=9eb60288-27b6-4fb1-aef1-4246455d588e
<br>
DISKTYPE=2
<br>
PUUID=00000000-0000-0000-0000-000000000000
<br>
LEGALITY=LEGAL
<br>
MTIME=0
<br>
POOL_UUID=
<br>
SIZE=67108864
<br>
TYPE=PREALLOCATED
<br>
DESCRIPTION=
<br>
EOF
<br>
<br>
<br>
<br>
<blockquote type="cite">
<br>
Can you share the original vm disk metadata before the export?
<br>
</blockquote>
Could you please instruct me how to ? It's on a FC-LUN so it's
then hiding on a lv somewhere. I could perhaps just move it to an
nfs data domain .. ?
<br>
<blockquote type="cite">Looking at the metadata before the export,
after the export, and after
<br>
the import, we can understand what is the root cause.
<br>
<br>
It will be hard to find the metadata after the failed copy since
vdsm try
<br>
hard to clean up after errors, but the information should be
available
<br>
in vdsm log.
<br>
</blockquote>
Yes I noticed, hence the qemu-img wrapper
<br>
<blockquote type="cite">
<blockquote type="cite">* Disk3 - 32GB (Thin)
<br>
<br>
Where the two thin disk (1 & 3) are successfully imported
but disk2, the
<br>
preallocated always fail.
<br>
<br>
</blockquote>
...
<br>
<blockquote type="cite">and from vdsm.log
<br>
<br>
</blockquote>
...
<br>
<blockquote type="cite">CopyImageError: low level Image copy
failed: ('ecode=1, stdout=,
<br>
stderr=qemu-img: error while writing sector 73912303: No space
left on
<br>
device\n, message=None',)
<br>
</blockquote>
We need log from the entire flow, starting at "Run and protect:
copyImage..."
<br>
<br>
...
<br>
<blockquote type="cite">The first checking the size of the image
(37849399296) , and the second the
<br>
size of logical volume (34359738368) just created to hold this
image.
<br>
And as you can see the volume is smaller in size than the
image it should
<br>
hold, whereas we are under the impression something made an
incorrect
<br>
decision when creating that volume.
<br>
</blockquote>
The destination image size depend on the destination format. If
the destination
<br>
is preallocated, the logical volume size *must* be the virtual
size
<br>
(32G). If it is
<br>
sparse, the logical volume should be the file size on the export
domain (35G).
<br>
<br>
According to your findings, we created a destination image for a
preallocated
<br>
disk (32G), and then tried to run "qemu-img convert" with qcow2
format as
<br>
both source and destination. However this is only a guess, since
I don't have
<br>
the log showing the actual qemu-img command.
<br>
</blockquote>
12:37:15 685557156 --- Identifier: 51635 , Arguments: convert
-p -t none -T none -f raw
/rhev/data-center/9d200b26-359e-48b6-972a-90da179e4829/61842ad9-42da-40a9-8ec8-dd7807a82916/images/9eb60288-27b6-4fb1-aef1-4246455d588e/ddf8b402-514c-4a3c-9683-26810a7c41c0
-O raw
/rhev/data-center/mnt/blockSD/cb64e1fc-98b6-4b8c-916e-418d05bcd467/images/a1d70c22-cace-48d2-9809-caadc70b77e7/71f5fe82-81dd-47e9-aa3f-1a66622db4cb<br>
<blockquote type="cite">Please share complete engine and vdsm logs
showing the entire flow.
<br>
</blockquote>
<a class="moz-txt-link-freetext"
href="http://whs1.elementary.se/logs.tar.gz">http://whs1.elementary.se/logs.tar.gz</a>
<br>
In vdsm.log search for 12:37:15
<br>
</blockquote>
<br>
</body>
</html>
--------------8112B4A3C10F96F410CF2A58--