[ovirt-users] Problem with backing-file, how to fix the backing-chain ?

Claudio Soprano Claudio.Soprano at lnf.infn.it
Fri Oct 14 05:43:50 EDT 2016


Hi all,

We run an ovirt environment before with engine v3.6.5 (if remember good) 
and now with v4.0.4 (we upgraded because we read the bug with 
backing-file was resolved with v4).

We upgraded some of the hosts machines (but not all still) at v4.0.4 too 
to see if this would fix the problem, but nothing.

The problem is that we have several VMs with snapshots, we do daily, 
weekly and monthly snapshots, keep some of them (usually the fresh ones) 
and remove the olds (that in the case they are weekly snapshots, they 
are in the middle of a series of snapshots), this in the time has 
produced the famous

Backing file too long bug.

So we upgraded the engine from 3.6.5 to 4.0.4 (latest available).

We discovered this bug, when we tried to upgrade an host to v4.0.4, 
doing so a VM in the host didn't migrate, so we shutdown it and tried to 
run on another host, but never succeded for the bug.

We don't know if we have more VMs in this situation because we upgraded 
only 2 hosts on 10.

Investigating the problem we discovered that the backing file indicated 
in each of LVM snapshots report a path very long with 
/dev/storage-domain-id/../image-group-id/ with ../image-group-id/ 
repeated a lot of times and at the end /parentid.

So to understand which was the right path that it would contain, we 
cloned a VM in the v4.0.4 and then we did 4 snapshots, now the backing 
file path is

/dev/storage-domain-id/parentid

Is there a way to modify the path in the backing-file or a way to 
recover the VM from this state ?

Where do reside the informations about the backing-file path ?

I attach here all the commands we run

On the ovirt manager (host with the engine only) we run

ovirt-shell

[oVirt shell (connected)]# list disks --parent-vm-name vm1

id         : 2df25a13-6958-40a8-832f-9a26ce65de0f
name       : vm1_Disk2

id         : 8cda0aa6-9e25-4b50-ba00-b877232a1983
name       : vm1_Disk1

[oVirt shell (connected)]# show disk 8cda0aa6-9e25-4b50-ba00-b877232a1983

id                               : 8cda0aa6-9e25-4b50-ba00-b877232a1983
name                             : vm1_Disk1
actual_size                      : 1073741824
alias                            : vm1_Disk1
disk_profile-id                  : 1731f79a-5034-4270-9a87-94d93025deac
format                           : cow
image_id                         : 7b354e2a-2099-4f2a-80b7-fba7d1fd13ee
propagate_errors                 : False
provisioned_size                 : 17179869184
shareable                        : False
size                             : 17179869184
sparse                           : True
status-state                     : ok
storage_domains-storage_domain-id: 384f9059-ef2f-4d43-a54f-de71c5d589c8
storage_type                     : image
wipe_after_delete                : False

[root at ovc1mgr ~]# su - postgres
Last login: Fri Oct 14 01:02:14 CEST 2016
-bash-4.2$ psql -d engine -U postgres
psql (9.2.15)
Type "help" for help.

engine=#\x on
engine=# select * from images where image_group_id = 
'8cda0aa6-9e25-4b50-ba00-b877232a1983' order by creation_date;

-[ RECORD 1 ]---------+-------------------------------------
image_guid            | 60ba7acf-58cb-475b-b9ee-15b1be99fee6
creation_date         | 2016-03-29 15:12:34+02
size                  | 17179869184
it_guid               | 00000000-0000-0000-0000-000000000000
parentid              | 00000000-0000-0000-0000-000000000000
imagestatus           | 4
lastmodified          | 2016-04-21 11:25:59.972+02
vm_snapshot_id        | 27c187cd-989f-4f7a-ac05-49c4410de6c2
volume_type           | 1
volume_format         | 5
image_group_id        | 8cda0aa6-9e25-4b50-ba00-b877232a1983
_create_date          | 2016-03-29 15:12:31.994065+02
_update_date          | 2016-09-04 01:10:08.773649+02
active                | f
volume_classification | 1

-[ RECORD 2 ]---------+-------------------------------------
image_guid            | 68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e
creation_date         | 2016-07-03 01:01:30+02
size                  | 17179869184
it_guid               | 00000000-0000-0000-0000-000000000000
parentid              | 60ba7acf-58cb-475b-b9ee-15b1be99fee6
imagestatus           | 1
lastmodified          | 2016-07-04 01:03:33.732+02
vm_snapshot_id        | 175c2071-a06b-4b0e-a069-5cc4bb236a34
volume_type           | 2
volume_format         | 4
image_group_id        | 8cda0aa6-9e25-4b50-ba00-b877232a1983
_create_date          | 2016-07-03 01:01:15.069585+02
_update_date          | 2016-09-11 02:06:04.420965+02
active                | f
volume_classification | 1

-[ RECORD 3 ]---------+-------------------------------------
image_guid            | 37ca6494-e990-44e5-8597-28845a0a19b5
creation_date         | 2016-08-07 01:06:15+02
size                  | 17179869184
it_guid               | 00000000-0000-0000-0000-000000000000
parentid              | 68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e
imagestatus           | 1
lastmodified          | 2016-08-08 01:00:03.778+02
vm_snapshot_id        | 4c0e5ac0-2ef3-4996-b3e9-7fd566d97b1a
volume_type           | 2
volume_format         | 4
image_group_id        | 8cda0aa6-9e25-4b50-ba00-b877232a1983
_create_date          | 2016-08-07 01:06:01.777156+02
_update_date          | 2016-09-25 01:55:54.090026+02
active                | f
volume_classification | 1

I removed other 10 snapshot infos from the list to easily read all

On a host running v4.0.4 we run

root at ovc2n06 ~]# lvchange -aey -pr 
384f9059-ef2f-4d43-a54f-de71c5d589c8/60ba7acf-58cb-475b-b9ee-15b1be99fee6
   Logical volume "60ba7acf-58cb-475b-b9ee-15b1be99fee6" changed.

This is the base image infact it doesn't contain a backing-file

[root at ovc2n06 ~]# qemu-img info --backing-chain 
/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/60ba7acf-58cb-475b-b9ee-15b1be99fee6
image: 
/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/60ba7acf-58cb-475b-b9ee-15b1be99fee6
file format: raw
virtual size: 16G (17179869184 bytes)
disk size: 0

[root at ovc2n06 ~]# lvchange -aey -pr 
384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e
   Logical volume "68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e" changed.

[root at ovc2n06 ~]# qemu-img info --backing-chain 
/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e
qemu-img: Could not open 
'/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1be99fee6': 
Could not open 
'/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1be99fee6': 
No such file or directory

To fix this problem we made a link

[root at ovc2n06 ~]# lvchange -aey -pr 
384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e
   Logical volume "68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e" changed.
[root at ovc2n06 ~]# qemu-img info --backing-chain 
/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e
qemu-img: Could not open 
'/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1be99fee6': 
Could not open 
'/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1be99fee6': 
No such file or directory
[root at ovc2n06 ~]# ln -s /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8 
/dev/8cda0aa6-9e25-4b50-ba00-b877232a1983
[root at ovc2n06 ~]# qemu-img info --backing-chain 
/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e
image: 
/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e
file format: qcow2
virtual size: 16G (17179869184 bytes)
disk size: 0
cluster_size: 65536
backing file: 
../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1be99fee6 
(actual path: 
/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1be99fee6)
backing file format: raw
Format specific information:
     compat: 0.10
     refcount bits: 16

image: 
/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1be99fee6
file format: raw
virtual size: 16G (17179869184 bytes)
disk size: 0

We moved to the next snapshot

[root at ovc2n06 ~]# lvchange -aey -pr 
384f9059-ef2f-4d43-a54f-de71c5d589c8/37ca6494-e990-44e5-8597-28845a0a19b5
   Logical volume "37ca6494-e990-44e5-8597-28845a0a19b5" changed.

And now the error from the bug

qemu-img info --backing-chain 
/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/37ca6494-e990-44e5-8597-28845a0a19b5
qemu-img: Could not open 
'/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/37ca6494-e990-44e5-8597-28845a0a19b5': 
Backing file name too long

Any snapshot starting from this snapshot is not usable, the VM was 
running ok up to we did shutdown it, it never starts again.

How you can see the path of the backing-file is wrong and too long 
already in the previous snapshots, is there a way to fix it or to edit 
it manually ?

Obviously we tried to clone, export, create a qcow2 image, from all the 
snapshot later 07-august but the operation didn't complete, we can 
recover only from the snapshot of 7-august that is missing 2 months of 
new data.

Please if you have a workaround or a solution, can you write the 
commands we need to run with examples, we searched a lot about 
backing-file but only the manual of the qemu-img command was found with 
no examples on how to recover or change it.

Thanks again
Claudio Soprano


-- 

        /        |    /   _____/           /    |    /   _____/    |    /
       /       / |   /   /                /   / |   /   /        / |   /
      /       /  |  /   ___/   _____/    /   /  |  /   ___/     /  |  /
     /       /   | /   /                /   /   | /   /        /   | /
   ______/ _/   __/  _/               _/  _/   __/  _/       _/   __/

Claudio Soprano                phone:  (+39)-06-9403.2349/2355
Computing Service              fax:    (+39)-06-9403.2649
LNF-INFN                       e-mail: Claudio.Soprano at lnf.infn.it
Via Enrico Fermi, 40           www:    http://www.lnf.infn.it/
I-00044 Frascati, Italy




More information about the Users mailing list