Problem with backing-file, how to fix the backing-chain ?
Hi all, We run an ovirt environment before with engine v3.6.5 (if remember good) and now with v4.0.4 (we upgraded because we read the bug with backing-file was resolved with v4). We upgraded some of the hosts machines (but not all still) at v4.0.4 too to see if this would fix the problem, but nothing. The problem is that we have several VMs with snapshots, we do daily, weekly and monthly snapshots, keep some of them (usually the fresh ones) and remove the olds (that in the case they are weekly snapshots, they are in the middle of a series of snapshots), this in the time has produced the famous Backing file too long bug. So we upgraded the engine from 3.6.5 to 4.0.4 (latest available). We discovered this bug, when we tried to upgrade an host to v4.0.4, doing so a VM in the host didn't migrate, so we shutdown it and tried to run on another host, but never succeded for the bug. We don't know if we have more VMs in this situation because we upgraded only 2 hosts on 10. Investigating the problem we discovered that the backing file indicated in each of LVM snapshots report a path very long with /dev/storage-domain-id/../image-group-id/ with ../image-group-id/ repeated a lot of times and at the end /parentid. So to understand which was the right path that it would contain, we cloned a VM in the v4.0.4 and then we did 4 snapshots, now the backing file path is /dev/storage-domain-id/parentid Is there a way to modify the path in the backing-file or a way to recover the VM from this state ? Where do reside the informations about the backing-file path ? I attach here all the commands we run On the ovirt manager (host with the engine only) we run ovirt-shell [oVirt shell (connected)]# list disks --parent-vm-name vm1 id : 2df25a13-6958-40a8-832f-9a26ce65de0f name : vm1_Disk2 id : 8cda0aa6-9e25-4b50-ba00-b877232a1983 name : vm1_Disk1 [oVirt shell (connected)]# show disk 8cda0aa6-9e25-4b50-ba00-b877232a1983 id : 8cda0aa6-9e25-4b50-ba00-b877232a1983 name : vm1_Disk1 actual_size : 1073741824 alias : vm1_Disk1 disk_profile-id : 1731f79a-5034-4270-9a87-94d93025deac format : cow image_id : 7b354e2a-2099-4f2a-80b7-fba7d1fd13ee propagate_errors : False provisioned_size : 17179869184 shareable : False size : 17179869184 sparse : True status-state : ok storage_domains-storage_domain-id: 384f9059-ef2f-4d43-a54f-de71c5d589c8 storage_type : image wipe_after_delete : False [root@ovc1mgr ~]# su - postgres Last login: Fri Oct 14 01:02:14 CEST 2016 -bash-4.2$ psql -d engine -U postgres psql (9.2.15) Type "help" for help. engine=#\x on engine=# select * from images where image_group_id = '8cda0aa6-9e25-4b50-ba00-b877232a1983' order by creation_date; -[ RECORD 1 ]---------+------------------------------------- image_guid | 60ba7acf-58cb-475b-b9ee-15b1be99fee6 creation_date | 2016-03-29 15:12:34+02 size | 17179869184 it_guid | 00000000-0000-0000-0000-000000000000 parentid | 00000000-0000-0000-0000-000000000000 imagestatus | 4 lastmodified | 2016-04-21 11:25:59.972+02 vm_snapshot_id | 27c187cd-989f-4f7a-ac05-49c4410de6c2 volume_type | 1 volume_format | 5 image_group_id | 8cda0aa6-9e25-4b50-ba00-b877232a1983 _create_date | 2016-03-29 15:12:31.994065+02 _update_date | 2016-09-04 01:10:08.773649+02 active | f volume_classification | 1 -[ RECORD 2 ]---------+------------------------------------- image_guid | 68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e creation_date | 2016-07-03 01:01:30+02 size | 17179869184 it_guid | 00000000-0000-0000-0000-000000000000 parentid | 60ba7acf-58cb-475b-b9ee-15b1be99fee6 imagestatus | 1 lastmodified | 2016-07-04 01:03:33.732+02 vm_snapshot_id | 175c2071-a06b-4b0e-a069-5cc4bb236a34 volume_type | 2 volume_format | 4 image_group_id | 8cda0aa6-9e25-4b50-ba00-b877232a1983 _create_date | 2016-07-03 01:01:15.069585+02 _update_date | 2016-09-11 02:06:04.420965+02 active | f volume_classification | 1 -[ RECORD 3 ]---------+------------------------------------- image_guid | 37ca6494-e990-44e5-8597-28845a0a19b5 creation_date | 2016-08-07 01:06:15+02 size | 17179869184 it_guid | 00000000-0000-0000-0000-000000000000 parentid | 68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e imagestatus | 1 lastmodified | 2016-08-08 01:00:03.778+02 vm_snapshot_id | 4c0e5ac0-2ef3-4996-b3e9-7fd566d97b1a volume_type | 2 volume_format | 4 image_group_id | 8cda0aa6-9e25-4b50-ba00-b877232a1983 _create_date | 2016-08-07 01:06:01.777156+02 _update_date | 2016-09-25 01:55:54.090026+02 active | f volume_classification | 1 I removed other 10 snapshot infos from the list to easily read all On a host running v4.0.4 we run root@ovc2n06 ~]# lvchange -aey -pr 384f9059-ef2f-4d43-a54f-de71c5d589c8/60ba7acf-58cb-475b-b9ee-15b1be99fee6 Logical volume "60ba7acf-58cb-475b-b9ee-15b1be99fee6" changed. This is the base image infact it doesn't contain a backing-file [root@ovc2n06 ~]# qemu-img info --backing-chain /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/60ba7acf-58cb-475b-b9ee-15b1be99fee6 image: /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/60ba7acf-58cb-475b-b9ee-15b1be99fee6 file format: raw virtual size: 16G (17179869184 bytes) disk size: 0 [root@ovc2n06 ~]# lvchange -aey -pr 384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e Logical volume "68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e" changed. [root@ovc2n06 ~]# qemu-img info --backing-chain /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e qemu-img: Could not open '/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1be99fee6': Could not open '/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1be99fee6': No such file or directory To fix this problem we made a link [root@ovc2n06 ~]# lvchange -aey -pr 384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e Logical volume "68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e" changed. [root@ovc2n06 ~]# qemu-img info --backing-chain /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e qemu-img: Could not open '/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1be99fee6': Could not open '/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1be99fee6': No such file or directory [root@ovc2n06 ~]# ln -s /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8 /dev/8cda0aa6-9e25-4b50-ba00-b877232a1983 [root@ovc2n06 ~]# qemu-img info --backing-chain /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e image: /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e file format: qcow2 virtual size: 16G (17179869184 bytes) disk size: 0 cluster_size: 65536 backing file: ../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1be99fee6 (actual path: /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1be99fee6) backing file format: raw Format specific information: compat: 0.10 refcount bits: 16 image: /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1be99fee6 file format: raw virtual size: 16G (17179869184 bytes) disk size: 0 We moved to the next snapshot [root@ovc2n06 ~]# lvchange -aey -pr 384f9059-ef2f-4d43-a54f-de71c5d589c8/37ca6494-e990-44e5-8597-28845a0a19b5 Logical volume "37ca6494-e990-44e5-8597-28845a0a19b5" changed. And now the error from the bug qemu-img info --backing-chain /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/37ca6494-e990-44e5-8597-28845a0a19b5 qemu-img: Could not open '/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/37ca6494-e990-44e5-8597-28845a0a19b5': Backing file name too long Any snapshot starting from this snapshot is not usable, the VM was running ok up to we did shutdown it, it never starts again. How you can see the path of the backing-file is wrong and too long already in the previous snapshots, is there a way to fix it or to edit it manually ? Obviously we tried to clone, export, create a qcow2 image, from all the snapshot later 07-august but the operation didn't complete, we can recover only from the snapshot of 7-august that is missing 2 months of new data. Please if you have a workaround or a solution, can you write the commands we need to run with examples, we searched a lot about backing-file but only the manual of the qemu-img command was found with no examples on how to recover or change it. Thank you!
Have you tried a qemu-img 'unsafe' rebase? Let's say you have the chain: base <- middle <- top and due to the old backing chain links we have base parent: None middle parent: ../uuid/../uuid/../uuid/base top parent: ../uuid/../uuid/middle Since the volumes are always stored in the same directory, the repeated ../uuid/ parts of the path are unnecessary and can always be replaced with just the last part. Once making the LVs available on the system you should be able to fix the paths with the following commands: qemu-img rebase -u -b base middle qemu-img rebase -u -b middle top You can then verify the whole backing chain by: qemu-img info --backing-chain top Let me know if this helps. On 17/10/16 10:34 +0200, Dael Maselli wrote:
Hi all,
We run an ovirt environment before with engine v3.6.5 (if remember good) and now with v4.0.4 (we upgraded because we read the bug with backing-file was resolved with v4).
We upgraded some of the hosts machines (but not all still) at v4.0.4 too to see if this would fix the problem, but nothing.
The problem is that we have several VMs with snapshots, we do daily, weekly and monthly snapshots, keep some of them (usually the fresh ones) and remove the olds (that in the case they are weekly snapshots, they are in the middle of a series of snapshots), this in the time has produced the famous
Backing file too long bug.
So we upgraded the engine from 3.6.5 to 4.0.4 (latest available).
We discovered this bug, when we tried to upgrade an host to v4.0.4, doing so a VM in the host didn't migrate, so we shutdown it and tried to run on another host, but never succeded for the bug.
We don't know if we have more VMs in this situation because we upgraded only 2 hosts on 10.
Investigating the problem we discovered that the backing file indicated in each of LVM snapshots report a path very long with /dev/storage-domain-id/../image-group-id/ with ../image-group-id/ repeated a lot of times and at the end /parentid.
So to understand which was the right path that it would contain, we cloned a VM in the v4.0.4 and then we did 4 snapshots, now the backing file path is
/dev/storage-domain-id/parentid
Is there a way to modify the path in the backing-file or a way to recover the VM from this state ?
Where do reside the informations about the backing-file path ?
I attach here all the commands we run
On the ovirt manager (host with the engine only) we run
ovirt-shell
[oVirt shell (connected)]# list disks --parent-vm-name vm1
id : 2df25a13-6958-40a8-832f-9a26ce65de0f name : vm1_Disk2
id : 8cda0aa6-9e25-4b50-ba00-b877232a1983 name : vm1_Disk1
[oVirt shell (connected)]# show disk 8cda0aa6-9e25-4b50-ba00-b877232a1983
id : 8cda0aa6-9e25-4b50-ba00-b877232a1983 name : vm1_Disk1 actual_size : 1073741824 alias : vm1_Disk1 disk_profile-id : 1731f79a-5034-4270-9a87-94d93025deac format : cow image_id : 7b354e2a-2099-4f2a-80b7-fba7d1fd13ee propagate_errors : False provisioned_size : 17179869184 shareable : False size : 17179869184 sparse : True status-state : ok storage_domains-storage_domain-id: 384f9059-ef2f-4d43-a54f-de71c5d589c8 storage_type : image wipe_after_delete : False
[root@ovc1mgr ~]# su - postgres Last login: Fri Oct 14 01:02:14 CEST 2016 -bash-4.2$ psql -d engine -U postgres psql (9.2.15) Type "help" for help.
engine=#\x on engine=# select * from images where image_group_id = '8cda0aa6-9e25-4b50-ba00-b877232a1983' order by creation_date;
-[ RECORD 1 ]---------+------------------------------------- image_guid | 60ba7acf-58cb-475b-b9ee-15b1be99fee6 creation_date | 2016-03-29 15:12:34+02 size | 17179869184 it_guid | 00000000-0000-0000-0000-000000000000 parentid | 00000000-0000-0000-0000-000000000000 imagestatus | 4 lastmodified | 2016-04-21 11:25:59.972+02 vm_snapshot_id | 27c187cd-989f-4f7a-ac05-49c4410de6c2 volume_type | 1 volume_format | 5 image_group_id | 8cda0aa6-9e25-4b50-ba00-b877232a1983 _create_date | 2016-03-29 15:12:31.994065+02 _update_date | 2016-09-04 01:10:08.773649+02 active | f volume_classification | 1
-[ RECORD 2 ]---------+------------------------------------- image_guid | 68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e creation_date | 2016-07-03 01:01:30+02 size | 17179869184 it_guid | 00000000-0000-0000-0000-000000000000 parentid | 60ba7acf-58cb-475b-b9ee-15b1be99fee6 imagestatus | 1 lastmodified | 2016-07-04 01:03:33.732+02 vm_snapshot_id | 175c2071-a06b-4b0e-a069-5cc4bb236a34 volume_type | 2 volume_format | 4 image_group_id | 8cda0aa6-9e25-4b50-ba00-b877232a1983 _create_date | 2016-07-03 01:01:15.069585+02 _update_date | 2016-09-11 02:06:04.420965+02 active | f volume_classification | 1
-[ RECORD 3 ]---------+------------------------------------- image_guid | 37ca6494-e990-44e5-8597-28845a0a19b5 creation_date | 2016-08-07 01:06:15+02 size | 17179869184 it_guid | 00000000-0000-0000-0000-000000000000 parentid | 68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e imagestatus | 1 lastmodified | 2016-08-08 01:00:03.778+02 vm_snapshot_id | 4c0e5ac0-2ef3-4996-b3e9-7fd566d97b1a volume_type | 2 volume_format | 4 image_group_id | 8cda0aa6-9e25-4b50-ba00-b877232a1983 _create_date | 2016-08-07 01:06:01.777156+02 _update_date | 2016-09-25 01:55:54.090026+02 active | f volume_classification | 1
I removed other 10 snapshot infos from the list to easily read all
On a host running v4.0.4 we run
root@ovc2n06 ~]# lvchange -aey -pr 384f9059-ef2f-4d43-a54f-de71c5d589c8/60ba7acf-58cb-475b-b9ee-15b1be99fee6 Logical volume "60ba7acf-58cb-475b-b9ee-15b1be99fee6" changed.
This is the base image infact it doesn't contain a backing-file
[root@ovc2n06 ~]# qemu-img info --backing-chain /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/60ba7acf-58cb-475b-b9ee-15b1be99fee6 image: /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/60ba7acf-58cb-475b-b9ee-15b1be99fee6 file format: raw virtual size: 16G (17179869184 bytes) disk size: 0
[root@ovc2n06 ~]# lvchange -aey -pr 384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e Logical volume "68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e" changed.
[root@ovc2n06 ~]# qemu-img info --backing-chain /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e qemu-img: Could not open '/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b 1be99fee6': Could not open '/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b 1be99fee6': No such file or directory
To fix this problem we made a link
[root@ovc2n06 ~]# lvchange -aey -pr 384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e Logical volume "68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e" changed. [root@ovc2n06 ~]# qemu-img info --backing-chain /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e qemu-img: Could not open '/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b 1be99fee6': Could not open '/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b 1be99fee6': No such file or directory [root@ovc2n06 ~]# ln -s /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8 /dev/8cda0aa6-9e25-4b50-ba00-b877232a1983 [root@ovc2n06 ~]# qemu-img info --backing-chain /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e image: /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e file format: qcow2 virtual size: 16G (17179869184 bytes) disk size: 0 cluster_size: 65536 backing file: ../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1be99fee6 (actual path: /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1 be99fee6) backing file format: raw Format specific information: compat: 0.10 refcount bits: 16
image: /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1 be99fee6 file format: raw virtual size: 16G (17179869184 bytes) disk size: 0
We moved to the next snapshot
[root@ovc2n06 ~]# lvchange -aey -pr 384f9059-ef2f-4d43-a54f-de71c5d589c8/37ca6494-e990-44e5-8597-28845a0a19b5 Logical volume "37ca6494-e990-44e5-8597-28845a0a19b5" changed.
And now the error from the bug
qemu-img info --backing-chain /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/37ca6494-e990-44e5-8597-28845a0a19b5 qemu-img: Could not open '/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/37ca6494-e990-44e5-8597-28845a0a19b5': Backing file name too long
Any snapshot starting from this snapshot is not usable, the VM was running ok up to we did shutdown it, it never starts again.
How you can see the path of the backing-file is wrong and too long already in the previous snapshots, is there a way to fix it or to edit it manually ?
Obviously we tried to clone, export, create a qcow2 image, from all the snapshot later 07-august but the operation didn't complete, we can recover only from the snapshot of 7-august that is missing 2 months of new data.
Please if you have a workaround or a solution, can you write the commands we need to run with examples, we searched a lot about backing-file but only the manual of the qemu-img command was found with no examples on how to recover or change it.
Thank you!
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Adam Litke
Hi Adam, we tried your solution. This is our situation with the current VM that has 2 disks base -> snap1 -> snap2 -> snap3 -> snap4 -> snap5 -> .. -> snap15 for each disk We tried to do qemu-img rebase -u -b base snap1 results OK qemu-img rebase -u -b snap1 snap2 results: qemu-img: Could not open 'snap2': Backing file name too long our qemu version is qemu-img version 2.3.0 (qemu-kvm-ev-2.3.0-31.el7.16.1), Copyright (c) 2004-2008 Fabrice Bellard How do you think can we resolve ? Thank you Claudio Soprano On 17/10/16 21:57, Adam Litke wrote:
Have you tried a qemu-img 'unsafe' rebase?
Let's say you have the chain: base <- middle <- top
and due to the old backing chain links we have
base parent: None middle parent: ../uuid/../uuid/../uuid/base top parent: ../uuid/../uuid/middle
Since the volumes are always stored in the same directory, the repeated ../uuid/ parts of the path are unnecessary and can always be replaced with just the last part. Once making the LVs available on the system you should be able to fix the paths with the following commands:
qemu-img rebase -u -b base middle qemu-img rebase -u -b middle top
You can then verify the whole backing chain by: qemu-img info --backing-chain top
Let me know if this helps.
On 17/10/16 10:34 +0200, Dael Maselli wrote:
Hi all,
We run an ovirt environment before with engine v3.6.5 (if remember good) and now with v4.0.4 (we upgraded because we read the bug with backing-file was resolved with v4).
We upgraded some of the hosts machines (but not all still) at v4.0.4 too to see if this would fix the problem, but nothing.
The problem is that we have several VMs with snapshots, we do daily, weekly and monthly snapshots, keep some of them (usually the fresh ones) and remove the olds (that in the case they are weekly snapshots, they are in the middle of a series of snapshots), this in the time has produced the famous
Backing file too long bug.
So we upgraded the engine from 3.6.5 to 4.0.4 (latest available).
We discovered this bug, when we tried to upgrade an host to v4.0.4, doing so a VM in the host didn't migrate, so we shutdown it and tried to run on another host, but never succeded for the bug.
We don't know if we have more VMs in this situation because we upgraded only 2 hosts on 10.
Investigating the problem we discovered that the backing file indicated in each of LVM snapshots report a path very long with /dev/storage-domain-id/../image-group-id/ with ../image-group-id/ repeated a lot of times and at the end /parentid.
So to understand which was the right path that it would contain, we cloned a VM in the v4.0.4 and then we did 4 snapshots, now the backing file path is
/dev/storage-domain-id/parentid
Is there a way to modify the path in the backing-file or a way to recover the VM from this state ?
Where do reside the informations about the backing-file path ?
I attach here all the commands we run
On the ovirt manager (host with the engine only) we run
ovirt-shell
[oVirt shell (connected)]# list disks --parent-vm-name vm1
id : 2df25a13-6958-40a8-832f-9a26ce65de0f name : vm1_Disk2
id : 8cda0aa6-9e25-4b50-ba00-b877232a1983 name : vm1_Disk1
[oVirt shell (connected)]# show disk 8cda0aa6-9e25-4b50-ba00-b877232a1983
id : 8cda0aa6-9e25-4b50-ba00-b877232a1983 name : vm1_Disk1 actual_size : 1073741824 alias : vm1_Disk1 disk_profile-id : 1731f79a-5034-4270-9a87-94d93025deac format : cow image_id : 7b354e2a-2099-4f2a-80b7-fba7d1fd13ee propagate_errors : False provisioned_size : 17179869184 shareable : False size : 17179869184 sparse : True status-state : ok storage_domains-storage_domain-id: 384f9059-ef2f-4d43-a54f-de71c5d589c8 storage_type : image wipe_after_delete : False
[root@ovc1mgr ~]# su - postgres Last login: Fri Oct 14 01:02:14 CEST 2016 -bash-4.2$ psql -d engine -U postgres psql (9.2.15) Type "help" for help.
engine=#\x on engine=# select * from images where image_group_id = '8cda0aa6-9e25-4b50-ba00-b877232a1983' order by creation_date;
-[ RECORD 1 ]---------+------------------------------------- image_guid | 60ba7acf-58cb-475b-b9ee-15b1be99fee6 creation_date | 2016-03-29 15:12:34+02 size | 17179869184 it_guid | 00000000-0000-0000-0000-000000000000 parentid | 00000000-0000-0000-0000-000000000000 imagestatus | 4 lastmodified | 2016-04-21 11:25:59.972+02 vm_snapshot_id | 27c187cd-989f-4f7a-ac05-49c4410de6c2 volume_type | 1 volume_format | 5 image_group_id | 8cda0aa6-9e25-4b50-ba00-b877232a1983 _create_date | 2016-03-29 15:12:31.994065+02 _update_date | 2016-09-04 01:10:08.773649+02 active | f volume_classification | 1
-[ RECORD 2 ]---------+------------------------------------- image_guid | 68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e creation_date | 2016-07-03 01:01:30+02 size | 17179869184 it_guid | 00000000-0000-0000-0000-000000000000 parentid | 60ba7acf-58cb-475b-b9ee-15b1be99fee6 imagestatus | 1 lastmodified | 2016-07-04 01:03:33.732+02 vm_snapshot_id | 175c2071-a06b-4b0e-a069-5cc4bb236a34 volume_type | 2 volume_format | 4 image_group_id | 8cda0aa6-9e25-4b50-ba00-b877232a1983 _create_date | 2016-07-03 01:01:15.069585+02 _update_date | 2016-09-11 02:06:04.420965+02 active | f volume_classification | 1
-[ RECORD 3 ]---------+------------------------------------- image_guid | 37ca6494-e990-44e5-8597-28845a0a19b5 creation_date | 2016-08-07 01:06:15+02 size | 17179869184 it_guid | 00000000-0000-0000-0000-000000000000 parentid | 68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e imagestatus | 1 lastmodified | 2016-08-08 01:00:03.778+02 vm_snapshot_id | 4c0e5ac0-2ef3-4996-b3e9-7fd566d97b1a volume_type | 2 volume_format | 4 image_group_id | 8cda0aa6-9e25-4b50-ba00-b877232a1983 _create_date | 2016-08-07 01:06:01.777156+02 _update_date | 2016-09-25 01:55:54.090026+02 active | f volume_classification | 1
I removed other 10 snapshot infos from the list to easily read all
On a host running v4.0.4 we run
root@ovc2n06 ~]# lvchange -aey -pr 384f9059-ef2f-4d43-a54f-de71c5d589c8/60ba7acf-58cb-475b-b9ee-15b1be99fee6
Logical volume "60ba7acf-58cb-475b-b9ee-15b1be99fee6" changed.
This is the base image infact it doesn't contain a backing-file
[root@ovc2n06 ~]# qemu-img info --backing-chain /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/60ba7acf-58cb-475b-b9ee-15b1be99fee6
image: /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/60ba7acf-58cb-475b-b9ee-15b1be99fee6
file format: raw virtual size: 16G (17179869184 bytes) disk size: 0
[root@ovc2n06 ~]# lvchange -aey -pr 384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e
Logical volume "68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e" changed.
[root@ovc2n06 ~]# qemu-img info --backing-chain /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e
qemu-img: Could not open '/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b
1be99fee6': Could not open '/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b
1be99fee6': No such file or directory
To fix this problem we made a link
[root@ovc2n06 ~]# lvchange -aey -pr 384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e
Logical volume "68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e" changed. [root@ovc2n06 ~]# qemu-img info --backing-chain /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e
qemu-img: Could not open '/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b
1be99fee6': Could not open '/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b
1be99fee6': No such file or directory [root@ovc2n06 ~]# ln -s /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8 /dev/8cda0aa6-9e25-4b50-ba00-b877232a1983 [root@ovc2n06 ~]# qemu-img info --backing-chain /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e
image: /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/68d764ec-bc2e-4e1d-b8f2-b44afd9fcb2e
file format: qcow2 virtual size: 16G (17179869184 bytes) disk size: 0 cluster_size: 65536 backing file: ../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1be99fee6
(actual path: /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1
be99fee6) backing file format: raw Format specific information: compat: 0.10 refcount bits: 16
image: /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/../8cda0aa6-9e25-4b50-ba00-b877232a1983/60ba7acf-58cb-475b-b9ee-15b1
be99fee6 file format: raw virtual size: 16G (17179869184 bytes) disk size: 0
We moved to the next snapshot
[root@ovc2n06 ~]# lvchange -aey -pr 384f9059-ef2f-4d43-a54f-de71c5d589c8/37ca6494-e990-44e5-8597-28845a0a19b5
Logical volume "37ca6494-e990-44e5-8597-28845a0a19b5" changed.
And now the error from the bug
qemu-img info --backing-chain /dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/37ca6494-e990-44e5-8597-28845a0a19b5 qemu-img: Could not open '/dev/384f9059-ef2f-4d43-a54f-de71c5d589c8/37ca6494-e990-44e5-8597-28845a0a19b5': Backing file name too long
Any snapshot starting from this snapshot is not usable, the VM was running ok up to we did shutdown it, it never starts again.
How you can see the path of the backing-file is wrong and too long already in the previous snapshots, is there a way to fix it or to edit it manually ?
Obviously we tried to clone, export, create a qcow2 image, from all the snapshot later 07-august but the operation didn't complete, we can recover only from the snapshot of 7-august that is missing 2 months of new data.
Please if you have a workaround or a solution, can you write the commands we need to run with examples, we searched a lot about backing-file but only the manual of the qemu-img command was found with no examples on how to recover or change it.
Thank you!
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- / | / _____/ / | / _____/ | / / / | / / / / | / / / | / / / | / ___/ _____/ / / | / ___/ / | / / / | / / / / | / / / | / ______/ _/ __/ _/ _/ _/ __/ _/ _/ __/ Claudio Soprano phone: (+39)-06-9403.2349/2355 Computing Service fax: (+39)-06-9403.2649 LNF-INFN e-mail: Claudio.Soprano@lnf.infn.it Via Enrico Fermi, 40 www: http://www.lnf.infn.it/ I-00044 Frascati, Italy
On 19/10/16 14:43 +0200, Claudio Soprano wrote:
Hi Adam, we tried your solution.
This is our situation with the current VM that has 2 disks
base -> snap1 -> snap2 -> snap3 -> snap4 -> snap5 -> .. -> snap15 for each disk
We tried to do
qemu-img rebase -u -b base snap1
results OK
qemu-img rebase -u -b snap1 snap2
results:
qemu-img: Could not open 'snap2': Backing file name too long
our qemu version is
qemu-img version 2.3.0 (qemu-kvm-ev-2.3.0-31.el7.16.1), Copyright (c) 2004-2008 Fabrice Bellard
How do you think can we resolve ?
I talked with the qemu developers about this issue and the best way to fix this is by using a patched version of qemu-img that ignores invalid backing_file values when doing an unsafe rebase. Here is what you will need to do to fix your images. 1. Save the attached patch 2. Grab a copy of the latest qemu.git 3. Apply the patch to the source 4. Install qemu build dependencies 5. Build qemu 6. Run the built version of qemu-img when fixing your chain as I suggested above: ./qemu-img rebase -u -b snap1 snap2 The patch disables other qemu-img functionality since you should not be using this for anything but the rebase part. After the rebase you can use the system qemu-img binary to check the image. Please try this on one VM disk and make sure everything is okay. -- Adam Litke
We are trying a Fedora 24, because the qemu-img from git require a differnt version of glibc 2.12 and gthread 2.0 from that running on the hosts. In the while we fixed the problem in another way: 1) we dumped (with dd) all the snapshots and the base on files 2) then checked where the backing file becomes too long (in our case in the 2nd snapshot) with qemu-img info --backing chain <dumped snapshot filename> 3) we fixed the backing file with hexedit on that file only (be sure to correct too the lenght of the backing file read qcow2 format) 4) we checked others snapshots starting from 3rd and all were OK 5) we converted all in a qcow2 and in a raw image with qemu-img convert <latest snapshot filename> -O qcow2 <output filename in qcow2> qemu-img convert <latest snapshot filename> -O raw <output filename in raw> 6) we mounted with guestfish the qcow2 image and checked all was correct and it was 7) we did the same steps (from 1 to 6) for all the disks (base + snapshots) in the VM 8) we create a new VM in ovirt with the same structure of the old one (RAM, disks, disks size, network) 9) we started the new VM with a livecd and then dumped back (using the raw files) all the disks fixed using dd with netcat 10) we stopped and started again the new VM and all was working like we left the last working time I know our solution was longer and harder, but in this way if something was wrong we had destroyed a copy of the original datas and we had no success to compile the qemu-img from git on our hosts 4.0.4 for glibc and gthread version. Anyway we will have to check all the VMs (about 200) for snapshots and backing file too long but then we would like to fix the problem directly on the LVs snapshot (our method would require i suppose 2-3 months of work), so we need a working qemu-img patched. I will let you know if fedora 24 still give problems about the compiler and gthread version. I opened another case in the mailing list, because from when we upgraded to 4.0.4 each VM that we shutdown, never starts again with the following error "Unable to get size for domain ..." Do you think it can be related at the backing file problem ? For now thanks to all Claudio On 19/10/16 16:55, Adam Litke wrote:
On 19/10/16 14:43 +0200, Claudio Soprano wrote:
Hi Adam, we tried your solution.
This is our situation with the current VM that has 2 disks
base -> snap1 -> snap2 -> snap3 -> snap4 -> snap5 -> .. -> snap15 for each disk
We tried to do
qemu-img rebase -u -b base snap1
results OK
qemu-img rebase -u -b snap1 snap2
results:
qemu-img: Could not open 'snap2': Backing file name too long
our qemu version is
qemu-img version 2.3.0 (qemu-kvm-ev-2.3.0-31.el7.16.1), Copyright (c) 2004-2008 Fabrice Bellard
How do you think can we resolve ?
I talked with the qemu developers about this issue and the best way to fix this is by using a patched version of qemu-img that ignores invalid backing_file values when doing an unsafe rebase. Here is what you will need to do to fix your images.
1. Save the attached patch 2. Grab a copy of the latest qemu.git 3. Apply the patch to the source 4. Install qemu build dependencies 5. Build qemu 6. Run the built version of qemu-img when fixing your chain as I suggested above:
./qemu-img rebase -u -b snap1 snap2
The patch disables other qemu-img functionality since you should not be using this for anything but the rebase part. After the rebase you can use the system qemu-img binary to check the image. Please try this on one VM disk and make sure everything is okay.
-- / | / _____/ / | / _____/ | / / / | / / / / | / / / | / / / | / ___/ _____/ / / | / ___/ / | / / / | / / / / | / / / | / ______/ _/ __/ _/ _/ _/ __/ _/ _/ __/ Claudio Soprano phone: (+39)-06-9403.2349/2355 Computing Service fax: (+39)-06-9403.2649 LNF-INFN e-mail: Claudio.Soprano@lnf.infn.it Via Enrico Fermi, 40 www: http://www.lnf.infn.it/ I-00044 Frascati, Italy
participants (3)
-
Adam Litke -
Claudio Soprano -
Dael Maselli