Hi, I'm having (another) issue working with snapshots. Since a few days my backup system (storware vprotect) cannot delete a snapshot. Message reported by SPM is: HSMGetAllTasksStatusesVDS failed: value=Volume does not exist: ('6180c2e8-141a-4126-bcb1-5af5caa98175',) abortedcode=201 Other snapshots from other VMs are working fine (created and deleted as needed). I verified it, I can reach on SPM filesystem other snapshots from the same VM, but that one doesn't exist. I suspect that during normal backup management that snapshot has been correctly removed from file system, but still live in ovirt database. I'm pretty sure vProtect use ovirt API to operate, it doesn't directly interact with ovirt database. That's why I'm talking to you and not to storware guys. Is there a (safe) way to get rid of that snapshot?

Hi, a colleague has submitted a bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1948599 Regards, Giulio Casella On 12/04/2021 14:33, Nir Soffer wrote: > On Mon, Apr 12, 2021 at 11:19 AM Giulio Casella <giulio(a)di.unimi.it&gt; wrote: >> >> Hi, >> I'm having (another) issue working with snapshots. >> >> Since a few days my backup system (storware vprotect) cannot delete a >> snapshot. Message reported by SPM is: >> >> HSMGetAllTasksStatusesVDS failed: value=Volume does not exist: >> ('6180c2e8-141a-4126-bcb1-5af5caa98175',) abortedcode=201 >> >> Other snapshots from other VMs are working fine (created and deleted as >> needed). >> >> I verified it, I can reach on SPM filesystem other snapshots from the >> same VM, but that one doesn't exist. >> >> I suspect that during normal backup management that snapshot has been >> correctly removed from file system, but still live in ovirt database. >> >> I'm pretty sure vProtect use ovirt API to operate, it doesn't directly >> interact with ovirt database. That's why I'm talking to you and not to >> storware guys. >> >> Is there a (safe) way to get rid of that snapshot? > > If the snapshot was already removed, ovirt should detect this and > complete the operation successfully. > > Please file ovirt bug for this, and attach engine and vdsm logs showing > the effected snapshot and disk snapshot id. > > You can get find the snapshot id is the UI. You can see the disk snapshot > id in vm > snapshots > disks > disk id > > Then grep engine and vdsm log for the disk id an attach all the logs > mentioning this disk id. > > It will be also useful to get a database dump so we can see what engine > knows about this snapshot. > > Benny, can you recommend some queries to extract relevant info? > > Nir > _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/CXESZXGGVOV...

I have used these tools to get rid of snapshots that wouldn't go away any other way: https://www.ovirt.org/develop/developer-guide/db-issues/helperutilities.html _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/37K5J2X2OUD...

Thanks, Ritesh. In order to delete that snapshot, run this command on psql: SELECT DeleteSnapshot('snapshot-id'); *Regards, * *Shani Leviim * On Tue, Apr 20, 2021 at 3:25 PM Ritesh Chikatwar <rchikatw(a)redhat.com <mailto:rchikatw@redhat.com>> wrote: adding @Shani Leviim <mailto:sleviim@redhat.com>  On Tue, Apr 20, 2021 at 5:35 PM Thomas Hoberg <thomas(a)hoberg.net <mailto:thomas@hoberg.net>> wrote: I have used these tools to get rid of snapshots that wouldn't go away any other way: https://www.ovirt.org/develop/developer-guide/db-issues/helperutilities.html <https://www.ovirt.org/develop/developer-guide/db-issues/helperutilities.h... _______________________________________________ Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave(a)ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/37K5J2X2OUD... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/37K5J2X2OUD... _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LS3BUPQIVHQ...

Hi, You can use the DeleteImage tool [1], but it's first recommended to backup your engine. To be on the safe side, here are some checks it's better to execute before deleting: - In order to locate the illegal image, you can run this command on the host: vdsm-tool dump-volume-chains <sd_id>

- It's also recommended to verify that the engine and vdsm are synced about that image, once you've found it, by running this command on the engine: SELECT *      FROM images_storage_domain_view images_storage_domain_view      WHERE image_group_id = <image_group_id>;

[1] DeleteImage tool: SELECT DeleteImage('image-id');     DELETE     FROM images     WHERE image_guid = v_image_guid;

Since 6180c2e8-141a-4126-bcb1-5af5caa98175 doesn't exist on the VDSM, it should be removed from the DB, and since it's the parent id for image e77e2773-eecb-46a6-87c7-14df14f7d95b, it also should be changed. Something like this: UPDATE images set parentid = '19fa612d-2f93-46a5-93e7-c74afc2937fa' where image_guid = 'e77e2773-eecb-46a6-87c7-14df14f7d95b'; Then you can remove the 6180c2e8 image. It will be safe if you'll have a backup before making any DB changes. *Regards, * *Shani Leviim * On Thu, Apr 22, 2021 at 3:06 PM Giulio Casella <giulio(a)di.unimi.it <mailto:giulio@di.unimi.it>> wrote: On 22/04/2021 13:20, Shani Leviim wrote: > Hi, > You can use the DeleteImage tool [1], but it's first recommended to > backup your engine. > > To be on the safe side, here are some checks it's better to execute > before deleting: > - In order to locate the illegal image, you can run this command on the > host: > vdsm-tool dump-volume-chains <sd_id> Ok, that is the relevant part of the output: image:    2fd9b340-14ce-42cd-a880-8f75eb3b72e7              - 19fa612d-2f93-46a5-93e7-c74afc2937fa                status: OK, voltype: INTERNAL, format: COW, legality: LEGAL, type: SPARSE, capacity: 53687091200, truesize: 45902462976              - e77e2773-eecb-46a6-87c7-14df14f7d95b                status: OK, voltype: LEAF, format: COW, legality: LEGAL, type: SPARSE, capacity: 53687091200, truesize: 27917287424 So I have on storage two images, first one is a snapshot, second one is Active VM. Being the VM up and running I think I could say that on storage the situation is consistent. > > - It's also recommended to verify that the engine and vdsm are synced > about that image, once you've found it, by running this command on the > engine: > SELECT * >      FROM images_storage_domain_view images_storage_domain_view >      WHERE image_group_id = <image_group_id>; > That's the point: engine (database) sees 3 images, whilst vdsm sees only 2. And this is output of: select image_guid,parentid,imagestatus,vm_snapshot_id,volume_type,volume_format,active from images where image_group_id='image-group-id';               image_guid              |               parentid      | imagestatus |            vm_snapshot_id            | volume_type | volume_format | active --------------------------------------+--------------------------------------+-------------+--------------------------------------+-------------+---------------+--------  6180c2e8-141a-4126-bcb1-5af5caa98175 | 19fa612d-2f93-46a5-93e7-c74afc2937fa |           1 | 281a4331-89fa-4544-b6ab-22c0e5ac6459 |           2 |             4 | f  19fa612d-2f93-46a5-93e7-c74afc2937fa | 00000000-0000-0000-0000-000000000000 |           4 | 6488f4c8-efc4-43df-aafc-497fb7d51fbb |           2 |             4 | f  e77e2773-eecb-46a6-87c7-14df14f7d95b | 6180c2e8-141a-4126-bcb1-5af5caa98175 |           1 | 59f69d6f-f300-45e4-8487-1fe4368a5933 |           2 |             4 | t So there is in the database an image (the one with image-guid "6180c2e8-141a-4126-bcb1-5af5caa98175") not existing on storage. > [1] > DeleteImage tool: > SELECT DeleteImage('image-id'); > >     DELETE >     FROM images >     WHERE image_guid = v_image_guid; > I think it could not be enough, I think I should adjust parentship relations in "images" table. Just looking at that table it seems not so difficult, but: is it sufficient? Is it safe? Regards, Giulio