Hi Nir, in bugzilla I see it could be a bug, so I'm waiting for software update to fix it. In the meanwhile is there some fix I can do on the database? The VM is working fine, but I cannot operate with snapshots, and my backup is stuck... TIA, Giulio Casella On 12/04/2021 16:54, Giulio Casella wrote:

Hi, a colleague has submitted a bugzilla:

https://bugzilla.redhat.com/show_bug.cgi?id=1948599

Regards, Giulio Casella

On 12/04/2021 14:33, Nir Soffer wrote:

...

On Mon, Apr 12, 2021 at 11:19 AM Giulio Casella <giulio@di.unimi.it> wrote:

...

Hi, I'm having (another) issue working with snapshots.

Since a few days my backup system (storware vprotect) cannot delete a snapshot. Message reported by SPM is:

HSMGetAllTasksStatusesVDS failed: value=Volume does not exist: ('6180c2e8-141a-4126-bcb1-5af5caa98175',) abortedcode=201

Other snapshots from other VMs are working fine (created and deleted as needed).

I verified it, I can reach on SPM filesystem other snapshots from the same VM, but that one doesn't exist.

I suspect that during normal backup management that snapshot has been correctly removed from file system, but still live in ovirt database.

I'm pretty sure vProtect use ovirt API to operate, it doesn't directly interact with ovirt database. That's why I'm talking to you and not to storware guys.

Is there a (safe) way to get rid of that snapshot?

If the snapshot was already removed, ovirt should detect this and complete the operation successfully.

Please file ovirt bug for this, and attach engine and vdsm logs showing the effected snapshot and disk snapshot id.

You can get find the snapshot id is the UI. You can see the disk snapshot id in vm > snapshots > disks > disk id

Then grep engine and vdsm log for the disk id an attach all the logs mentioning this disk id.

It will be also useful to get a database dump so we can see what engine knows about this snapshot.

Benny, can you recommend some queries to extract relevant info?

Nir

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/CXESZXGGVOVFVK...

adding @Shani Leviim <sleviim@redhat.com> On Tue, Apr 20, 2021 at 5:35 PM Thomas Hoberg <thomas@hoberg.net> wrote:

I have used these tools to get rid of snapshots that wouldn't go away any other way:

https://www.ovirt.org/develop/developer-guide/db-issues/helperutilities.html _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/37K5J2X2OUDQKN...

I tried, with no luck. Giving "disk snapshot id" as a parameter to that psql procedure nothing happened. Giving the "vm_snapshot_id" as found in the "images" table made the snapshot disappear from the gui, but not from the "images" table. SO I decided to look into DeleteSnapshot procedure, that resolve as: DELETE FROM snapshots WHERE snapshot_id = v_snapshot_id; It only acts on the "snapshots" table, and not on the "image" table, where the parentship between snapshots is still wrong. How safe is to manually modify "images" table and set correct parentship? On 20/04/2021 15:56, Shani Leviim wrote:

Thanks, Ritesh.

In order to delete that snapshot, run this command on psql: SELECT DeleteSnapshot('snapshot-id');

*Regards, * *Shani Leviim *

On Tue, Apr 20, 2021 at 3:25 PM Ritesh Chikatwar <rchikatw@redhat.com <mailto:rchikatw@redhat.com>> wrote:

adding @Shani Leviim <mailto:sleviim@redhat.com> 

On Tue, Apr 20, 2021 at 5:35 PM Thomas Hoberg <thomas@hoberg.net <mailto:thomas@hoberg.net>> wrote:

I have used these tools to get rid of snapshots that wouldn't go away any other way: https://www.ovirt.org/develop/developer-guide/db-issues/helperutilities.html <https://www.ovirt.org/develop/developer-guide/db-issues/helperutilities.html> _______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/37K5J2X2OUDQKN... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/37K5J2X2OUDQKN5J3J7ISOV26FMTHCTY/>

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LS3BUPQIVHQMG7...

On 22/04/2021 13:20, Shani Leviim wrote:

Hi, You can use the DeleteImage tool [1], but it's first recommended to backup your engine.

To be on the safe side, here are some checks it's better to execute before deleting: - In order to locate the illegal image, you can run this command on the host: vdsm-tool dump-volume-chains <sd_id>

Ok, that is the relevant part of the output: image: 2fd9b340-14ce-42cd-a880-8f75eb3b72e7 - 19fa612d-2f93-46a5-93e7-c74afc2937fa status: OK, voltype: INTERNAL, format: COW, legality: LEGAL, type: SPARSE, capacity: 53687091200, truesize: 45902462976 - e77e2773-eecb-46a6-87c7-14df14f7d95b status: OK, voltype: LEAF, format: COW, legality: LEGAL, type: SPARSE, capacity: 53687091200, truesize: 27917287424 So I have on storage two images, first one is a snapshot, second one is Active VM. Being the VM up and running I think I could say that on storage the situation is consistent.

- It's also recommended to verify that the engine and vdsm are synced about that image, once you've found it, by running this command on the engine: SELECT *      FROM images_storage_domain_view images_storage_domain_view      WHERE image_group_id = <image_group_id>;

That's the point: engine (database) sees 3 images, whilst vdsm sees only 2. And this is output of: select image_guid,parentid,imagestatus,vm_snapshot_id,volume_type,volume_format,active from images where image_group_id='image-group-id'; image_guid | parentid | imagestatus | vm_snapshot_id | volume_type | volume_format | active --------------------------------------+--------------------------------------+-------------+--------------------------------------+-------------+---------------+-------- 6180c2e8-141a-4126-bcb1-5af5caa98175 | 19fa612d-2f93-46a5-93e7-c74afc2937fa | 1 | 281a4331-89fa-4544-b6ab-22c0e5ac6459 | 2 | 4 | f 19fa612d-2f93-46a5-93e7-c74afc2937fa | 00000000-0000-0000-0000-000000000000 | 4 | 6488f4c8-efc4-43df-aafc-497fb7d51fbb | 2 | 4 | f e77e2773-eecb-46a6-87c7-14df14f7d95b | 6180c2e8-141a-4126-bcb1-5af5caa98175 | 1 | 59f69d6f-f300-45e4-8487-1fe4368a5933 | 2 | 4 | t So there is in the database an image (the one with image-guid "6180c2e8-141a-4126-bcb1-5af5caa98175") not existing on storage.

[1] DeleteImage tool: SELECT DeleteImage('image-id');

    DELETE     FROM images     WHERE image_guid = v_image_guid;

I think it could not be enough, I think I should adjust parentship relations in "images" table. Just looking at that table it seems not so difficult, but: is it sufficient? Is it safe? Regards, Giulio

Thanks Shani, this did the trick. I managed to tweak the database and then correctly remove the snapshot. In the meanwhile, in bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1948599, this has been confirmed as a bug. Regards, Giulio Casella On 22/04/2021 14:31, Shani Leviim wrote:

Since 6180c2e8-141a-4126-bcb1-5af5caa98175 doesn't exist on the VDSM, it should be removed from the DB, and since it's the parent id for image e77e2773-eecb-46a6-87c7-14df14f7d95b, it also should be changed. Something like this: UPDATE images set parentid = '19fa612d-2f93-46a5-93e7-c74afc2937fa' where image_guid = 'e77e2773-eecb-46a6-87c7-14df14f7d95b';

Then you can remove the 6180c2e8 image.

It will be safe if you'll have a backup before making any DB changes.

*Regards, * *Shani Leviim *

On Thu, Apr 22, 2021 at 3:06 PM Giulio Casella <giulio@di.unimi.it <mailto:giulio@di.unimi.it>> wrote:

On 22/04/2021 13:20, Shani Leviim wrote: > Hi, > You can use the DeleteImage tool [1], but it's first recommended to > backup your engine. > > To be on the safe side, here are some checks it's better to execute > before deleting: > - In order to locate the illegal image, you can run this command on the > host: > vdsm-tool dump-volume-chains <sd_id>

Ok, that is the relevant part of the output:

image:    2fd9b340-14ce-42cd-a880-8f75eb3b72e7

             - 19fa612d-2f93-46a5-93e7-c74afc2937fa                status: OK, voltype: INTERNAL, format: COW, legality: LEGAL, type: SPARSE, capacity: 53687091200, truesize: 45902462976

             - e77e2773-eecb-46a6-87c7-14df14f7d95b                status: OK, voltype: LEAF, format: COW, legality: LEGAL, type: SPARSE, capacity: 53687091200, truesize: 27917287424

So I have on storage two images, first one is a snapshot, second one is Active VM.

Being the VM up and running I think I could say that on storage the situation is consistent.

> > - It's also recommended to verify that the engine and vdsm are synced > about that image, once you've found it, by running this command on the > engine: > SELECT * >      FROM images_storage_domain_view images_storage_domain_view >      WHERE image_group_id = <image_group_id>; >

That's the point: engine (database) sees 3 images, whilst vdsm sees only 2.

And this is output of:

select image_guid,parentid,imagestatus,vm_snapshot_id,volume_type,volume_format,active from images where image_group_id='image-group-id';

              image_guid              |               parentid      | imagestatus |            vm_snapshot_id            | volume_type | volume_format | active --------------------------------------+--------------------------------------+-------------+--------------------------------------+-------------+---------------+--------  6180c2e8-141a-4126-bcb1-5af5caa98175 | 19fa612d-2f93-46a5-93e7-c74afc2937fa |           1 | 281a4331-89fa-4544-b6ab-22c0e5ac6459 |           2 |             4 | f  19fa612d-2f93-46a5-93e7-c74afc2937fa | 00000000-0000-0000-0000-000000000000 |           4 | 6488f4c8-efc4-43df-aafc-497fb7d51fbb |           2 |             4 | f  e77e2773-eecb-46a6-87c7-14df14f7d95b | 6180c2e8-141a-4126-bcb1-5af5caa98175 |           1 | 59f69d6f-f300-45e4-8487-1fe4368a5933 |           2 |             4 | t

So there is in the database an image (the one with image-guid "6180c2e8-141a-4126-bcb1-5af5caa98175") not existing on storage.

> [1] > DeleteImage tool: > SELECT DeleteImage('image-id'); > >     DELETE >     FROM images >     WHERE image_guid = v_image_guid; >

I think it could not be enough, I think I should adjust parentship relations in "images" table.

Just looking at that table it seems not so difficult, but: is it sufficient? Is it safe?

Regards, Giulio