On 23. 8. 2022, at 10:53, Andrei Verovski <andreil1@starlett.lv> wrote:

Hi,

I had same problem with VM of our accounting / stock control system, which is a top priority which can’t be down no matter what.

Solution: Make a new snapshot on problematic VM, then make a new VM from that snapshot. This is only a workaround I found so far.

Make sure you run latest version, there were plenty of live merge bugs solved in the last months. Also if you're using vProtect you can likely use the new incremental backup(CBT) feature, that avoids all the snapshotting in the first place.

...

On 23 Aug 2022, at 10:50, Jirka Simon <jirka@vesim.cz <mailto:jirka@vesim.cz>> wrote:

Hello there.

we have troubles with one VM ad it's snapshot

here is short log from engine.log

2022-08-23 09:02:38,735+02 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-1265784) [4d79272b-ffac-4b2b-a987-8c19577cd8c4] CommandAsyncTask:: HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity '5842cafc-60d4-4131-8da1-8a909123f08c' 2022-08-23 09:02:41,435+02 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-68) [4d79272b -ffac-4b2b-a987-8c19577cd8c4] Command id: 'ead3fbba-7439-4337-8aaf-1d6cf65bbb15 failed child command status for step 'REDUCE_IMAGE' 2022-08-23 09:02:41,435+02 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-68) [ 4d79272b-ffac-4b2b-a987-8c19577cd8c4] Command 'RemoveSnapshotSingleDiskLive' id: 'ead3fbba-7439-4337-8aaf-1d6cf65bbb15' child commands '[5b33d970-6ea0-4b8b-bc31-607afd8af1ca, 185d3fec-8c76- 44d3-a0a1-fe8fff331354, 308baf9d-1fc5-43a1-a624-b5f08f7bec1b, f11f3550-4356-417e-af10-a3afb2051dec, 5842cafc-60d4-4131-8da1-8a909123f08c]' executions were completed, status 'FAILED' 2022-08-23 09:02:42,445+02 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-79) [4d79272b -ffac-4b2b-a987-8c19577cd8c4] Merging of snapshot '6634db4e-23d8-4849-a008-864f15d28c3d' images 'd24a9199-741b-49cb-96d2-0d76fcd21f48'..'48af3301-0cbb-4a6e-97d1-7299e7de883f' failed. Images have been marked illegal and can no longer be previewed or reverted to. Please retry Live Merge on the snapshot to complete the operation. 2022-08-23 09:02:42,451+02 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-79) [4d79272b -ffac-4b2b-a987-8c19577cd8c4] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand' with failure. 2022-08-23 09:02:42,457+02 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-79) [4d79272b-ffac -4b2b-a987-8c19577cd8c4] Command 'RemoveSnapshot' id: '73340ac8-07d7-4de2-bfeb-8062fa1e8cfd' child commands '[ead3fbba-7439-4337-8aaf-1d6cf65bbb15]' executions were completed, status 'FAILE D' 2022-08-23 09:02:43,470+02 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-22) [4d79272b-ffac-4b2b-a98 7-8c19577cd8c4] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand' with failure. 2022-08-23 09:02:43,490+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-22) [4d79272b-ff ac-4b2b-a987-8c19577cd8c4] EVENT_ID: USER_REMOVE_SNAPSHOT_FINISHED_FAILURE(357), Failed to delete snapshot 'vProtect 2022-08-05 22:33:44.088143' for VM 'web1.wiki.prod.hq.sldev.cz <http://wiki.prod.hq.sldev.cz/>-2'.

Snapshot can't be deleted, and VM is not possible to clone.

thank you for any help.

Jirka

_______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZOVL22OGZKEG6B... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZOVL22OGZKEG6BXPEB5JQUXX2GXDA2GD/>

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/AWITGPKJKLTA2U...

Hi, I use latest release of 4.4 and it started afret last upgrade to 4.4.10-7. @Andrei, yes we use vProtect for  backups. Jirka On 8/23/22 11:27, Michal Skrivanek wrote:

...

On 23. 8. 2022, at 10:53, Andrei Verovski <andreil1@starlett.lv> wrote:

Hi,

I had same problem with VM of our accounting / stock control system, which is a top priority which can’t be down no matter what.

Solution: Make a new snapshot on problematic VM, then make a new VM from that snapshot. This is only a workaround I found so far.

Make sure you run latest version, there were plenty of live merge bugs solved in the last months. Also if you're using vProtect you can likely use the new incremental backup(CBT) feature, that avoids all the snapshotting in the first place.

...

...

On 23 Aug 2022, at 10:50, Jirka Simon <jirka@vesim.cz> wrote:

Hello there.

we have troubles with one VM ad it's snapshot

here is  short log  from engine.log

2022-08-23 09:02:38,735+02 INFO  [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-1265784) [4d79272b-ffac-4b2b-a987-8c19577cd8c4] CommandAsyncTask:: HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity '5842cafc-60d4-4131-8da1-8a909123f08c' 2022-08-23 09:02:41,435+02 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-68) [4d79272b -ffac-4b2b-a987-8c19577cd8c4] Command id: 'ead3fbba-7439-4337-8aaf-1d6cf65bbb15 failed child command status for step 'REDUCE_IMAGE' 2022-08-23 09:02:41,435+02 INFO  [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-68) [ 4d79272b-ffac-4b2b-a987-8c19577cd8c4] Command 'RemoveSnapshotSingleDiskLive' id: 'ead3fbba-7439-4337-8aaf-1d6cf65bbb15' child commands '[5b33d970-6ea0-4b8b-bc31-607afd8af1ca, 185d3fec-8c76- 44d3-a0a1-fe8fff331354, 308baf9d-1fc5-43a1-a624-b5f08f7bec1b, f11f3550-4356-417e-af10-a3afb2051dec, 5842cafc-60d4-4131-8da1-8a909123f08c]' executions were completed, status 'FAILED' 2022-08-23 09:02:42,445+02 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-79) [4d79272b -ffac-4b2b-a987-8c19577cd8c4] Merging of snapshot '6634db4e-23d8-4849-a008-864f15d28c3d' images 'd24a9199-741b-49cb-96d2-0d76fcd21f48'..'48af3301-0cbb-4a6e-97d1-7299e7de883f' failed. Images have been marked illegal and can no longer be previewed or reverted to. Please retry Live Merge on the snapshot to complete the operation. 2022-08-23 09:02:42,451+02 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-79) [4d79272b -ffac-4b2b-a987-8c19577cd8c4] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand' with failure. 2022-08-23 09:02:42,457+02 INFO  [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-79) [4d79272b-ffac -4b2b-a987-8c19577cd8c4] Command 'RemoveSnapshot' id: '73340ac8-07d7-4de2-bfeb-8062fa1e8cfd' child commands '[ead3fbba-7439-4337-8aaf-1d6cf65bbb15]' executions were completed, status 'FAILE D' 2022-08-23 09:02:43,470+02 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-22) [4d79272b-ffac-4b2b-a98 7-8c19577cd8c4] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand' with failure. 2022-08-23 09:02:43,490+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-22) [4d79272b-ff ac-4b2b-a987-8c19577cd8c4] EVENT_ID: USER_REMOVE_SNAPSHOT_FINISHED_FAILURE(357), Failed to delete snapshot 'vProtect 2022-08-05 22:33:44.088143' for VM 'web1.wiki.prod.hq.sldev.cz <http://wiki.prod.hq.sldev.cz/>-2'.

Snapshot can't be deleted, and VM is not possible to clone.

thank you for any help.

Jirka

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZOVL22OGZKEG6B...

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/AWITGPKJKLTA2U...

_______________________________________________ Users mailing list --users@ovirt.org To unsubscribe send an email tousers-leave@ovirt.org Privacy Statement:https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct:https://www.ovirt.org/community/about/community-guidelines/ List Archives:https://lists.ovirt.org/archives/list/users@ovirt.org/message/GLHFQ4KQITT72O...

What is the snapshot issue and is there a bug for it? On Wed, Aug 24, 2022 at 4:21 PM Sven Jansen via Users <users@ovirt.org> wrote:

We are currently sitting on a not really working backup solution. oVirt 4.5.x has constant issues removing snapshots, same like OP. vProtect with Snapshots has issues with oVirt deleting snapshots. vProtect 5.1 has broken CBT with oVirt 4.5.x.

vProtect support tells us that the snapshot issues is in oVirt (true) and CBT is a preview feature not really supported. So currently vProtect 5.1 and oVirt 4.5 is a no go solution. We are thinking going to xcp-ng due to constant issues with oVirt and or vProtect. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/V7VSQLOGJDTBTZ...

Hi, I filed a bug about this last year... https://bugzilla.redhat.com/show_bug.cgi?id=1948599 We've dome some tests with vProtect after the leatest Ovirt release 4.5.2.4-1.el8 but we still have problems with snapshots removal and disks left in locked or illegal state. To bring back a disk from illegal state we live migrate the disk(s) to another data domain. After this the snapshots are back to legal and can be removed from the UI. This is pratical only for VMs with small disks. Sadly one of the VMs we have problems with has a 2.5Tb disk attached :( Hope this get solved soon. Regards Il 25/08/2022 10:26, Jirka Simon ha scritto:

Hello Sven,

Please do you have any details about it? Like bug report od knowledge base or article ?   We have ovit and vprotect as well and our troubles started after last update of ovirt 4.4 ( frozen vm with snapshot crating job, then after unlock some snapshots is not possible to delete.)  And now I'm preparing upgrade to 4.5.2 Yesterday we upgraded vProtect to the latest version.

Thank you for any detail.

Jirka

On 8/24/22 15:20, Sven Jansen via Users wrote:

...

We are currently sitting on a not really working backup solution. oVirt 4.5.x has constant issues removing snapshots, same like OP. vProtect with Snapshots has issues with oVirt deleting snapshots. vProtect 5.1 has broken CBT with oVirt 4.5.x.

vProtect support tells us that the snapshot issues is in oVirt (true) and CBT is a preview feature not really supported. So currently vProtect 5.1 and oVirt 4.5 is a no go solution. We are thinking going to xcp-ng due to constant issues with oVirt and or vProtect. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/V7VSQLOGJDTBTZ...

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FMK2Z5B5K7Y5XH...

-- gb PGP Key: http://pgp.mit.edu/ Primary key fingerprint: C510 0765 943E EBED A4F2 69D3 16CC DC90 B9CB 0F34

So it seems there's another bug. Since the bug was opened for an issue caused by the removal of a snapshot by vprotect (backup application) I thought that the fix should also fix the issue we have with vprotect, but this didn't happened. I think we'll file a new bug as soon as possible. Regards Il 26/08/2022 20:25, Benny Zlotnik ha scritto:

The bug you mentioned was fixed in 4.5.2.2. But to clarify the issue that's fixed: The fix is for a situation where a snapshot removal failed for whatever reason, but the volume was removed in vdsm and remains present in the engine's database. Without the fix, attempting to remove the snapshot again would fail.

Is this the scenario you are facing now?

There is another question of why the snapshot removal failed initially, and I did not get logs with this information (although it could be due to network issues)

On Fri, Aug 26, 2022 at 7:19 PM Giorgio Biacchi <giorgio@di.unimi.it> wrote:

...

Hi, I filed a bug about this last year...

https://bugzilla.redhat.com/show_bug.cgi?id=1948599

We've dome some tests with vProtect after the leatest Ovirt release 4.5.2.4-1.el8 but we still have problems with snapshots removal and disks left in locked or illegal state.

To bring back a disk from illegal state we live migrate the disk(s) to another data domain. After this the snapshots are back to legal and can be removed from the UI. This is pratical only for VMs with small disks. Sadly one of the VMs we have problems with has a 2.5Tb disk attached :(

Hope this get solved soon.

Regards

Il 25/08/2022 10:26, Jirka Simon ha scritto:

...

Hello Sven,

Please do you have any details about it? Like bug report od knowledge base or article ? We have ovit and vprotect as well and our troubles started after last update of ovirt 4.4 ( frozen vm with snapshot crating job, then after unlock some snapshots is not possible to delete.) And now I'm preparing upgrade to 4.5.2 Yesterday we upgraded vProtect to the latest version.

Thank you for any detail.

Jirka

On 8/24/22 15:20, Sven Jansen via Users wrote:

...

We are currently sitting on a not really working backup solution. oVirt 4.5.x has constant issues removing snapshots, same like OP. vProtect with Snapshots has issues with oVirt deleting snapshots. vProtect 5.1 has broken CBT with oVirt 4.5.x.

vProtect support tells us that the snapshot issues is in oVirt (true) and CBT is a preview feature not really supported. So currently vProtect 5.1 and oVirt 4.5 is a no go solution. We are thinking going to xcp-ng due to constant issues with oVirt and or vProtect. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/V7VSQLOGJDTBTZ...

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FMK2Z5B5K7Y5XH...

-- gb

PGP Key: http://pgp.mit.edu/ Primary key fingerprint: C510 0765 943E EBED A4F2 69D3 16CC DC90 B9CB 0F34 _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/K5P7G25WSOUCW6...

-- gb PGP Key: http://pgp.mit.edu/ Primary key fingerprint: C510 0765 943E EBED A4F2 69D3 16CC DC90 B9CB 0F34

Hi guys, is there any news about this bug? On 31/08/2022 10:03, Giulio Casella wrote:

Hi Benny, we filed a new bug: https://bugzilla.redhat.com/show_bug.cgi?id=2122525

On 30/08/2022 16:00, Giorgio Biacchi wrote:

...

So it seems there's another bug.

Since the bug was opened for an issue caused by the removal of a snapshot by vprotect (backup application) I thought that the fix should also fix the issue we have with vprotect, but this didn't happened.

I think we'll file a new bug as soon as possible.

Regards

Il 26/08/2022 20:25, Benny Zlotnik ha scritto:

...

The bug you mentioned was fixed in 4.5.2.2. But to clarify the issue that's fixed: The fix is for a situation where a snapshot removal failed for whatever reason, but the volume was removed in vdsm and remains present in the engine's database. Without the fix, attempting to remove the snapshot again would fail.

Is this the scenario you are facing now?

There is another question of why the snapshot removal failed initially, and I did not get logs with this information (although it could be due to network issues)

On Fri, Aug 26, 2022 at 7:19 PM Giorgio Biacchi <giorgio@di.unimi.it> wrote:

...

Hi, I filed a bug about this last year...

https://bugzilla.redhat.com/show_bug.cgi?id=1948599

We've dome some tests with vProtect after the leatest Ovirt release 4.5.2.4-1.el8  but we still have problems with snapshots removal and disks left in locked or illegal state.

To bring back a disk from illegal state we live migrate the disk(s) to another data domain. After this the snapshots are back to legal and can be removed from the UI. This is pratical only for VMs with small disks. Sadly one of the VMs we have problems with has a 2.5Tb disk attached :(

Hope this get solved soon.

Regards

Il 25/08/2022 10:26, Jirka Simon ha scritto:

...

Hello Sven,

Please do you have any details about it? Like bug report od knowledge base or article ?   We have ovit and vprotect as well and our troubles started after last update of ovirt 4.4 ( frozen vm with snapshot crating job, then after unlock some snapshots is not possible to delete.)  And now I'm preparing upgrade to 4.5.2 Yesterday we upgraded vProtect to the latest version.

Thank you for any detail.

Jirka

On 8/24/22 15:20, Sven Jansen via Users wrote:

...

We are currently sitting on a not really working backup solution. oVirt 4.5.x has constant issues removing snapshots, same like OP. vProtect with Snapshots has issues with oVirt deleting snapshots. vProtect 5.1 has broken CBT with oVirt 4.5.x.

vProtect support tells us that the snapshot issues is in oVirt (true) and CBT is a preview feature not really supported. So currently vProtect 5.1 and oVirt 4.5 is a no go solution. We are thinking going to xcp-ng due to constant issues with oVirt and or vProtect. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/V7VSQLOGJDTBTZ...

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FMK2Z5B5K7Y5XH...

-- gb

PGP Key: http://pgp.mit.edu/ Primary key fingerprint: C510 0765 943E EBED A4F2 69D3 16CC DC90 B9CB 0F34 _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/K5P7G25WSOUCW6...

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/E5CWFX34ZVFMT5...

-- Giulio Casella giulio at di.unimi.it System and network architect Computer Science Dept. - University of Milano