Failed to delete snapshot
Hello all. We use vprotect to make snapshot backups of our vm's. This VM, let's call it OVIRTVM, has it's disks created as thin. A full snapshot is created daily on sunday and the daily ones are incremental. After the backup vprotect tries to delete the snapshot and almost always fails with the error: "Failed to delete snapshot 'vProtect 2021-10-25 22:30:17.641654' for VM 'OVIRTVM'." From engine.log can't get much more information: 2021-10-26 22:39:02,510+01 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-511666) [] User admin@internal successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2021-10-26 22:39:02,558+01 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-15) [666e0f97-8b02-4b1e-80d4-2a640dd28d90] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand' with failure. 2021-10-26 22:39:02,595+01 INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-511666) [1a9981] Running command: CreateUserSessionCommand internal: false. 2021-10-26 22:39:02,630+01 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-511612) [] User admin@internal successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2021-10-26 22:39:02,674+01 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-511669) [] User admin@internal successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2021-10-26 22:39:02,729+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-15) [666e0f97-8b02-4b1e-80d4-2a640dd28d90] EVENT_ID: USER_REMOVE_SNAPSHOT_FINISHED_FAILURE(357), Failed to delete snapshot 'vProtect 2021-10-25 22:30:17.641654' for VM 'OVIRTVM'. 2021-10-26 22:39:02,739+01 INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-511675) [55891f0c] Running command: CreateUserSessionCommand internal: false. 2021-10-26 22:39:02,755+01 INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-511660) [7895798d] Running command: CreateUserSessionCommand internal: false. What else should I be looking for? Thanks in advance!
We need full logs (what happened before the snippet you pasted), as well as vdsm logs from the SPM host, and vdsm logs from the host running this VM On Wed, Oct 27, 2021 at 5:04 PM <ivohccarvalho@gmail.com> wrote:
Hello all.
We use vprotect to make snapshot backups of our vm's. This VM, let's call it OVIRTVM, has it's disks created as thin. A full snapshot is created daily on sunday and the daily ones are incremental.
After the backup vprotect tries to delete the snapshot and almost always fails with the error: "Failed to delete snapshot 'vProtect 2021-10-25 22:30:17.641654' for VM 'OVIRTVM'."
From engine.log can't get much more information:
2021-10-26 22:39:02,510+01 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-511666) [] User admin@internal successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2021-10-26 22:39:02,558+01 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-15) [666e0f97-8b02-4b1e-80d4-2a640dd28d90] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand' with failure. 2021-10-26 22:39:02,595+01 INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-511666) [1a9981] Running command: CreateUserSessionCommand internal: false. 2021-10-26 22:39:02,630+01 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-511612) [] User admin@internal successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2021-10-26 22:39:02,674+01 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-511669) [] User admin@internal successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2021-10-26 22:39:02,729+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-15) [666e0f97-8b02-4b1e-80d4-2a640dd28d90] EVENT_ID: USER_REMOVE_SNAPSHOT_FINISHED_FAILURE(357), Failed to delete snapshot 'vProtect 2021-10-25 22:30:17.641654' for VM 'OVIRTVM'. 2021-10-26 22:39:02,739+01 INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-511675) [55891f0c] Running command: CreateUserSessionCommand internal: false. 2021-10-26 22:39:02,755+01 INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-511660) [7895798d] Running command: CreateUserSessionCommand internal: false.
What else should I be looking for?
Thanks in advance! _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/63FSCAGHIQJTCW...
Hi, Can you please provide the engine and VDSM logs that contains the failure? On Wed, Oct 27, 2021, 17:04 <ivohccarvalho@gmail.com> wrote:
Hello all.
We use vprotect to make snapshot backups of our vm's. This VM, let's call it OVIRTVM, has it's disks created as thin. A full snapshot is created daily on sunday and the daily ones are incremental.
After the backup vprotect tries to delete the snapshot and almost always fails with the error: "Failed to delete snapshot 'vProtect 2021-10-25 22:30:17.641654' for VM 'OVIRTVM'."
From engine.log can't get much more information:
2021-10-26 22:39:02,510+01 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-511666) [] User admin@internal successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2021-10-26 22:39:02,558+01 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-15) [666e0f97-8b02-4b1e-80d4-2a640dd28d90] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand' with failure. 2021-10-26 22:39:02,595+01 INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-511666) [1a9981] Running command: CreateUserSessionCommand internal: false. 2021-10-26 22:39:02,630+01 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-511612) [] User admin@internal successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2021-10-26 22:39:02,674+01 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-511669) [] User admin@internal successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2021-10-26 22:39:02,729+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-15) [666e0f97-8b02-4b1e-80d4-2a640dd28d90] EVENT_ID: USER_REMOVE_SNAPSHOT_FINISHED_FAILURE(357), Failed to delete snapshot 'vProtect 2021-10-25 22:30:17.641654' for VM 'OVIRTVM'. 2021-10-26 22:39:02,739+01 INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-511675) [55891f0c] Running command: CreateUserSessionCommand internal: false. 2021-10-26 22:39:02,755+01 INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-511660) [7895798d] Running command: CreateUserSessionCommand internal: false.
What else should I be looking for?
Thanks in advance! _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/63FSCAGHIQJTCW...
Logs are here for the host, SPM and Engine - https://we.tl/t-v5Vm2QYBIl (sorry, did not find a better way) The VM ID with this problem is 81e23292-4e43-4d9f-99ce-917936cce441. Thanks again.
Hi, From the logs it seems that the live merge operation failed because of a volume that still found on the chain and didn't removed - 2021-10-26 22:36:20,654+01 ERROR [org.ovirt.engine.core.bll.MergeStatusCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-10) [666e0f97-8b02-4b1e-80d4-2a640dd28d90] Failed to live merge. Top volume 069d1cc2-cd2d-4160-8ca0-657f6f57c168 is still in qemu chain [069d1cc2-cd2d-4160-8ca0-657f6f57c168, 751c65f5-868f-4b05-9986-e3f028fa1579, 2b1f324c-31a4-4db5-80fa-278e631a59bb, 4e8ea055-54bc-4a19-813c-79f6fe3e2db1] And - 2021-10-26 22:36:20,664+01 ERROR [org.ovirt.engine.core.bll.MergeStatusCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [666e0f97-8b02-4b1e-80d4-2a640dd28d90] Failed to live merge. Top volume 20640e9a-e8fc-48f6-af45-7c4e80729717 is still in qemu chain [625cf5c8-4cfb-4356-8bb7-e0103e01943b, ce345f57-b12b-463e-b5d0-bc4f71fa3624, 46f180de-d855-4cda-be39-0c766d6b9bec, 20640e9a-e8fc-48f6-af45-7c4e80729717, 1b668588-2533-4ab0-8775-5fab04ade133, e96fb224-be29-486b-9692-9959f7090bd5] The logs suggest to retry to remove the snapshot again - 2021-10-26 22:36:22,628+01 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [666e0f97-8b02-4b1e-80d4-2a640dd28d90] Merging of snapshot '679b1d4f-b36e-427a-8545-7ff7409a2e8a' images '4e8ea055-54bc-4a19-813c-79f6fe3e2db1'..'069d1cc2-cd2d-4160-8ca0-657f6f57c168' failed. Images have been marked illegal and can no longer be previewed or reverted to. Please retry Live Merge on the snapshot to complete the operation. Can you try it? On Thu, 28 Oct 2021 at 18:08, <ivohccarvalho@gmail.com> wrote:
Logs are here for the host, SPM and Engine - https://we.tl/t-v5Vm2QYBIl (sorry, did not find a better way)
The VM ID with this problem is 81e23292-4e43-4d9f-99ce-917936cce441.
Thanks again. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/4IC2RC44FR4XUP...
-- Regards, Eyal Shenitzky
Hello again. Yes, doing it manually usually works but next day fails again. Could this cause fs corruption on the volumes? Is it possible?
participants (3)
-
Benny Zlotnik -
Eyal Shenitzky -
ivohccarvalho＠gmail.com