disk pending in "finalizing" state
ovirt-engine-4.5.1.3-1.el8.noarch Hello I have a situation where a disk is stuck in finalizing state derived by trying to backup via veeam. backup process is interrupted and I have cleared the job states with the dbutils scripts (/usr/share/ovirt-engine/setup/dbutils/task_cleaner.sh) even if the script didn't advice about any pending job... I tryied to unlock_entity of the disk but for the script there is no disk locked. In the ovirt-engine gui, the disk is "finalizing" And I'm stuck here Can someone address me somewhere?
this means the transfer is finalizing can you provide the output of: $ psql -U engine -d engine -c "\x on" -c "select * from image_transfers" as well as engine logs On Mon, Aug 1, 2022 at 2:41 PM Diego Ercolani <diego.ercolani@ssis.sm> wrote:
ovirt-engine-4.5.1.3-1.el8.noarch
Hello I have a situation where a disk is stuck in finalizing state derived by trying to backup via veeam.
backup process is interrupted and I have cleared the job states with the dbutils scripts (/usr/share/ovirt-engine/setup/dbutils/task_cleaner.sh) even if the script didn't advice about any pending job...
I tryied to unlock_entity of the disk but for the script there is no disk locked. In the ovirt-engine gui, the disk is "finalizing" And I'm stuck here
Can someone address me somewhere? _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/6VCE6SC6RZKKH6...
Expanded display is on. -[ RECORD 1 ]-------------+---------------------------------------- command_id | eecdc5fc-4b7a-44f4-afed-9abb0cd12534 command_type | 1024 phase | 7 last_updated | 2022-07-31 10:49:06.791+00 message | vds_id | 29e930c3-d72b-4fc4-828e-e6d5b2b2582a disk_id | 7a05ff72-370e-4d0c-ab56-5f161cc98318 imaged_ticket_id | proxy_uri | https://ovirt-engine.ovirt:54323/images bytes_sent | 0 bytes_total | 64424509440 type | 1 active | f daemon_uri | https://ovirt-node2.ovirt:54322/images client_inactivity_timeout | 600 image_format | 5 backend | 1 backup_id | d64e0f6a-fbfe-4d13-a155-fdb62f4d09af client_type | 2 shallow | f timeout_policy | legacy This is the /var/log/ovirt-engine directory: https://cloud.ssis.sm/index.php/s/baazZPS8e8FQDFF
I see: engine.log-20220727.gz:2022-07-26 22:33:35,407Z INFO [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [ee684838-5d33-4f03-b07f-453a69effa97] Updating image transfer 'eecdc5fc-4b7a-44f4-afed-9abb0cd12534' phase from 'Finalizing Failure' to 'Finished Failure' engine.log-20220727.gz:2022-07-26 22:43:37,572Z INFO [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (default task-39) [a9e25b07-9a2f-40d8-886d-d9b613a83c10] Updating image transfer 'eecdc5fc-4b7a-44f4-afed-9abb0cd12534' phase from 'Finished Failure' to 'Finalizing Success' So looks like the transfer failed, but it was later finalized moving it from FINISHED_FAILURE to FINALIZING_SUCCESS, we have a bug to prevent clients from changing the transfer's status like this, fix should land in 4.5.3 You can remove the transfer from the image_transfers table (after backing up the database) On Mon, Aug 1, 2022 at 4:24 PM Diego Ercolani <diego.ercolani@ssis.sm> wrote:
Expanded display is on. -[ RECORD 1 ]-------------+---------------------------------------- command_id | eecdc5fc-4b7a-44f4-afed-9abb0cd12534 command_type | 1024 phase | 7 last_updated | 2022-07-31 10:49:06.791+00 message | vds_id | 29e930c3-d72b-4fc4-828e-e6d5b2b2582a disk_id | 7a05ff72-370e-4d0c-ab56-5f161cc98318 imaged_ticket_id | proxy_uri | https://ovirt-engine.ovirt:54323/images bytes_sent | 0 bytes_total | 64424509440 type | 1 active | f daemon_uri | https://ovirt-node2.ovirt:54322/images client_inactivity_timeout | 600 image_format | 5 backend | 1 backup_id | d64e0f6a-fbfe-4d13-a155-fdb62f4d09af client_type | 2 shallow | f timeout_policy | legacy
This is the /var/log/ovirt-engine directory: https://cloud.ssis.sm/index.php/s/baazZPS8e8FQDFF _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RU5GRIWKAUVWNI...
In data lunedì 1 agosto 2022 15:55:56 CEST, Benny Zlotnik ha scritto:
So looks like the transfer failed, but it was later finalized moving it from FINISHED_FAILURE to FINALIZING_SUCCESS, we have a bug to prevent clients from changing the transfer's status like this, fix should land in 4.5.3 When you talk "client" in this case I can assure you I didn't change the DB manually. But I don't know if Veeam did it. Anyway thank you for the support
Diego
There was no manual database change, I meant calling the /finalize endpoint can change the phase of a transfer and move it to an incorrect state like happened in your case. In your scenario the client is Veeam, but this is not their fault, but an issue in oVirt, since we should not change the phase in such cases. Let me know if manually removing the transfer helped. On Mon, Aug 1, 2022 at 4:59 PM Diego Ercolani <diego.ercolani@ssis.sm> wrote:
In data lunedì 1 agosto 2022 15:55:56 CEST, Benny Zlotnik ha scritto:
So looks like the transfer failed, but it was later finalized moving it from FINISHED_FAILURE to FINALIZING_SUCCESS, we have a bug to prevent clients from changing the transfer's status like this, fix should land in 4.5.3 When you talk "client" in this case I can assure you I didn't change the DB manually. But I don't know if Veeam did it. Anyway thank you for the support
Diego
Anyway, I removed the image from the image_transfer table, in the GUI the state changed to "OK" but when I try to remove the snapshots Veeam left it says thai engine cannot remove during backup: 2022-08-01 14:07:26,181Z INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-88) [] FINISH, GlusterTasksListVDSCommand, return: [], log id: 66bc2d26 2022-08-01 14:07:33,387Z INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (default task-151) [0c676089-1b52-4593-a0b3-af70b56ec134] Lock Acquired to object 'EngineLock:{exclusiveLocks='[86f0ec22-7897-4a42-98dc-9bd54fb8441f=VM]', sharedLocks=''}' 2022-08-01 14:07:33,388Z WARN [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (default task-151) [0c676089-1b52-4593-a0b3-af70b56ec134] Validation of action 'RemoveSnapshot' failed for user admin@internal-authz. Reasons: VAR__TYPE__SNAPSHOT,VAR__ACTION__REMOVE,ACTION_TYPE_FAILED_VM_IS_DURING_BACKUP 2022-08-01 14:07:33,388Z INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (default task-151) [0c676089-1b52-4593-a0b3-af70b56ec134] Lock freed to object 'EngineLock:{exclusiveLocks='[86f0ec22-7897-4a42-98dc-9bd54fb8441f=VM]', sharedLocks=''}' And if I try to copy the current disk it refuses saying that is locked: 2022-08-01 14:09:19,477Z INFO [org.ovirt.engine.core.bll.storage.disk.MoveOrCopyDiskCommand] (default task-159) [991ed393-0e88-4839-ac5a-ab9fcff9ecc1] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[7a05ff72-370e-4d0c-ab56-5f161cc98318=DISK]', sharedLocks=''}' 2022-08-01 14:09:19,477Z WARN [org.ovirt.engine.core.bll.storage.disk.MoveOrCopyDiskCommand] (default task-159) [991ed393-0e88-4839-ac5a-ab9fcff9ecc1] Validation of action 'MoveOrCopyDisk' failed for user admin@internal-authz. Reasons: VAR__ACTION__COPY,VAR__TYPE__DISK,ACTION_TYPE_FAILED_DISK_IS_LOCKED
Can you finalize the backup from Veeam? The /finalize endpoint for backups should initiate the snapshot removal On Mon, Aug 1, 2022 at 5:10 PM Diego Ercolani <diego.ercolani@ssis.sm> wrote:
Anyway, I removed the image from the image_transfer table, in the GUI the state changed to "OK" but when I try to remove the snapshots Veeam left it says thai engine cannot remove during backup: 2022-08-01 14:07:26,181Z INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-88) [] FINISH, GlusterTasksListVDSCommand, return: [], log id: 66bc2d26 2022-08-01 14:07:33,387Z INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (default task-151) [0c676089-1b52-4593-a0b3-af70b56ec134] Lock Acquired to object 'EngineLock:{exclusiveLocks='[86f0ec22-7897-4a42-98dc-9bd54fb8441f=VM]', sharedLocks=''}' 2022-08-01 14:07:33,388Z WARN [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (default task-151) [0c676089-1b52-4593-a0b3-af70b56ec134] Validation of action 'RemoveSnapshot' failed for user admin@internal-authz. Reasons: VAR__TYPE__SNAPSHOT,VAR__ACTION__REMOVE,ACTION_TYPE_FAILED_VM_IS_DURING_BACKUP 2022-08-01 14:07:33,388Z INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (default task-151) [0c676089-1b52-4593-a0b3-af70b56ec134] Lock freed to object 'EngineLock:{exclusiveLocks='[86f0ec22-7897-4a42-98dc-9bd54fb8441f=VM]', sharedLocks=''}'
And if I try to copy the current disk it refuses saying that is locked: 2022-08-01 14:09:19,477Z INFO [org.ovirt.engine.core.bll.storage.disk.MoveOrCopyDiskCommand] (default task-159) [991ed393-0e88-4839-ac5a-ab9fcff9ecc1] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[7a05ff72-370e-4d0c-ab56-5f161cc98318=DISK]', sharedLocks=''}' 2022-08-01 14:09:19,477Z WARN [org.ovirt.engine.core.bll.storage.disk.MoveOrCopyDiskCommand] (default task-159) [991ed393-0e88-4839-ac5a-ab9fcff9ecc1] Validation of action 'MoveOrCopyDisk' failed for user admin@internal-authz. Reasons: VAR__ACTION__COPY,VAR__TYPE__DISK,ACTION_TYPE_FAILED_DISK_IS_LOCKED
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/GO6HQ2X3Z6EIRP...
Today Veem seem that have unlocked the snapshot an finally I can remove "autogenerated" snapshots. Anyway, I just extracted last day logs from the engine, so you can analize it if you want: https://cloud.ssis.sm/index.php/s/RT9EHBys5eZ87oo
participants (2)
-
Benny Zlotnik -
Diego Ercolani