[ovirt-users] USER_CREATE_SNAPSHOT_FINISHED_FAILURE with Cinder storage stuck

[Matthias Leopold]

Hi,

we're experimenting with Cinder/Ceph Storage on oVirt 4.1.3. When we 
tried to snapshot a VM (2 disks on Cinder storage domain) the task never 
finished and now seems to be in an uninterruptible loop. We tried to 
stop it in various (brute force) ways, but the below messages (one of 
the disks as an example) are cluttering engine.log every 10 seconds. We 
tried the following:

- deleting the VM
- restarting ovirt-engine service
- vdsClient -s 0 getAllTasksStatuses on SPM host (no result)
- restarting vdsmd service on SPM host
- /usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh -u engine -d 
engine -c c841c979-70ea-4e06-b9c4-9c5ce014d76d

None of this helped. How do we get rid of this failed transaction?

thx
matthias

2017-08-21 16:40:44,798+02 INFO 
[org.ovirt.engine.core.utils.transaction.TransactionSupport] 
(DefaultQuartzScheduler7) [080af640-bac3-4990-8bf4-6829551b538d] 
transaction rolled back
2017-08-21 16:40:44,799+02 ERROR 
[org.ovirt.engine.core.bll.job.ExecutionHandler] 
(DefaultQuartzScheduler7) [080af640-bac3-4990-8bf4-6829551b538d] 
Exception: org.springframework.dao.DataIntegrityViolationException: 
CallableStatementCallback; SQL [{call insertstep(?, ?, ?, ?, ?, ?, ?, ?, 
?, ?, ?, ?, ?, ?)}]; ERROR: insert or update on table "step" violates 
foreign key constraint "fk_step_job"
2017-08-21 16:40:44,805+02 ERROR 
[org.ovirt.engine.core.bll.snapshots.CreateAllSnapshotsFromVmCommand] 
(DefaultQuartzScheduler7) [080af640-bac3-4990-8bf4-6829551b538d] Ending 
command 
'org.ovirt.engine.core.bll.snapshots.CreateAllSnapshotsFromVmCommand' 
with failure.
2017-08-21 16:40:44,807+02 WARN 
[org.ovirt.engine.core.bll.snapshots.CreateAllSnapshotsFromVmCommand] 
(DefaultQuartzScheduler7) [080af640-bac3-4990-8bf4-6829551b538d] No 
snapshot was created for VM 'c0235316-81c4-48be-9521-b86b338c7d20' which 
is in LOCKED status
2017-08-21 16:40:44,810+02 INFO 
[org.ovirt.engine.core.utils.transaction.TransactionSupport] 
(DefaultQuartzScheduler7) [080af640-bac3-4990-8bf4-6829551b538d] 
transaction rolled back
2017-08-21 16:40:44,810+02 WARN 
[org.ovirt.engine.core.bll.lock.InMemoryLockManager] 
(DefaultQuartzScheduler7) [080af640-bac3-4990-8bf4-6829551b538d] Trying 
to release exclusive lock which does not exist, lock key: 
'c0235316-81c4-48be-9521-b86b338c7d20VM'
2017-08-21 16:40:44,810+02 INFO 
[org.ovirt.engine.core.bll.snapshots.CreateAllSnapshotsFromVmCommand] 
(DefaultQuartzScheduler7) [080af640-bac3-4990-8bf4-6829551b538d] Lock 
freed to object 
'EngineLock:{exclusiveLocks='[c0235316-81c4-48be-9521-b86b338c7d20=VM]', 
sharedLocks=''}'
2017-08-21 16:40:44,829+02 ERROR 
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] 
(DefaultQuartzScheduler7) [080af640-bac3-4990-8bf4-6829551b538d] 
EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_FAILURE(69), Correlation ID: 
080af640-bac3-4990-8bf4-6829551b538d, Job ID: 
a3be8af1-8d33-4d35-9672-215ac7c9959f, Call Stack: null, Custom Event ID: 
-1, Message: Failed to complete snapshot 'test' creation for VM '<UNKNOWN>'.
2017-08-21 16:40:44,829+02 ERROR 
[org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] 
(DefaultQuartzScheduler7) [080af640-bac3-4990-8bf4-6829551b538d] Failed 
invoking callback end method 'onFailed' for command 
'c841c979-70ea-4e06-b9c4-9c5ce014d76d' with exception 'null', the 
callback is marked for end method retries