[ovirt-users] Can't remove snapshot
Greg Padgett
gpadgett at redhat.com
Tue Feb 16 21:47:43 UTC 2016
On 02/16/2016 07:10 AM, Marcelo Leandro wrote:
> Hello,
> I have the same problem, i tried delete snapshot but it did not
> success, the status snapshot as illegal , look the engine.log follow
> you can see the messages error:
Hi Marcelo,
The problem in your log,
error = Drive image file could not be found, code = 13
is a little different but may have been triggered by the same bug in a
previous merge attempt.
In this case, would you run the "VM disk info gathering tool" from [1]?
See Adam's comment in the bug about its execution.
If this shows that the VM is no longer dependent on that missing volume,
then please shut down engine, back up the db, and run the "post-merge
failure repair script", also attached to [1].
Arguments in your case would be the VM name, snapshot name, and the UUID
of the image that is missing from your storage. (You may need to manually
mark the image as illegal first, [2]).
HTH,
Greg
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1306741
[2] UPDATE images SET imagestatus = 4 WHERE image_guid = '<imgId>'::UUID;
>
>
> 2016-02-16 08:46:20,059 INFO
> [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback]
> (DefaultQuartzScheduler_Worker-57) [46dd2ef7] Waiting on Live Merge
> child commands to complete
> 2016-02-16 08:46:21,069 INFO
> [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand]
> (DefaultQuartzScheduler_Worker-40) [15b703ee] Waiting on Live Merge
> command step 'MERGE' to complete
> 2016-02-16 08:46:22,072 INFO
> [org.ovirt.engine.core.bll.MergeCommandCallback]
> (DefaultQuartzScheduler_Worker-65) [30cdf6ed] Waiting on merge command
> to complete
> 2016-02-16 08:46:23,670 INFO
> [org.ovirt.engine.core.bll.RemoveSnapshotCommand] (default task-48)
> [5e0c088f] Lock Acquired to object
> 'EngineLock:{exclusiveLocks='[94d788f4-eba4-49ee-8091-80028cc46627=<VM,
> ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
> 2016-02-16 08:46:23,795 INFO
> [org.ovirt.engine.core.bll.RemoveSnapshotCommand] (default task-48)
> [5e0c088f] Running command: RemoveSnapshotCommand internal: false.
> Entities affected : ID: 94d788f4-eba4-49ee-8091-80028cc46627 Type:
> VMAction group MANIPULATE_VM_SNAPSHOTS with role type USER
> 2016-02-16 08:46:23,824 INFO
> [org.ovirt.engine.core.bll.RemoveSnapshotCommand] (default task-48)
> [5e0c088f] Lock freed to object
> 'EngineLock:{exclusiveLocks='[94d788f4-eba4-49ee-8091-80028cc46627=<VM,
> ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
> 2016-02-16 08:46:23,876 INFO
> [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand]
> (pool-7-thread-5) [1be123ac] Running command:
> RemoveSnapshotSingleDiskLiveCommand internal: true. Entities affected
> : ID: 00000000-0000-0000-0000-000000000000 Type: Storage
> 2016-02-16 08:46:23,921 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (default task-48) [] Correlation ID: 5e0c088f, Job ID:
> aa811e83-24fb-4658-b849-d36439f58d95, Call Stack: null, Custom Event
> ID: -1, Message: Snapshot 'BKP the VM' deletion for VM
> 'Servidor-Cliente' was initiated by admin at internal.
> 2016-02-16 08:46:24,093 INFO
> [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand]
> (DefaultQuartzScheduler_Worker-14) [1be123ac] Executing Live Merge
> command step 'EXTEND'
> 2016-02-16 08:46:24,122 INFO
> [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback]
> (DefaultQuartzScheduler_Worker-14) [] Waiting on Live Merge child
> commands to complete
> 2016-02-16 08:46:24,133 INFO
> [org.ovirt.engine.core.bll.MergeExtendCommand] (pool-7-thread-6)
> [766ffc9f] Running command: MergeExtendCommand internal: true.
> Entities affected : ID: c2dc0101-748e-4a7b-9913-47993eaa52bd Type:
> Storage
> 2016-02-16 08:46:24,134 INFO
> [org.ovirt.engine.core.bll.MergeExtendCommand] (pool-7-thread-6)
> [766ffc9f] Base and top image sizes are the same; no image size update
> required
> 2016-02-16 08:46:25,133 INFO
> [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand]
> (DefaultQuartzScheduler_Worker-16) [1be123ac] Executing Live Merge
> command step 'MERGE'
> 2016-02-16 08:46:25,168 INFO [org.ovirt.engine.core.bll.MergeCommand]
> (pool-7-thread-7) [1b7bc421] Running command: MergeCommand internal:
> true. Entities affected : ID: c2dc0101-748e-4a7b-9913-47993eaa52bd
> Type: Storage
> 2016-02-16 08:46:25,169 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand]
> (pool-7-thread-7) [1b7bc421] START, MergeVDSCommand(HostName = Host01,
> MergeVDSCommandParameters:{runAsync='true',
> hostId='d4f29978-1540-44d9-ab22-1e6ff750059f',
> vmId='94d788f4-eba4-49ee-8091-80028cc46627',
> storagePoolId='77e24b20-9d21-4952-a089-3c5c592b4e6d',
> storageDomainId='c2dc0101-748e-4a7b-9913-47993eaa52bd',
> imageGroupId='b7a27d0c-57cc-490e-a3f8-b4981310a9b0',
> imageId='7f8bb099-9a18-4e89-bf48-57e56e5770d2',
> baseImageId='2e59f7f2-9e30-460e-836a-5e0d3d625059',
> topImageId='7f8bb099-9a18-4e89-bf48-57e56e5770d2', bandwidth='0'}),
> log id: 2a7ab7b7
> 2016-02-16 08:46:25,176 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand]
> (pool-7-thread-7) [1b7bc421] Failed in 'MergeVDS' method
> 2016-02-16 08:46:25,179 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (pool-7-thread-7) [1b7bc421] Correlation ID: null, Call Stack: null,
> Custom Event ID: -1, Message: VDSM Host01 command failed: Drive image
> file could not be found
> 2016-02-16 08:46:25,179 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand]
> (pool-7-thread-7) [1b7bc421] Command
> 'org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand' return
> value 'StatusOnlyReturnForXmlRpc [status=StatusForXmlRpc [code=13,
> message=Drive image file could not be found]]'
> 2016-02-16 08:46:25,179 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand]
> (pool-7-thread-7) [1b7bc421] HostName = Host01
> 2016-02-16 08:46:25,179 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand]
> (pool-7-thread-7) [1b7bc421] Command 'MergeVDSCommand(HostName =
> Host01, MergeVDSCommandParameters:{runAsync='true',
> hostId='d4f29978-1540-44d9-ab22-1e6ff750059f',
> vmId='94d788f4-eba4-49ee-8091-80028cc46627',
> storagePoolId='77e24b20-9d21-4952-a089-3c5c592b4e6d',
> storageDomainId='c2dc0101-748e-4a7b-9913-47993eaa52bd',
> imageGroupId='b7a27d0c-57cc-490e-a3f8-b4981310a9b0',
> imageId='7f8bb099-9a18-4e89-bf48-57e56e5770d2',
> baseImageId='2e59f7f2-9e30-460e-836a-5e0d3d625059',
> topImageId='7f8bb099-9a18-4e89-bf48-57e56e5770d2', bandwidth='0'})'
> execution failed: VDSGenericException: VDSErrorException: Failed to
> MergeVDS, error = Drive image file could not be found, code = 13
> 2016-02-16 08:46:25,179 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand]
> (pool-7-thread-7) [1b7bc421] FINISH, MergeVDSCommand, log id: 2a7ab7b7
> 2016-02-16 08:46:25,180 ERROR [org.ovirt.engine.core.bll.MergeCommand]
> (pool-7-thread-7) [1b7bc421] Command
> 'org.ovirt.engine.core.bll.MergeCommand' failed: EngineException:
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
> VDSGenericException: VDSErrorException: Failed to MergeVDS, error =
> Drive image file could not be found, code = 13 (Failed with error
> imageErr and code 13)
> 2016-02-16 08:46:25,186 ERROR [org.ovirt.engine.core.bll.MergeCommand]
> (pool-7-thread-7) [1b7bc421] Transaction rolled-back for command
> 'org.ovirt.engine.core.bll.MergeCommand'.
> 2016-02-16 08:46:26,159 INFO
> [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback]
> (DefaultQuartzScheduler_Worker-25) [15b703ee] Waiting on Live Merge
> child commands to complete
> 2016-02-16 08:46:27,164 ERROR
> [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand]
> (DefaultQuartzScheduler_Worker-32) [1be123ac] Failed child command
> status for step 'MERGE'
> 2016-02-16 08:46:27,497 INFO
> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
> (DefaultQuartzScheduler_Worker-37) [30cdf6ed] VM job
> '77669e28-4aa2-4038-b7b6-1a949a1d039e': In progress, updating
> 2016-02-16 08:46:28,192 ERROR
> [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand]
> (DefaultQuartzScheduler_Worker-87) [1be123ac] Merging of snapshot
> '119f668e-af60-49ea-aa08-735be8af0a7d' images
> '2e59f7f2-9e30-460e-836a-5e0d3d625059'..'7f8bb099-9a18-4e89-bf48-57e56e5770d2'
> failed. Images have been marked illegal and can no longer be previewed
> or reverted to. Please retry Live Merge on the snapshot to complete
> the operation.
> 2016-02-16 08:46:28,204 INFO
> [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback]
> (DefaultQuartzScheduler_Worker-87) [5e0c088f] All Live Merge child
> commands have completed, status 'FAILED'
> 2016-02-16 08:46:29,216 ERROR
> [org.ovirt.engine.core.bll.RemoveSnapshotCommand]
> (DefaultQuartzScheduler_Worker-89) [5e0c088f] Ending command
> 'org.ovirt.engine.core.bll.RemoveSnapshotCommand' with failure.
> 2016-02-16 08:46:29,263 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler_Worker-89) [5e0c088f] Correlation ID:
> 5e0c088f, Job ID: aa811e83-24fb-4658-b849-d36439f58d95, Call Stack:
> null, Custom Event ID: -1, Message: Failed to delete snapshot 'BKP the
> VM' for VM 'Servidor-Cliente'.
> 2016-02-16 08:46:30,287 INFO
> [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback]
> (DefaultQuartzScheduler_Worker-33) [] Waiting on Live Merge child
> commands to complete
> 2016-02-16 08:46:31,298 INFO
> [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand]
> (DefaultQuartzScheduler_Worker-21) [15b703ee] Waiting on Live Merge
> command step 'MERGE' to complete
> 2016-02-16 08:46:32,301 INFO
> [org.ovirt.engine.core.bll.MergeCommandCallback]
> (DefaultQuartzScheduler_Worker-68) [30cdf6ed] Waiting on merge command
> to complete
> 2016-02-16 08:46:40,304 INFO
> [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback]
> (DefaultQuartzScheduler_Worker-55) [280a8a32] Waiting on Live Merge
> child commands to complete
> 2016-02-16 08:46:41,308 INFO
> [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand]
> (DefaultQuartzScheduler_Worker-54) [15b703ee] Waiting on Live Merge
> command step 'MERGE' to complete
> 2016-02-16 08:46:42,312 INFO
> [org.ovirt.engine.core.bll.MergeCommandCallback]
> (DefaultQuartzScheduler_Worker-57) [30cdf6ed] Waiting on merge command
> to complete
> 2016-02-16 08:46:42,850 INFO
> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
> (DefaultQuartzScheduler_Worker-84) [] VM job
> '77669e28-4aa2-4038-b7b6-1a949a1d039e': In progress, updating
> 2016-02-16 08:46:42,854 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand]
> (DefaultQuartzScheduler_Worker-84) [] START,
> FullListVDSCommand(HostName = ,
> FullListVDSCommandParameters:{runAsync='true',
> hostId='aebc403a-ec4e-4346-9029-6353d5d76f01',
> vds='Host[,aebc403a-ec4e-4346-9029-6353d5d76f01]',
> vmIds='[6af1f9c3-7210-45c3-90dc-bd7793346c0c]'}), log id: 74961dad
>
> I cannot see the snapshot disk at the storage domain:
>
> [root@ ~]# cd /rhev/data-center/77e24b20-9d21-4952-a089-3c5c592b4e6d/c1938052-7524-404c-bac9-f238227269ea/images/b7a27d0c-57cc-490e-a3f8-b4981310a9b0/
> [root@ b7a27d0c-57cc-490e-a3f8-b4981310a9b0]# ls
> 2e59f7f2-9e30-460e-836a-5e0d3d625059 2e59f7f2-9e30-460e-836a-5e0d3d625059.meta
>
> Thanks.
>
> 2016-02-09 21:30 GMT-03:00 Greg Padgett <gpadgett at redhat.com>:
>> On 02/09/2016 06:08 AM, Michal Skrivanek wrote:
>>>
>>>
>>>> On 03 Feb 2016, at 10:37, Rik Theys <Rik.Theys at esat.kuleuven.be> wrote:
>>>>
>>>> Hi,
>>>>
>>>> In the mean time I've noticed the following entries in our periodic
>>>> logcheck output:
>>>>
>>>> Feb 3 09:05:53 orinoco journal: block copy still active: disk 'vda' not
>>>> ready for pivot yet
>>>> Feb 3 09:05:53 orinoco journal: vdsm root ERROR Unhandled
>>>> exception#012Traceback (most recent call last):#012 File
>>>> "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 734, in
>>>> wrapper#012 return f(*a, **kw)#012 File
>>>> "/usr/share/vdsm/virt/vm.py", line 5168, in run#012
>>>> self.tryPivot()#012 File "/usr/share/vdsm/virt/vm.py", line 5137, in
>>>> tryPivot#012 ret = self.vm._dom.blockJobAbort(self.drive.name,
>>>> flags)#012 File "/usr/share/vdsm/virt/virdomain.py", line 68, in f#012
>>>> ret = attr(*args, **kwargs)#012 File
>>>> "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124,
>>>> in wrapper#012 ret = f(*args, **kwargs)#012 File
>>>> "/usr/lib64/python2.7/site-packages/libvirt.py", line 733, in
>>>> blockJobAbort#012 if ret == -1: raise libvirtError
>>>> ('virDomainBlockJobAbort() failed', dom=self)#012libvirtError: block
>>>> copy still active: disk 'vda' not ready for pivot yet
>>>>
>>>> This is from the host running the VM.
>>>>
>>>> Note that this host is not the SPM of the cluster. I always thought all
>>>> operations on disk volumes happened on the SPM host?
>>>>
>>>> My question still remains:
>>>>
>>>>> I can see the snapshot in the "Disk snapshot" tab of the storage. It has
>>>>> a status of "illegal". Is it OK to (try to) remove this snapshot? Will
>>>>> this impact the running VM and/or disk image?
>>>
>>>
>>> No, it’s not ok to remove it while live merge(apparently) is still ongoing
>>> I guess that’s a live merge bug?
>>
>>
>> Indeed, this is bug 1302215.
>>
>> I wrote a sql script to help with cleanup in this scenario, which you can
>> find attached to the bug along with a description of how to use it[1].
>>
>> However, Rik, before trying that, would you be able to run the attached
>> script [2] (or just the db query within) and forward the output to me? I'd
>> like to make sure everything looks as it should before modifying the db
>> directly.
>>
>> Thanks,
>> Greg
>>
>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1302215#c13
>> (Also note that the engine should be stopped before running this.)
>>
>> [2] Arguments are the ovirt db name, db user, and the name of the vm you
>> were performing live merge on.
>>
>>
>>> Thanks,
>>> michal
>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Rik
>>>>
>>>> On 02/03/2016 10:26 AM, Rik Theys wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I created a snapshot of a running VM prior to an OS upgrade. The OS
>>>>> upgrade has now been succesful and I would like to remove the snapshot.
>>>>> I've selected the snapshot in the UI and clicked Delete to start the
>>>>> task.
>>>>>
>>>>> After a few minutes, the task has failed. When I click delete again on
>>>>> the same snapshot, the failed message is returned after a few seconds.
>>>>>
>>>>>> From browsing through the engine log (attached) it seems the snapshot
>>>>>
>>>>> was correctly merged in the first try but something went wrong in the
>>>>> finalizing fase. On retries, the log indicates the snapshot/disk image
>>>>> no longer exists and the removal of the snapshot fails for this reason.
>>>>>
>>>>> Is there any way to clean up this snapshot?
>>>>>
>>>>> I can see the snapshot in the "Disk snapshot" tab of the storage. It has
>>>>> a status of "illegal". Is it OK to (try to) remove this snapshot? Will
>>>>> this impact the running VM and/or disk image?
>>>>>
>>>>> Regards,
>>>>>
>>>>> Rik
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>
>>>>
>>>>
>>>> --
>>>> Rik Theys
>>>> System Engineer
>>>> KU Leuven - Dept. Elektrotechniek (ESAT)
>>>> Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
>>>> +32(0)16/32.11.07
>>>> ----------------------------------------------------------------
>>>> <<Any errors in spelling, tact or fact are transmission errors>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
More information about the Users
mailing list