[ovirt-users] moving disk failed.. remained locked

Wed Feb 22 08:56:15 UTC 2017

On Wed, Feb 22, 2017 at 10:32 AM, Nir Soffer <nsoffer at redhat.com> wrote:
> On Wed, Feb 22, 2017 at 10:31 AM, Nir Soffer <nsoffer at redhat.com> wrote:
>> On Mon, Feb 20, 2017 at 4:49 PM, Gianluca Cecchi
>> <gianluca.cecchi at gmail.com> wrote:
>>> Hello,
>>> I'm trying to move a disk from one storage domain A to another B in oVirt
>>> 4.1
>>> The corresponding VM is powered on in the mean time
>>>
>>> When executing the action, there was already in place a disk move from
>>> storage domain C to A (this move was for a disk of a powered off VM and then
>>> completed ok)
>>> I got this in events of webadmin gui for the failed move A -> B:
>>>
>>> Feb 20, 2017 2:42:00 PM Failed to complete snapshot 'Auto-generated for Live
>>> Storage Migration' creation for VM 'dbatest6'.
>>> Feb 20, 2017 2:40:51 PM VDSM ovmsrv06 command HSMGetAllTasksStatusesVDS
>>> failed: Error creating a new volume
>>> Feb 20, 2017 2:40:51 PM Snapshot 'Auto-generated for Live Storage Migration'
>>> creation for VM 'dbatest6' was initiated by admin at internal-authz.
>>>
>>>
>>> And in relevant vdsm.log of referred host ovmsrv06
>>>
>>> 2017-02-20 14:41:44,899 ERROR (tasks/8) [storage.Volume] Unexpected error
>>> (volume:1087)
>>> Traceback (most recent call last):
>>>   File "/usr/share/vdsm/storage/volume.py", line 1081, in create
>>>     cls.newVolumeLease(metaId, sdUUID, volUUID)
>>>   File "/usr/share/vdsm/storage/volume.py", line 1361, in newVolumeLease
>>>     return cls.manifestClass.newVolumeLease(metaId, sdUUID, volUUID)
>>>   File "/usr/share/vdsm/storage/blockVolume.py", line 310, in newVolumeLease
>>>     sanlock.init_resource(sdUUID, volUUID, [(leasePath, leaseOffset)])
>>> SanlockException: (-202, 'Sanlock resource init failure', 'Sanlock
>>> exception')
>>
>> This means that sanlock could not initialize a lease in the new volume created
>> for the snapshot.

David, looking in sanlock log - we don't see any error matching this failure,
but the domain 900b1853-e192-4661-a0f9-7c7c396f6f49 has renewal errors.

I guess because sanlock_init_resource is implemented in the library,
not going trough sanlock deamon?

2017-02-20 14:30:09+0100 1050804 [11738]: 900b1853 aio timeout RD
0x7f41d00008c0:0x7f41d00008d0:0x7f41e2afa000 ioto 10 to_count 1
2017-02-20 14:30:09+0100 1050804 [11738]: s3 delta_renew read timeout
10 sec offset 0 /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/ids
2017-02-20 14:30:09+0100 1050804 [11738]: s3 renewal error -202
delta_length 10 last_success 1050773
2017-02-20 14:30:11+0100 1050806 [11738]: 900b1853 aio collect RD
0x7f41d00008c0:0x7f41d00008d0:0x7f41e2afa000 result 1048576:0 match
reap
2017-02-20 14:35:58+0100 1051153 [11738]: 900b1853 aio timeout RD
0x7f41d00008c0:0x7f41d00008d0:0x7f41e2afa000 ioto 10 to_count 2
2017-02-20 14:35:58+0100 1051153 [11738]: s3 delta_renew read timeout
10 sec offset 0 /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/ids
2017-02-20 14:35:58+0100 1051153 [11738]: s3 renewal error -202
delta_length 10 last_success 1051122
2017-02-20 14:36:01+0100 1051156 [11738]: 900b1853 aio collect RD
0x7f41d00008c0:0x7f41d00008d0:0x7f41e2afa000 result 1048576:0 match
reap
2017-02-20 14:44:36+0100 1051671 [11738]: 900b1853 aio timeout RD
0x7f41d00008c0:0x7f41d00008d0:0x7f41e2afa000 ioto 10 to_count 3
2017-02-20 14:44:36+0100 1051671 [11738]: s3 delta_renew read timeout
10 sec offset 0 /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/ids
2017-02-20 14:44:36+0100 1051671 [11738]: s3 renewal error -202
delta_length 10 last_success 1051641
2017-02-20 14:44:37+0100 1051672 [11738]: 900b1853 aio collect RD
0x7f41d00008c0:0x7f41d00008d0:0x7f41e2afa000 result 1048576:0 match
reap
2017-02-20 14:48:02+0100 1051877 [11738]: 900b1853 aio timeout RD
0x7f41d00008c0:0x7f41d00008d0:0x7f41e2afa000 ioto 10 to_count 4
2017-02-20 14:48:02+0100 1051877 [11738]: s3 delta_renew read timeout
10 sec offset 0 /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/ids
2017-02-20 14:48:02+0100 1051877 [11738]: s3 renewal error -202
delta_length 10 last_success 1051846
2017-02-20 14:48:02+0100 1051877 [11738]: 900b1853 aio collect RD
0x7f41d00008c0:0x7f41d00008d0:0x7f41e2afa000 result 1048576:0 match
reap

Gianluca, what is domain 900b1853-e192-4661-a0f9-7c7c396f6f49?

is this the domain you are migrating to in the same time?

Can you share the output of:

    sanlock client renewal -s 900b1853-e192-4661-a0f9-7c7c396f6f49

>>> 2017-02-20 14:41:44,900 ERROR (tasks/8) [storage.TaskManager.Task]
>>> (Task='d694b892-b078-4d86-a035-427ee4fb3b13') Unexpected error (task:870)
>>> Traceback (most recent call last):
>>>   File "/usr/share/vdsm/storage/task.py", line 877, in _run
>>>     return fn(*args, **kargs)
>>>   File "/usr/share/vdsm/storage/task.py", line 333, in run
>>>     return self.cmd(*self.argslist, **self.argsdict)
>>>   File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line
>>> 79, in wrapper
>>>     return method(self, *args, **kwargs)
>>>   File "/usr/share/vdsm/storage/sp.py", line 1929, in createVolume
>>>     initialSize=initialSize)
>>>   File "/usr/share/vdsm/storage/sd.py", line 762, in createVolume
>>>     initialSize=initialSize)
>>>   File "/usr/share/vdsm/storage/volume.py", line 1089, in create
>>>     (volUUID, e))
>>> VolumeCreationError: Error creating a new volume: (u"Volume creation
>>> d0d938bd-1479-49cb-93fb-85b6a32d6cb4 failed: (-202, 'Sanlock resource init
>>> failure', 'Sanlock exception')",)
>>> 2017-02-20 14:41:44,941 INFO  (tasks/8) [storage.Volume] Metadata rollback
>>> for sdUUID=900b1853-e192-4661-a0f9-7c7c396f6f49 offs=8 (blockVolume:448)
>>>
>>>
>>> Was the error generated due to the other migration still in progress?
>>> Is there a limit of concurrent migrations from/to a particular storage
>>> domain?
>>
>> No, maybe your network was overloaded by the concurrent migrations?
>>
>>>
>>> Now I would like to retry, but I see that the disk is in state locked with
>>> hourglass.
>>> The autogenerated snapshot of the failed action was apparently removed with
>>> success as I don't see it.
>>>
>>> How can I proceed to move the disk?
>>>
>>> Thanks in advance,
>>> Gianluca
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>