10:56 a.m.
On Wed, Feb 22, 2017 at 10:32 AM, Nir Soffer <nsoffer(a)redhat.com> wrote:
On Wed, Feb 22, 2017 at 10:31 AM, Nir Soffer
<nsoffer(a)redhat.com> wrote:
> On Mon, Feb 20, 2017 at 4:49 PM, Gianluca Cecchi
> <gianluca.cecchi(a)gmail.com> wrote:
>> Hello,
>> I'm trying to move a disk from one storage domain A to another B in oVirt
>> 4.1
>> The corresponding VM is powered on in the mean time
>>
>> When executing the action, there was already in place a disk move from
>> storage domain C to A (this move was for a disk of a powered off VM and then
>> completed ok)
>> I got this in events of webadmin gui for the failed move A -> B:
>>
>> Feb 20, 2017 2:42:00 PM Failed to complete snapshot 'Auto-generated for Live
>> Storage Migration' creation for VM 'dbatest6'.
>> Feb 20, 2017 2:40:51 PM VDSM ovmsrv06 command HSMGetAllTasksStatusesVDS
>> failed: Error creating a new volume
>> Feb 20, 2017 2:40:51 PM Snapshot 'Auto-generated for Live Storage
Migration'
>> creation for VM 'dbatest6' was initiated by admin@internal-authz.
>>
>>
>> And in relevant vdsm.log of referred host ovmsrv06
>>
>> 2017-02-20 14:41:44,899 ERROR (tasks/8) [storage.Volume] Unexpected error
>> (volume:1087)
>> Traceback (most recent call last):
>> File "/usr/share/vdsm/storage/volume.py", line 1081, in create
>> cls.newVolumeLease(metaId, sdUUID, volUUID)
>> File "/usr/share/vdsm/storage/volume.py", line 1361, in
newVolumeLease
>> return cls.manifestClass.newVolumeLease(metaId, sdUUID, volUUID)
>> File "/usr/share/vdsm/storage/blockVolume.py", line 310, in
newVolumeLease
>> sanlock.init_resource(sdUUID, volUUID, [(leasePath, leaseOffset)])
>> SanlockException: (-202, 'Sanlock resource init failure', 'Sanlock
>> exception')
>
> This means that sanlock could not initialize a lease in the new volume created
> for the snapshot.
David, looking in sanlock log - we don't see any error matching this failure,
but the domain 900b1853-e192-4661-a0f9-7c7c396f6f49 has renewal errors.
I guess because sanlock_init_resource is implemented in the library,
not going trough sanlock deamon?
2017-02-20 14:30:09+0100 1050804 [11738]: 900b1853 aio timeout RD
0x7f41d00008c0:0x7f41d00008d0:0x7f41e2afa000 ioto 10 to_count 1
2017-02-20 14:30:09+0100 1050804 [11738]: s3 delta_renew read timeout
10 sec offset 0 /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/ids
2017-02-20 14:30:09+0100 1050804 [11738]: s3 renewal error -202
delta_length 10 last_success 1050773
2017-02-20 14:30:11+0100 1050806 [11738]: 900b1853 aio collect RD
0x7f41d00008c0:0x7f41d00008d0:0x7f41e2afa000 result 1048576:0 match
reap
2017-02-20 14:35:58+0100 1051153 [11738]: 900b1853 aio timeout RD
0x7f41d00008c0:0x7f41d00008d0:0x7f41e2afa000 ioto 10 to_count 2
2017-02-20 14:35:58+0100 1051153 [11738]: s3 delta_renew read timeout
10 sec offset 0 /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/ids
2017-02-20 14:35:58+0100 1051153 [11738]: s3 renewal error -202
delta_length 10 last_success 1051122
2017-02-20 14:36:01+0100 1051156 [11738]: 900b1853 aio collect RD
0x7f41d00008c0:0x7f41d00008d0:0x7f41e2afa000 result 1048576:0 match
reap
2017-02-20 14:44:36+0100 1051671 [11738]: 900b1853 aio timeout RD
0x7f41d00008c0:0x7f41d00008d0:0x7f41e2afa000 ioto 10 to_count 3
2017-02-20 14:44:36+0100 1051671 [11738]: s3 delta_renew read timeout
10 sec offset 0 /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/ids
2017-02-20 14:44:36+0100 1051671 [11738]: s3 renewal error -202
delta_length 10 last_success 1051641
2017-02-20 14:44:37+0100 1051672 [11738]: 900b1853 aio collect RD
0x7f41d00008c0:0x7f41d00008d0:0x7f41e2afa000 result 1048576:0 match
reap
2017-02-20 14:48:02+0100 1051877 [11738]: 900b1853 aio timeout RD
0x7f41d00008c0:0x7f41d00008d0:0x7f41e2afa000 ioto 10 to_count 4
2017-02-20 14:48:02+0100 1051877 [11738]: s3 delta_renew read timeout
10 sec offset 0 /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/ids
2017-02-20 14:48:02+0100 1051877 [11738]: s3 renewal error -202
delta_length 10 last_success 1051846
2017-02-20 14:48:02+0100 1051877 [11738]: 900b1853 aio collect RD
0x7f41d00008c0:0x7f41d00008d0:0x7f41e2afa000 result 1048576:0 match
reap
Gianluca, what is domain 900b1853-e192-4661-a0f9-7c7c396f6f49?
is this the domain you are migrating to in the same time?
Can you share the output of:
sanlock client renewal -s 900b1853-e192-4661-a0f9-7c7c396f6f49
>> 2017-02-20 14:41:44,900 ERROR (tasks/8)
[storage.TaskManager.Task]
>> (Task='d694b892-b078-4d86-a035-427ee4fb3b13') Unexpected error
(task:870)
>> Traceback (most recent call last):
>> File "/usr/share/vdsm/storage/task.py", line 877, in _run
>> return fn(*args, **kargs)
>> File "/usr/share/vdsm/storage/task.py", line 333, in run
>> return self.cmd(*self.argslist, **self.argsdict)
>> File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py",
line
>> 79, in wrapper
>> return method(self, *args, **kwargs)
>> File "/usr/share/vdsm/storage/sp.py", line 1929, in createVolume
>> initialSize=initialSize)
>> File "/usr/share/vdsm/storage/sd.py", line 762, in createVolume
>> initialSize=initialSize)
>> File "/usr/share/vdsm/storage/volume.py", line 1089, in create
>> (volUUID, e))
>> VolumeCreationError: Error creating a new volume: (u"Volume creation
>> d0d938bd-1479-49cb-93fb-85b6a32d6cb4 failed: (-202, 'Sanlock resource init
>> failure', 'Sanlock exception')",)
>> 2017-02-20 14:41:44,941 INFO (tasks/8) [storage.Volume] Metadata rollback
>> for sdUUID=900b1853-e192-4661-a0f9-7c7c396f6f49 offs=8 (blockVolume:448)
>>
>>
>> Was the error generated due to the other migration still in progress?
>> Is there a limit of concurrent migrations from/to a particular storage
>> domain?
>
> No, maybe your network was overloaded by the concurrent migrations?
>
>>
>> Now I would like to retry, but I see that the disk is in state locked with
>> hourglass.
>> The autogenerated snapshot of the failed action was apparently removed with
>> success as I don't see it.
>>
>> How can I proceed to move the disk?
>>
>> Thanks in advance,
>> Gianluca
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>