<div dir="ltr">Hi Nir,<br><div><div class="gmail_extra"><br><div class="gmail_quote">On 16 February 2017 at 13:55, Nir Soffer <span dir="ltr"><<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Mon, Feb 13, 2017 at 3:35 PM, Doug Ingham <<a href="mailto:dougti@gmail.com">dougti@gmail.com</a>> wrote:<br>
> Hi Sahina,<br>
><br>
> On 13 February 2017 at 05:45, Sahina Bose <<a href="mailto:sabose@redhat.com">sabose@redhat.com</a>> wrote:<br>
>><br>
>> Any errors in the gluster mount logs for this gluster volume?<br>
>><br>
>> How about "gluster vol heal <volname> info" - does it list any entries to<br>
>> heal?<br>
><br>
><br>
> After more investigating, I found out that there is a sanlock daemon that<br>
> runs with VDSM, independently of the HE, so I'd basically have to bring the<br>
> volume down & wait for the leases to expire/delete them* before I can import<br>
> the domain.<br>
><br>
> *I understand removing /dom_md/leases/ should do the job?<br>
<br>
</span>No, the issue is probably dom_md/ids accessed by sanlock, but removing files<br>
accessed by sanlock will not help, an open file will remain open until sanlock<br>
close the file.<br></blockquote><div><br></div><div>I'm testing this with volume snapshots at the moment, so there are no processes accessing the new volume.<br></div><div> <br><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Did you try to reboot the host before installing it again? If you did and you<br>
still have these issues, you probably need to remove the previous installation<br>
properly before installing again.<br>
<br>
Adding Simone to help with uninstalling and reinstalling hosted engine.<br></blockquote><div><br></div><div>The Hosted-Engine database had been corrupted and the restore wasn't running correctly, so I installed a new engine on a new server - no restores or old data. The aim is to import the old storage domain into the new Engine & then import the VMs into the new storage domain.<br></div><div>My only problem with this is that there appear to be some file based leases somewhere that, unless I manage to locate & delete them, force me to wait for the leases to timeout before I can import the old storage domain.<br></div><div>To minimise downtime, I'm trying to avoid having to wait for the leases to timeout.<br><br></div><div>Regards,<br></div><div> Doug<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span class="HOEnZb"><font color="#888888"><br>
Nir<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
><br>
><br>
>><br>
>><br>
>> On Thu, Feb 9, 2017 at 11:57 PM, Doug Ingham <<a href="mailto:dougti@gmail.com">dougti@gmail.com</a>> wrote:<br>
>>><br>
>>> Some interesting output from the vdsm log...<br>
>>><br>
>>><br>
>>> 2017-02-09 15:16:24,051 INFO (jsonrpc/1) [storage.StorageDomain]<br>
>>> Resource namespace 01_img_60455567-ad30-42e3-<wbr>a9df-62fe86c7fd25 already<br>
>>> registered (sd:731)<br>
>>> 2017-02-09 15:16:24,051 INFO (jsonrpc/1) [storage.StorageDomain]<br>
>>> Resource namespace 02_vol_60455567-ad30-42e3-<wbr>a9df-62fe86c7fd25 already<br>
>>> registered (sd:740)<br>
>>> 2017-02-09 15:16:24,052 INFO (jsonrpc/1) [storage.SANLock] Acquiring<br>
>>> Lease(name='SDM',<br>
>>> path=u'/rhev/data-center/mnt/<wbr>glusterSD/localhost:data2/<wbr>60455567-ad30-42e3-a9df-6<br>
>>> 2fe86c7fd25/dom_md/leases', offset=1048576) for host id 1<br>
>>> (clusterlock:343)<br>
>>> 2017-02-09 15:16:24,057 INFO (jsonrpc/1) [storage.SANLock] Releasing<br>
>>> host id for domain 60455567-ad30-42e3-a9df-<wbr>62fe86c7fd25 (id: 1)<br>
>>> (clusterlock:305)<br>
>>> 2017-02-09 15:16:25,149 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC<br>
>>> call GlusterHost.list succeeded in 0.17 seconds (__init__:515)<br>
>>> 2017-02-09 15:16:25,264 INFO (Reactor thread)<br>
>>> [ProtocolDetector.<wbr>AcceptorImpl] Accepted connection from<br>
>>> ::ffff:<a href="http://127.0.0.1:55060" rel="noreferrer" target="_blank">127.0.0.1:55060</a> (protocoldetector:72)<br>
>>> 2017-02-09 15:16:25,270 INFO (Reactor thread)<br>
>>> [ProtocolDetector.Detector] Detected protocol stomp from<br>
>>> ::ffff:<a href="http://127.0.0.1:55060" rel="noreferrer" target="_blank">127.0.0.1:55060</a> (protocoldetector:127)<br>
>>> 2017-02-09 15:16:25,271 INFO (Reactor thread) [Broker.StompAdapter]<br>
>>> Processing CONNECT request (stompreactor:102)<br>
>>> 2017-02-09 15:16:25,271 INFO (JsonRpc (StompReactor))<br>
>>> [Broker.StompAdapter] Subscribe command received (stompreactor:129)<br>
>>> 2017-02-09 15:16:25,416 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC<br>
>>> call Host.getHardwareInfo succeeded in 0.01 seconds (__init__:515)<br>
>>> 2017-02-09 15:16:25,419 INFO (jsonrpc/6) [dispatcher] Run and protect:<br>
>>> repoStats(options=None) (logUtils:49)<br>
>>> 2017-02-09 15:16:25,419 INFO (jsonrpc/6) [dispatcher] Run and protect:<br>
>>> repoStats, Return response: {u'e8d04da7-ad3d-4227-a45d-<wbr>b5a29b2f43e5':<br>
>>> {'code': 0, 'actual': True<br>
>>> , 'version': 4, 'acquired': True, 'delay': '0.000854128', 'lastCheck':<br>
>>> '5.1', 'valid': True}, u'a77b8821-ff19-4d17-a3ce-<wbr>a6c3a69436d5': {'code': 0,<br>
>>> 'actual': True, 'vers<br>
>>> ion': 4, 'acquired': True, 'delay': '0.000966556', 'lastCheck': '2.6',<br>
>>> 'valid': True}} (logUtils:52)<br>
>>> 2017-02-09 15:16:25,447 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC<br>
>>> call Host.getStats succeeded in 0.03 seconds (__init__:515)<br>
>>> 2017-02-09 15:16:25,450 ERROR (JsonRpc (StompReactor)) [vds.dispatcher]<br>
>>> SSL error receiving from <yajsonrpc.betterAsyncore.<wbr>Dispatcher connected<br>
>>> ('::ffff:127.0.0.1', 55060, 0, 0) at 0x7f69c0043cf8>: unexpected eof<br>
>>> (betterAsyncore:113)<br>
>>> 2017-02-09 15:16:25,812 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC<br>
>>> call GlusterVolume.list succeeded in 0.10 seconds (__init__:515)<br>
>>> 2017-02-09 15:16:25,940 INFO (Reactor thread)<br>
>>> [ProtocolDetector.<wbr>AcceptorImpl] Accepted connection from<br>
>>> ::ffff:<a href="http://127.0.0.1:55062" rel="noreferrer" target="_blank">127.0.0.1:55062</a> (protocoldetector:72)<br>
>>> 2017-02-09 15:16:25,946 INFO (Reactor thread)<br>
>>> [ProtocolDetector.Detector] Detected protocol stomp from<br>
>>> ::ffff:<a href="http://127.0.0.1:55062" rel="noreferrer" target="_blank">127.0.0.1:55062</a> (protocoldetector:127)<br>
>>> 2017-02-09 15:16:25,947 INFO (Reactor thread) [Broker.StompAdapter]<br>
>>> Processing CONNECT request (stompreactor:102)<br>
>>> 2017-02-09 15:16:25,947 INFO (JsonRpc (StompReactor))<br>
>>> [Broker.StompAdapter] Subscribe command received (stompreactor:129)<br>
>>> 2017-02-09 15:16:26,058 ERROR (jsonrpc/1) [storage.TaskManager.Task]<br>
>>> (Task='02cad901-5fe8-4f2d-<wbr>895b-14184f67feab') Unexpected error (task:870)<br>
>>> Traceback (most recent call last):<br>
>>> File "/usr/share/vdsm/storage/task.<wbr>py", line 877, in _run<br>
>>> return fn(*args, **kargs)<br>
>>> File "/usr/lib/python2.7/site-<wbr>packages/vdsm/logUtils.py", line 50, in<br>
>>> wrapper<br>
>>> res = f(*args, **kwargs)<br>
>>> File "/usr/share/vdsm/storage/hsm.<wbr>py", line 812, in<br>
>>> forcedDetachStorageDomain<br>
>>> self._<wbr>deatchStorageDomainFromOldPool<wbr>s(sdUUID)<br>
>>> File "/usr/share/vdsm/storage/hsm.<wbr>py", line 790, in<br>
>>> _<wbr>deatchStorageDomainFromOldPool<wbr>s<br>
>>> dom.acquireClusterLock(host_<wbr>id)<br>
>>> File "/usr/share/vdsm/storage/sd.<wbr>py", line 810, in acquireClusterLock<br>
>>> self._manifest.<wbr>acquireDomainLock(hostID)<br>
>>> File "/usr/share/vdsm/storage/sd.<wbr>py", line 499, in acquireDomainLock<br>
>>> self._domainLock.acquire(<wbr>hostID, self.getDomainLease())<br>
>>> File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/<wbr>clusterlock.py",<br>
>>> line 362, in acquire<br>
>>> "Cannot acquire %s" % (lease,), str(e))<br>
>>> AcquireLockFailure: Cannot obtain lock:<br>
>>> u"id=60455567-ad30-42e3-a9df-<wbr>62fe86c7fd25, rc=5, out=Cannot acquire<br>
>>> Lease(name='SDM',<br>
>>> path=u'/rhev/data-center/mnt/<wbr>glusterSD/localhost:data2/<wbr>60455567-ad30-42e3-a9df-<wbr>62fe86c7fd25/dom_md/leases',<br>
>>> offset=1048576), err=(5, 'Sanlock resource not acquired', 'Input/output<br>
>>> error')"<br>
>>> 2017-02-09 15:16:26,058 INFO (jsonrpc/1) [storage.TaskManager.Task]<br>
>>> (Task='02cad901-5fe8-4f2d-<wbr>895b-14184f67feab') aborting: Task is aborted:<br>
>>> 'Cannot obtain lock' - code 651 (task:1175)<br>
>>> 2017-02-09 15:16:26,059 ERROR (jsonrpc/1) [storage.Dispatcher] {'status':<br>
>>> {'message': 'Cannot obtain lock: u"id=60455567-ad30-42e3-a9df-<wbr>62fe86c7fd25,<br>
>>> rc=5, out=Cannot acquire Lease(name=\'SDM\',<br>
>>> path=u\'/rhev/data-center/mnt/<wbr>glusterSD/localhost:data2/<wbr>60455567-ad30-42e3-a9df-<wbr>62fe86c7fd25/dom_md/leases\',<br>
>>> offset=1048576), err=(5, \'Sanlock resource not acquired\', \'Input/output<br>
>>> error\')"', 'code': 651}} (dispatcher:77)<br>
>>> 2017-02-09 15:16:26,059 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC<br>
>>> call StorageDomain.detach failed (error 651) in 23.04 seconds (__init__:515)<br>
>>><br>
>>> --<br>
>>> Doug<br>
>>><br>
>>> ______________________________<wbr>_________________<br>
>>> Users mailing list<br>
>>> <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
>>> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
>>><br>
>><br>
><br>
><br>
><br>
> --<br>
> Doug<br>
><br>
> ______________________________<wbr>_________________<br>
> Users mailing list<br>
> <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Doug</div>
</div></div></div>