<div dir="ltr"><div class="gmail_extra">Hi Sahina,<br><br></div><div class="gmail_extra"><div class="gmail_quote">On 13 February 2017 at 05:45, Sahina Bose <span dir="ltr">&lt;<a href="mailto:sabose@redhat.com" target="_blank">sabose@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Any errors in the gluster mount logs for this gluster volume?<br><br></div>How about &quot;gluster vol heal &lt;volname&gt; info&quot; - does it list any entries to heal?<br></div></blockquote><div><br></div><div>After more investigating, I found out that there is a sanlock daemon that runs with VDSM, independently of the HE, so I&#39;d basically have to bring the volume down &amp; wait for the leases to expire/delete them* before I can import the domain.<br><br></div><div>*I understand removing /dom_md/leases/ should do the job?<br></div><div> <br><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Thu, Feb 9, 2017 at 11:57 PM, Doug Ingham <span dir="ltr">&lt;<a href="mailto:dougti@gmail.com" target="_blank">dougti@gmail.com</a>&gt;</span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr">Some interesting output from the vdsm log...<br><br><br>2017-02-09 15:16:24,051 INFO  (jsonrpc/1) [storage.StorageDomain] Resource namespace 01_img_60455567-ad30-42e3-a9df<wbr>-62fe86c7fd25 already registered (sd:731)<br>2017-02-09 15:16:24,051 INFO  (jsonrpc/1) [storage.StorageDomain] Resource namespace 02_vol_60455567-ad30-42e3-a9df<wbr>-62fe86c7fd25 already registered (sd:740)<br>2017-02-09 15:16:24,052 INFO  (jsonrpc/1) [storage.SANLock] Acquiring Lease(name=&#39;SDM&#39;, path=u&#39;/rhev/data-center/mnt/g<wbr>lusterSD/localhost:data2/60455<wbr>567-ad30-42e3-a9df-6<br>2fe86c7fd25/dom_md/leases&#39;, offset=1048576) for host id 1 (clusterlock:343)<br>2017-02-09 15:16:24,057 INFO  (jsonrpc/1) [storage.SANLock] Releasing host id for domain 60455567-ad30-42e3-a9df-62fe86<wbr>c7fd25 (id: 1) (clusterlock:305)<br>2017-02-09 15:16:25,149 INFO  (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call GlusterHost.list succeeded in 0.17 seconds (__init__:515)<br>2017-02-09 15:16:25,264 INFO  (Reactor thread) [ProtocolDetector.AcceptorImpl<wbr>] Accepted connection from ::ffff:<a href="http://127.0.0.1:55060" target="_blank">127.0.0.1:55060</a> (protocoldetector:72)<br>2017-02-09 15:16:25,270 INFO  (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::ffff:<a href="http://127.0.0.1:55060" target="_blank">127.0.0.1:55060</a> (protocoldetector:127)<br>2017-02-09 15:16:25,271 INFO  (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompreactor:102)<br>2017-02-09 15:16:25,271 INFO  (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompreactor:129)<br>2017-02-09 15:16:25,416 INFO  (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getHardwareInfo succeeded in 0.01 seconds (__init__:515)<br>2017-02-09 15:16:25,419 INFO  (jsonrpc/6) [dispatcher] Run and protect: repoStats(options=None) (logUtils:49)<br>2017-02-09 15:16:25,419 INFO  (jsonrpc/6) [dispatcher] Run and protect: repoStats, Return response: {u&#39;e8d04da7-ad3d-4227-a45d-b5a<wbr>29b2f43e5&#39;: {&#39;code&#39;: 0, &#39;actual&#39;: True<br>, &#39;version&#39;: 4, &#39;acquired&#39;: True, &#39;delay&#39;: &#39;0.000854128&#39;, &#39;lastCheck&#39;: &#39;5.1&#39;, &#39;valid&#39;: True}, u&#39;a77b8821-ff19-4d17-a3ce-a6c3<wbr>a69436d5&#39;: {&#39;code&#39;: 0, &#39;actual&#39;: True, &#39;vers<br>ion&#39;: 4, &#39;acquired&#39;: True, &#39;delay&#39;: &#39;0.000966556&#39;, &#39;lastCheck&#39;: &#39;2.6&#39;, &#39;valid&#39;: True}} (logUtils:52)<br>2017-02-09 15:16:25,447 INFO  (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:515)<br>2017-02-09 15:16:25,450 ERROR (JsonRpc (StompReactor)) [vds.dispatcher] SSL error receiving from &lt;yajsonrpc.betterAsyncore.Disp<wbr>atcher connected (&#39;::ffff:127.0.0.1&#39;, 55060, 0, 0) at 0x7f69c0043cf8&gt;: unexpected eof (betterAsyncore:113)<br>2017-02-09 15:16:25,812 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call GlusterVolume.list succeeded in 0.10 seconds (__init__:515)<br>2017-02-09 15:16:25,940 INFO  (Reactor thread) [ProtocolDetector.AcceptorImpl<wbr>] Accepted connection from ::ffff:<a href="http://127.0.0.1:55062" target="_blank">127.0.0.1:55062</a> (protocoldetector:72)<br>2017-02-09 15:16:25,946 INFO  (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::ffff:<a href="http://127.0.0.1:55062" target="_blank">127.0.0.1:55062</a> (protocoldetector:127)<br>2017-02-09 15:16:25,947 INFO  (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompreactor:102)<br>2017-02-09 15:16:25,947 INFO  (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompreactor:129)<br>2017-02-09 15:16:26,058 ERROR (jsonrpc/1) [storage.TaskManager.Task] (Task=&#39;02cad901-5fe8-4f2d-895b<wbr>-14184f67feab&#39;) Unexpected error (task:870)<br>Traceback (most recent call last):<br>  File &quot;/usr/share/vdsm/storage/task.<wbr>py&quot;, line 877, in _run<br>    return fn(*args, **kargs)<br>  File &quot;/usr/lib/python2.7/site-packa<wbr>ges/vdsm/logUtils.py&quot;, line 50, in wrapper<br>    res = f(*args, **kwargs)<br>  File &quot;/usr/share/vdsm/storage/hsm.p<wbr>y&quot;, line 812, in forcedDetachStorageDomain<br>    self._deatchStorageDomainFromO<wbr>ldPools(sdUUID)<br>  File &quot;/usr/share/vdsm/storage/hsm.p<wbr>y&quot;, line 790, in _deatchStorageDomainFromOldPoo<wbr>ls<br>    dom.acquireClusterLock(host_id<wbr>)<br>  File &quot;/usr/share/vdsm/storage/sd.py<wbr>&quot;, line 810, in acquireClusterLock<br>    self._manifest.acquireDomainLo<wbr>ck(hostID)<br>  File &quot;/usr/share/vdsm/storage/sd.py<wbr>&quot;, line 499, in acquireDomainLock<br>    self._domainLock.acquire(hostI<wbr>D, self.getDomainLease())<br>  File &quot;/usr/lib/python2.7/site-packa<wbr>ges/vdsm/storage/clusterlock.<wbr>py&quot;, line 362, in acquire<br>    &quot;Cannot acquire %s&quot; % (lease,), str(e))<br>AcquireLockFailure: Cannot obtain lock: u&quot;id=60455567-ad30-42e3-a9df-6<wbr>2fe86c7fd25, rc=5, out=Cannot acquire Lease(name=&#39;SDM&#39;, path=u&#39;/rhev/data-center/mnt/g<wbr>lusterSD/localhost:data2/60455<wbr>567-ad30-42e3-a9df-62fe86c7fd2<wbr>5/dom_md/leases&#39;, offset=1048576), err=(5, &#39;Sanlock resource not acquired&#39;, &#39;Input/output error&#39;)&quot;<br>2017-02-09 15:16:26,058 INFO  (jsonrpc/1) [storage.TaskManager.Task] (Task=&#39;02cad901-5fe8-4f2d-895b<wbr>-14184f67feab&#39;) aborting: Task is aborted: &#39;Cannot obtain lock&#39; - code 651 (task:1175)<br clear="all"><div>2017-02-09 15:16:26,059 ERROR (jsonrpc/1) [storage.Dispatcher] {&#39;status&#39;: {&#39;message&#39;: &#39;Cannot obtain lock: u&quot;id=60455567-ad30-42e3-a9df-6<wbr>2fe86c7fd25, rc=5, out=Cannot acquire Lease(name=\&#39;SDM\&#39;, path=u\&#39;/rhev/data-center/mnt/<wbr>glusterSD/localhost:data2/6045<wbr>5567-ad30-42e3-a9df-62fe86c7fd<wbr>25/dom_md/leases\&#39;, offset=1048576), err=(5, \&#39;Sanlock resource not acquired\&#39;, \&#39;Input/output error\&#39;)&quot;&#39;, &#39;code&#39;: 651}} (dispatcher:77)<br>2017-02-09 15:16:26,059 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StorageDomain.detach failed (error 651) in 23.04 seconds (__init__:515)<span class="m_-5121127259329448661HOEnZb"><font color="#888888"><br><br>-- <br><div class="m_-5121127259329448661m_-792972286646668071gmail_signature">Doug</div>
</font></span></div></div>
<br></div></div>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Doug</div>
</div></div>