<div dir="ltr"><div>Any errors in the gluster mount logs for this gluster volume?<br><br></div>How about "gluster vol heal <volname> info" - does it list any entries to heal?<br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 9, 2017 at 11:57 PM, Doug Ingham <span dir="ltr"><<a href="mailto:dougti@gmail.com" target="_blank">dougti@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Some interesting output from the vdsm log...<br><br><br>2017-02-09 15:16:24,051 INFO (jsonrpc/1) [storage.StorageDomain] Resource namespace 01_img_60455567-ad30-42e3-<wbr>a9df-62fe86c7fd25 already registered (sd:731)<br>2017-02-09 15:16:24,051 INFO (jsonrpc/1) [storage.StorageDomain] Resource namespace 02_vol_60455567-ad30-42e3-<wbr>a9df-62fe86c7fd25 already registered (sd:740)<br>2017-02-09 15:16:24,052 INFO (jsonrpc/1) [storage.SANLock] Acquiring Lease(name='SDM', path=u'/rhev/data-center/mnt/<wbr>glusterSD/localhost:data2/<wbr>60455567-ad30-42e3-a9df-6<br>2fe86c7fd25/dom_md/leases', offset=1048576) for host id 1 (clusterlock:343)<br>2017-02-09 15:16:24,057 INFO (jsonrpc/1) [storage.SANLock] Releasing host id for domain 60455567-ad30-42e3-a9df-<wbr>62fe86c7fd25 (id: 1) (clusterlock:305)<br>2017-02-09 15:16:25,149 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call GlusterHost.list succeeded in 0.17 seconds (__init__:515)<br>2017-02-09 15:16:25,264 INFO (Reactor thread) [ProtocolDetector.<wbr>AcceptorImpl] Accepted connection from ::ffff:<a href="http://127.0.0.1:55060" target="_blank">127.0.0.1:55060</a> (protocoldetector:72)<br>2017-02-09 15:16:25,270 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::ffff:<a href="http://127.0.0.1:55060" target="_blank">127.0.0.1:55060</a> (protocoldetector:127)<br>2017-02-09 15:16:25,271 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompreactor:102)<br>2017-02-09 15:16:25,271 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompreactor:129)<br>2017-02-09 15:16:25,416 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getHardwareInfo succeeded in 0.01 seconds (__init__:515)<br>2017-02-09 15:16:25,419 INFO (jsonrpc/6) [dispatcher] Run and protect: repoStats(options=None) (logUtils:49)<br>2017-02-09 15:16:25,419 INFO (jsonrpc/6) [dispatcher] Run and protect: repoStats, Return response: {u'e8d04da7-ad3d-4227-a45d-<wbr>b5a29b2f43e5': {'code': 0, 'actual': True<br>, 'version': 4, 'acquired': True, 'delay': '0.000854128', 'lastCheck': '5.1', 'valid': True}, u'a77b8821-ff19-4d17-a3ce-<wbr>a6c3a69436d5': {'code': 0, 'actual': True, 'vers<br>ion': 4, 'acquired': True, 'delay': '0.000966556', 'lastCheck': '2.6', 'valid': True}} (logUtils:52)<br>2017-02-09 15:16:25,447 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:515)<br>2017-02-09 15:16:25,450 ERROR (JsonRpc (StompReactor)) [vds.dispatcher] SSL error receiving from <yajsonrpc.betterAsyncore.<wbr>Dispatcher connected ('::ffff:127.0.0.1', 55060, 0, 0) at 0x7f69c0043cf8>: unexpected eof (betterAsyncore:113)<br>2017-02-09 15:16:25,812 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call GlusterVolume.list succeeded in 0.10 seconds (__init__:515)<br>2017-02-09 15:16:25,940 INFO (Reactor thread) [ProtocolDetector.<wbr>AcceptorImpl] Accepted connection from ::ffff:<a href="http://127.0.0.1:55062" target="_blank">127.0.0.1:55062</a> (protocoldetector:72)<br>2017-02-09 15:16:25,946 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::ffff:<a href="http://127.0.0.1:55062" target="_blank">127.0.0.1:55062</a> (protocoldetector:127)<br>2017-02-09 15:16:25,947 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompreactor:102)<br>2017-02-09 15:16:25,947 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompreactor:129)<br>2017-02-09 15:16:26,058 ERROR (jsonrpc/1) [storage.TaskManager.Task] (Task='02cad901-5fe8-4f2d-<wbr>895b-14184f67feab') Unexpected error (task:870)<br>Traceback (most recent call last):<br> File "/usr/share/vdsm/storage/task.<wbr>py", line 877, in _run<br> return fn(*args, **kargs)<br> File "/usr/lib/python2.7/site-<wbr>packages/vdsm/logUtils.py", line 50, in wrapper<br> res = f(*args, **kwargs)<br> File "/usr/share/vdsm/storage/hsm.<wbr>py", line 812, in forcedDetachStorageDomain<br> self._<wbr>deatchStorageDomainFromOldPool<wbr>s(sdUUID)<br> File "/usr/share/vdsm/storage/hsm.<wbr>py", line 790, in _<wbr>deatchStorageDomainFromOldPool<wbr>s<br> dom.acquireClusterLock(host_<wbr>id)<br> File "/usr/share/vdsm/storage/sd.<wbr>py", line 810, in acquireClusterLock<br> self._manifest.<wbr>acquireDomainLock(hostID)<br> File "/usr/share/vdsm/storage/sd.<wbr>py", line 499, in acquireDomainLock<br> self._domainLock.acquire(<wbr>hostID, self.getDomainLease())<br> File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/<wbr>clusterlock.py", line 362, in acquire<br> "Cannot acquire %s" % (lease,), str(e))<br>AcquireLockFailure: Cannot obtain lock: u"id=60455567-ad30-42e3-a9df-<wbr>62fe86c7fd25, rc=5, out=Cannot acquire Lease(name='SDM', path=u'/rhev/data-center/mnt/<wbr>glusterSD/localhost:data2/<wbr>60455567-ad30-42e3-a9df-<wbr>62fe86c7fd25/dom_md/leases', offset=1048576), err=(5, 'Sanlock resource not acquired', 'Input/output error')"<br>2017-02-09 15:16:26,058 INFO (jsonrpc/1) [storage.TaskManager.Task] (Task='02cad901-5fe8-4f2d-<wbr>895b-14184f67feab') aborting: Task is aborted: 'Cannot obtain lock' - code 651 (task:1175)<br clear="all"><div>2017-02-09 15:16:26,059 ERROR (jsonrpc/1) [storage.Dispatcher] {'status': {'message': 'Cannot obtain lock: u"id=60455567-ad30-42e3-a9df-<wbr>62fe86c7fd25, rc=5, out=Cannot acquire Lease(name=\'SDM\', path=u\'/rhev/data-center/mnt/<wbr>glusterSD/localhost:data2/<wbr>60455567-ad30-42e3-a9df-<wbr>62fe86c7fd25/dom_md/leases\', offset=1048576), err=(5, \'Sanlock resource not acquired\', \'Input/output error\')"', 'code': 651}} (dispatcher:77)<br>2017-02-09 15:16:26,059 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StorageDomain.detach failed (error 651) in 23.04 seconds (__init__:515)<span class="HOEnZb"><font color="#888888"><br><br>-- <br><div class="m_-792972286646668071gmail_signature">Doug</div>
</font></span></div></div>
<br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
<br></blockquote></div><br></div>