On Mon, Feb 13, 2017 at 3:35 PM, Doug Ingham <dougti@gmail.com> wrote:
Hi Sahina,
On 13 February 2017 at 05:45, Sahina Bose <sabose@redhat.com> wrote:
Any errors in the gluster mount logs for this gluster volume?
How about "gluster vol heal <volname> info" - does it list any entries to heal?
After more investigating, I found out that there is a sanlock daemon that runs with VDSM, independently of the HE, so I'd basically have to bring the volume down & wait for the leases to expire/delete them* before I can import the domain.
*I understand removing /dom_md/leases/ should do the job?
No, the issue is probably dom_md/ids accessed by sanlock, but removing files accessed by sanlock will not help, an open file will remain open until sanlock close the file. Did you try to reboot the host before installing it again? If you did and you still have these issues, you probably need to remove the previous installation properly before installing again. Adding Simone to help with uninstalling and reinstalling hosted engine. Nir
On Thu, Feb 9, 2017 at 11:57 PM, Doug Ingham <dougti@gmail.com> wrote:
Some interesting output from the vdsm log...
2017-02-09 15:16:24,051 INFO (jsonrpc/1) [storage.StorageDomain] Resource namespace 01_img_60455567-ad30-42e3-a9df-62fe86c7fd25 already registered (sd:731) 2017-02-09 15:16:24,051 INFO (jsonrpc/1) [storage.StorageDomain] Resource namespace 02_vol_60455567-ad30-42e3-a9df-62fe86c7fd25 already registered (sd:740) 2017-02-09 15:16:24,052 INFO (jsonrpc/1) [storage.SANLock] Acquiring Lease(name='SDM', path=u'/rhev/data-center/mnt/glusterSD/localhost:data2/60455567-ad30-42e3-a9df-6 2fe86c7fd25/dom_md/leases', offset=1048576) for host id 1 (clusterlock:343) 2017-02-09 15:16:24,057 INFO (jsonrpc/1) [storage.SANLock] Releasing host id for domain 60455567-ad30-42e3-a9df-62fe86c7fd25 (id: 1) (clusterlock:305) 2017-02-09 15:16:25,149 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call GlusterHost.list succeeded in 0.17 seconds (__init__:515) 2017-02-09 15:16:25,264 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::ffff:127.0.0.1:55060 (protocoldetector:72) 2017-02-09 15:16:25,270 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::ffff:127.0.0.1:55060 (protocoldetector:127) 2017-02-09 15:16:25,271 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompreactor:102) 2017-02-09 15:16:25,271 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompreactor:129) 2017-02-09 15:16:25,416 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getHardwareInfo succeeded in 0.01 seconds (__init__:515) 2017-02-09 15:16:25,419 INFO (jsonrpc/6) [dispatcher] Run and protect: repoStats(options=None) (logUtils:49) 2017-02-09 15:16:25,419 INFO (jsonrpc/6) [dispatcher] Run and protect: repoStats, Return response: {u'e8d04da7-ad3d-4227-a45d-b5a29b2f43e5': {'code': 0, 'actual': True , 'version': 4, 'acquired': True, 'delay': '0.000854128', 'lastCheck': '5.1', 'valid': True}, u'a77b8821-ff19-4d17-a3ce-a6c3a69436d5': {'code': 0, 'actual': True, 'vers ion': 4, 'acquired': True, 'delay': '0.000966556', 'lastCheck': '2.6', 'valid': True}} (logUtils:52) 2017-02-09 15:16:25,447 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:515) 2017-02-09 15:16:25,450 ERROR (JsonRpc (StompReactor)) [vds.dispatcher] SSL error receiving from <yajsonrpc.betterAsyncore.Dispatcher connected ('::ffff:127.0.0.1', 55060, 0, 0) at 0x7f69c0043cf8>: unexpected eof (betterAsyncore:113) 2017-02-09 15:16:25,812 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call GlusterVolume.list succeeded in 0.10 seconds (__init__:515) 2017-02-09 15:16:25,940 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::ffff:127.0.0.1:55062 (protocoldetector:72) 2017-02-09 15:16:25,946 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::ffff:127.0.0.1:55062 (protocoldetector:127) 2017-02-09 15:16:25,947 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompreactor:102) 2017-02-09 15:16:25,947 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompreactor:129) 2017-02-09 15:16:26,058 ERROR (jsonrpc/1) [storage.TaskManager.Task] (Task='02cad901-5fe8-4f2d-895b-14184f67feab') Unexpected error (task:870) Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 877, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 812, in forcedDetachStorageDomain self._deatchStorageDomainFromOldPools(sdUUID) File "/usr/share/vdsm/storage/hsm.py", line 790, in _deatchStorageDomainFromOldPools dom.acquireClusterLock(host_id) File "/usr/share/vdsm/storage/sd.py", line 810, in acquireClusterLock self._manifest.acquireDomainLock(hostID) File "/usr/share/vdsm/storage/sd.py", line 499, in acquireDomainLock self._domainLock.acquire(hostID, self.getDomainLease()) File "/usr/lib/python2.7/site-packages/vdsm/storage/clusterlock.py", line 362, in acquire "Cannot acquire %s" % (lease,), str(e)) AcquireLockFailure: Cannot obtain lock: u"id=60455567-ad30-42e3-a9df-62fe86c7fd25, rc=5, out=Cannot acquire Lease(name='SDM', path=u'/rhev/data-center/mnt/glusterSD/localhost:data2/60455567-ad30-42e3-a9df-62fe86c7fd25/dom_md/leases', offset=1048576), err=(5, 'Sanlock resource not acquired', 'Input/output error')" 2017-02-09 15:16:26,058 INFO (jsonrpc/1) [storage.TaskManager.Task] (Task='02cad901-5fe8-4f2d-895b-14184f67feab') aborting: Task is aborted: 'Cannot obtain lock' - code 651 (task:1175) 2017-02-09 15:16:26,059 ERROR (jsonrpc/1) [storage.Dispatcher] {'status': {'message': 'Cannot obtain lock: u"id=60455567-ad30-42e3-a9df-62fe86c7fd25, rc=5, out=Cannot acquire Lease(name=\'SDM\', path=u\'/rhev/data-center/mnt/glusterSD/localhost:data2/60455567-ad30-42e3-a9df-62fe86c7fd25/dom_md/leases\', offset=1048576), err=(5, \'Sanlock resource not acquired\', \'Input/output error\')"', 'code': 651}} (dispatcher:77) 2017-02-09 15:16:26,059 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StorageDomain.detach failed (error 651) in 23.04 seconds (__init__:515)
-- Doug
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Doug
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users