<div dir="ltr"><div><div>Well that didn't go so well. I deleted both dom_md/ids & dom_md/leases in the cloned volume, and I still can't import the storage domain.<br></div>The snapshot was also taken some 4 hours before the attempted import, so I'm surprised the locks haven't expired by themselves...<br><br></div><div><div><div><br>2017-02-16 21:58:24,630-03 INFO [org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default task-45) [d59bc8c0-3c53-4a34-9d7c-8c982ee14e14] Lock Acquired to object 'EngineLock:{exclusiveLocks='[localhost:data-teste2=<STORAGE_CONNECTION, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'<br>2017-02-16 21:58:24,645-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-45) [d59bc8c0-3c53-4a34-9d7c-8c982ee14e14] START, ConnectStorageServerVDSCommand(HostName = <a href="http://v5.dc0.example.com">v5.dc0.example.com</a>, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='1a3f10f2-e4ce-44b9-9495-06e445cfa0b0', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='null', connection='localhost:data-teste2', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 726df65e<br>2017-02-16 21:58:26,046-03 INFO [org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default task-45) [d59bc8c0-3c53-4a34-9d7c-8c982ee14e14] Lock freed to object 'EngineLock:{exclusiveLocks='[localhost:data teste2=<STORAGE_CONNECTION, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'<br>2017-02-16 21:58:26,206-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (default task-52) [85548427-713f-4ffb-a385-a97a7ee4109d] START, HSMGetStorageDomainsListVDSCommand(HostName = <a href="http://v5.dc0.example.com">v5.dc0.example.com</a>, HSMGetStorageDomainsListVDSCommandParameters:{runAsync='true', hostId='1a3f10f2-e4ce-44b9-9495-06e445cfa0b0', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='null', storageDomainType='Data', path='localhost:data-teste2'}), log id: 79f6cc88<br>2017-02-16 21:58:27,899-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (default task-50) [38e87311-a7a5-49a8-bf18-857dd969cd5f] START, HSMGetStorageDomainsListVDSCommand(HostName = <a href="http://v5.dc0.example.com">v5.dc0.example.com</a>, HSMGetStorageDomainsListVDSCommandParameters:{runAsync='true', hostId='1a3f10f2-e4ce-44b9-9495-06e445cfa0b0', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='null', storageDomainType='Data', path='localhost:data-teste2'}), log id: 7280d13<br>2017-02-16 21:58:29,156-03 INFO [org.ovirt.engine.core.bll.storage.connection.RemoveStorageServerConnectionCommand] (default task-56) [1b3826e4-4890-43d4-8854-16f3c573a31f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[localhost:data-teste2=<STORAGE_CONNECTION, ACTION_TYPE_FAILED_OBJECT_LOCKED>, 5e5f6610-c759-448b-a53d-9a456f513681=<STORAGE_CONNECTION, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'<br>2017-02-16 21:58:29,168-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (default task-57) [5e4b20cf-60d2-4ae9-951b-c2693603aa6f] START, DisconnectStorageServerVDSCommand(HostName = <a href="http://v5.dc0.example.com">v5.dc0.example.com</a>, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='1a3f10f2-e4ce-44b9-9495-06e445cfa0b0', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='5e5f6610-c759-448b-a53d-9a456f513681', connection='localhost:data-teste2', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6042b108<br>2017-02-16 21:58:29,193-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (default task-56) [1b3826e4-4890-43d4-8854-16f3c573a31f] START, DisconnectStorageServerVDSCommand(HostName = <a href="http://v5.dc0.example.com">v5.dc0.example.com</a>, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='1a3f10f2-e4ce-44b9-9495-06e445cfa0b0', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='5e5f6610-c759-448b-a53d-9a456f513681', connection='localhost:data-teste2', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4e9421cf<br>2017-02-16 21:58:31,398-03 INFO [org.ovirt.engine.core.bll.storage.connection.RemoveStorageServerConnectionCommand] (default task-56) [1b3826e4-4890-43d4-8854-16f3c573a31f] Lock freed to object 'EngineLock:{exclusiveLocks='[localhost:data-teste2=<STORAGE_CONNECTION, ACTION_TYPE_FAILED_OBJECT_LOCKED>, 5e5f6610-c759-448b-a53d-9a456f513681=<STORAGE_CONNECTION, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'<br><br></div><div>Again, many thanks!<br></div><div> Doug<br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 16 February 2017 at 18:53, Doug Ingham <span dir="ltr"><<a href="mailto:dougti@gmail.com" target="_blank">dougti@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Nir,<br><div><div class="gmail_extra"><br><div class="gmail_quote"><span>On 16 February 2017 at 13:55, Nir Soffer <span dir="ltr"><<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span>On Mon, Feb 13, 2017 at 3:35 PM, Doug Ingham <<a href="mailto:dougti@gmail.com" target="_blank">dougti@gmail.com</a>> wrote:<br>
> Hi Sahina,<br>
><br>
> On 13 February 2017 at 05:45, Sahina Bose <<a href="mailto:sabose@redhat.com" target="_blank">sabose@redhat.com</a>> wrote:<br>
>><br>
>> Any errors in the gluster mount logs for this gluster volume?<br>
>><br>
>> How about "gluster vol heal <volname> info" - does it list any entries to<br>
>> heal?<br>
><br>
><br>
> After more investigating, I found out that there is a sanlock daemon that<br>
> runs with VDSM, independently of the HE, so I'd basically have to bring the<br>
> volume down & wait for the leases to expire/delete them* before I can import<br>
> the domain.<br>
><br>
> *I understand removing /dom_md/leases/ should do the job?<br>
<br>
</span>No, the issue is probably dom_md/ids accessed by sanlock, but removing files<br>
accessed by sanlock will not help, an open file will remain open until sanlock<br>
close the file.<br></blockquote><div><br></div></span><div>I'm testing this with volume snapshots at the moment, so there are no processes accessing the new volume.<br></div><span><div> <br><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Did you try to reboot the host before installing it again? If you did and you<br>
still have these issues, you probably need to remove the previous installation<br>
properly before installing again.<br>
<br>
Adding Simone to help with uninstalling and reinstalling hosted engine.<br></blockquote><div><br></div></span><div>The Hosted-Engine database had been corrupted and the restore wasn't running correctly, so I installed a new engine on a new server - no restores or old data. The aim is to import the old storage domain into the new Engine & then import the VMs into the new storage domain.<br></div><div>My only problem with this is that there appear to be some file based leases somewhere that, unless I manage to locate & delete them, force me to wait for the leases to timeout before I can import the old storage domain.<br></div><div>To minimise downtime, I'm trying to avoid having to wait for the leases to timeout.<br><br></div><div>Regards,<br></div><div> Doug<br></div><div><div class="gmail-m_-5388494886097110460h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="gmail-m_-5388494886097110460m_6367117638838566862HOEnZb"><font color="#888888"><br>
Nir<br>
</font></span><div class="gmail-m_-5388494886097110460m_6367117638838566862HOEnZb"><div class="gmail-m_-5388494886097110460m_6367117638838566862h5"><br>
><br>
><br>
>><br>
>><br>
>> On Thu, Feb 9, 2017 at 11:57 PM, Doug Ingham <<a href="mailto:dougti@gmail.com" target="_blank">dougti@gmail.com</a>> wrote:<br>
>>><br>
>>> Some interesting output from the vdsm log...<br>
>>><br>
>>><br>
>>> 2017-02-09 15:16:24,051 INFO (jsonrpc/1) [storage.StorageDomain]<br>
>>> Resource namespace 01_img_60455567-ad30-42e3-a9df<wbr>-62fe86c7fd25 already<br>
>>> registered (sd:731)<br>
>>> 2017-02-09 15:16:24,051 INFO (jsonrpc/1) [storage.StorageDomain]<br>
>>> Resource namespace 02_vol_60455567-ad30-42e3-a9df<wbr>-62fe86c7fd25 already<br>
>>> registered (sd:740)<br>
>>> 2017-02-09 15:16:24,052 INFO (jsonrpc/1) [storage.SANLock] Acquiring<br>
>>> Lease(name='SDM',<br>
>>> path=u'/rhev/data-center/mnt/g<wbr>lusterSD/localhost:data2/60455<wbr>567-ad30-42e3-a9df-6<br>
>>> 2fe86c7fd25/dom_md/leases', offset=1048576) for host id 1<br>
>>> (clusterlock:343)<br>
>>> 2017-02-09 15:16:24,057 INFO (jsonrpc/1) [storage.SANLock] Releasing<br>
>>> host id for domain 60455567-ad30-42e3-a9df-62fe86<wbr>c7fd25 (id: 1)<br>
>>> (clusterlock:305)<br>
>>> 2017-02-09 15:16:25,149 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC<br>
>>> call GlusterHost.list succeeded in 0.17 seconds (__init__:515)<br>
>>> 2017-02-09 15:16:25,264 INFO (Reactor thread)<br>
>>> [ProtocolDetector.AcceptorImpl<wbr>] Accepted connection from<br>
>>> ::ffff:<a href="http://127.0.0.1:55060" rel="noreferrer" target="_blank">127.0.0.1:55060</a> (protocoldetector:72)<br>
>>> 2017-02-09 15:16:25,270 INFO (Reactor thread)<br>
>>> [ProtocolDetector.Detector] Detected protocol stomp from<br>
>>> ::ffff:<a href="http://127.0.0.1:55060" rel="noreferrer" target="_blank">127.0.0.1:55060</a> (protocoldetector:127)<br>
>>> 2017-02-09 15:16:25,271 INFO (Reactor thread) [Broker.StompAdapter]<br>
>>> Processing CONNECT request (stompreactor:102)<br>
>>> 2017-02-09 15:16:25,271 INFO (JsonRpc (StompReactor))<br>
>>> [Broker.StompAdapter] Subscribe command received (stompreactor:129)<br>
>>> 2017-02-09 15:16:25,416 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC<br>
>>> call Host.getHardwareInfo succeeded in 0.01 seconds (__init__:515)<br>
>>> 2017-02-09 15:16:25,419 INFO (jsonrpc/6) [dispatcher] Run and protect:<br>
>>> repoStats(options=None) (logUtils:49)<br>
>>> 2017-02-09 15:16:25,419 INFO (jsonrpc/6) [dispatcher] Run and protect:<br>
>>> repoStats, Return response: {u'e8d04da7-ad3d-4227-a45d-b5a<wbr>29b2f43e5':<br>
>>> {'code': 0, 'actual': True<br>
>>> , 'version': 4, 'acquired': True, 'delay': '0.000854128', 'lastCheck':<br>
>>> '5.1', 'valid': True}, u'a77b8821-ff19-4d17-a3ce-a6c3<wbr>a69436d5': {'code': 0,<br>
>>> 'actual': True, 'vers<br>
>>> ion': 4, 'acquired': True, 'delay': '0.000966556', 'lastCheck': '2.6',<br>
>>> 'valid': True}} (logUtils:52)<br>
>>> 2017-02-09 15:16:25,447 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC<br>
>>> call Host.getStats succeeded in 0.03 seconds (__init__:515)<br>
>>> 2017-02-09 15:16:25,450 ERROR (JsonRpc (StompReactor)) [vds.dispatcher]<br>
>>> SSL error receiving from <yajsonrpc.betterAsyncore.Disp<wbr>atcher connected<br>
>>> ('::ffff:127.0.0.1', 55060, 0, 0) at 0x7f69c0043cf8>: unexpected eof<br>
>>> (betterAsyncore:113)<br>
>>> 2017-02-09 15:16:25,812 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC<br>
>>> call GlusterVolume.list succeeded in 0.10 seconds (__init__:515)<br>
>>> 2017-02-09 15:16:25,940 INFO (Reactor thread)<br>
>>> [ProtocolDetector.AcceptorImpl<wbr>] Accepted connection from<br>
>>> ::ffff:<a href="http://127.0.0.1:55062" rel="noreferrer" target="_blank">127.0.0.1:55062</a> (protocoldetector:72)<br>
>>> 2017-02-09 15:16:25,946 INFO (Reactor thread)<br>
>>> [ProtocolDetector.Detector] Detected protocol stomp from<br>
>>> ::ffff:<a href="http://127.0.0.1:55062" rel="noreferrer" target="_blank">127.0.0.1:55062</a> (protocoldetector:127)<br>
>>> 2017-02-09 15:16:25,947 INFO (Reactor thread) [Broker.StompAdapter]<br>
>>> Processing CONNECT request (stompreactor:102)<br>
>>> 2017-02-09 15:16:25,947 INFO (JsonRpc (StompReactor))<br>
>>> [Broker.StompAdapter] Subscribe command received (stompreactor:129)<br>
>>> 2017-02-09 15:16:26,058 ERROR (jsonrpc/1) [storage.TaskManager.Task]<br>
>>> (Task='02cad901-5fe8-4f2d-895b<wbr>-14184f67feab') Unexpected error (task:870)<br>
>>> Traceback (most recent call last):<br>
>>> File "/usr/share/vdsm/storage/task.<wbr>py", line 877, in _run<br>
>>> return fn(*args, **kargs)<br>
>>> File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/logUtils.py", line 50, in<br>
>>> wrapper<br>
>>> res = f(*args, **kwargs)<br>
>>> File "/usr/share/vdsm/storage/hsm.p<wbr>y", line 812, in<br>
>>> forcedDetachStorageDomain<br>
>>> self._deatchStorageDomainFrom<wbr>OldPools(sdUUID)<br>
>>> File "/usr/share/vdsm/storage/hsm.p<wbr>y", line 790, in<br>
>>> _deatchStorageDomainFromOldPoo<wbr>ls<br>
>>> dom.acquireClusterLock(host_i<wbr>d)<br>
>>> File "/usr/share/vdsm/storage/sd.py<wbr>", line 810, in acquireClusterLock<br>
>>> self._manifest.acquireDomainL<wbr>ock(hostID)<br>
>>> File "/usr/share/vdsm/storage/sd.py<wbr>", line 499, in acquireDomainLock<br>
>>> self._domainLock.acquire(host<wbr>ID, self.getDomainLease())<br>
>>> File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/storage/clusterlock.p<wbr>y",<br>
>>> line 362, in acquire<br>
>>> "Cannot acquire %s" % (lease,), str(e))<br>
>>> AcquireLockFailure: Cannot obtain lock:<br>
>>> u"id=60455567-ad30-42e3-a9df-6<wbr>2fe86c7fd25, rc=5, out=Cannot acquire<br>
>>> Lease(name='SDM',<br>
>>> path=u'/rhev/data-center/mnt/g<wbr>lusterSD/localhost:data2/60455<wbr>567-ad30-42e3-a9df-62fe86c7fd2<wbr>5/dom_md/leases',<br>
>>> offset=1048576), err=(5, 'Sanlock resource not acquired', 'Input/output<br>
>>> error')"<br>
>>> 2017-02-09 15:16:26,058 INFO (jsonrpc/1) [storage.TaskManager.Task]<br>
>>> (Task='02cad901-5fe8-4f2d-895b<wbr>-14184f67feab') aborting: Task is aborted:<br>
>>> 'Cannot obtain lock' - code 651 (task:1175)<br>
>>> 2017-02-09 15:16:26,059 ERROR (jsonrpc/1) [storage.Dispatcher] {'status':<br>
>>> {'message': 'Cannot obtain lock: u"id=60455567-ad30-42e3-a9df-6<wbr>2fe86c7fd25,<br>
>>> rc=5, out=Cannot acquire Lease(name=\'SDM\',<br>
>>> path=u\'/rhev/data-center/mnt/<wbr>glusterSD/localhost:data2/6045<wbr>5567-ad30-42e3-a9df-62fe86c7fd<wbr>25/dom_md/leases\',<br>
>>> offset=1048576), err=(5, \'Sanlock resource not acquired\', \'Input/output<br>
>>> error\')"', 'code': 651}} (dispatcher:77)<br>
>>> 2017-02-09 15:16:26,059 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC<br>
>>> call StorageDomain.detach failed (error 651) in 23.04 seconds (__init__:515)<br>
>>><br>
>>> --<br>
>>> Doug<br>
>>><br>
>>> ______________________________<wbr>_________________<br>
>>> Users mailing list<br>
>>> <a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
>>> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
>>><br>
>><br>
><br>
><br>
><br>
> --<br>
> Doug<br>
><br>
> ______________________________<wbr>_________________<br>
> Users mailing list<br>
> <a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
><br>
</div></div></blockquote></div></div></div><span class="gmail-m_-5388494886097110460HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div class="gmail-m_-5388494886097110460m_6367117638838566862gmail_signature">Doug</div>
</font></span></div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail-m_-5388494886097110460gmail_signature">Doug</div>
</div></div>