<div dir="auto"><div>Hi Nir,<br><div class="gmail_extra"><br><div class="gmail_quote">On 16 Feb 2017 22:41, "Nir Soffer" <<a href="mailto:nsoffer@redhat.com">nsoffer@redhat.com</a>> wrote:<br type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="quoted-text">On Fri, Feb 17, 2017 at 3:16 AM, Doug Ingham <<a href="mailto:dougti@gmail.com">dougti@gmail.com</a>> wrote:<br>
> Well that didn't go so well. I deleted both dom_md/ids & dom_md/leases in<br>
> the cloned volume, and I still can't import the storage domain.<br>
<br>
</div>You cannot delete files from dom_md, this will invalidate the storage<br>
domain and you<br>
will not be able to use it without restoring these files.<br>
<br>
The leases file on file storage domain is unused, so creating empty<br>
file is enough.<br>
<br>
The ids file must be created and initialized using sanlock, please you<br>
should find<br>
instructions how to do it in the archives.<br>
<div class="quoted-text"><br>
> The snapshot was also taken some 4 hours before the attempted import, so I'm<br>
> surprised the locks haven't expired by themselves...<br>
<br>
</div>Leases do not expire if vdsm is connected to storage, and sanlock can access<br>
the storage.<br>
<br>
I'm not sure what do you mean by volume snapshots.<br></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">A snapshot is like a save-point, the state of a storage volume from a specific point in time.</div><div dir="auto"><br></div><div dir="auto">In this case, it means I have created a copy/clone of my active data volume. It's a completely new, separate volume, and is not attached to any running services.</div><div dir="auto"><br></div><div dir="auto">I'm using this copy/clone to test the import process, before doing it with my live volume. If I "break" something in the cloned volume, no worries, I can just delete it and recreate it from the snapshot.</div><div dir="auto"><br></div><div dir="auto">Hope that clears things up a bit!</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
To import a storage domain, you should first make that no other setup is using<br>
the storage domain.<br>
<br>
The best way to do it is to detach the storage domain from the other setup.<br>
<br>
If you are using hosted engine, you must also put the hosted engine agent in<br>
global maintenance mode.<br>
<br>
If your engine is broken, the best way to disconnect from storage is to reboot<br>
the hosts.<br></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">So this is the issue. My current tests emulate exactly this, however I'm still not able to import the domain into the new Engine. When I try to do so, I get the resulting logs I copied in my earlier email.</div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><font color="#888888"></font></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">Doug</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><font color="#888888"><br>
Nir</font></blockquote></div></div></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><font color="#888888">
</font><div class="elided-text"><br>
><br>
><br>
> 2017-02-16 21:58:24,630-03 INFO<br>
> [org.ovirt.engine.core.bll.<wbr>storage.connection.<wbr>AddStorageServerConnectionComm<wbr>and]<br>
> (default task-45) [d59bc8c0-3c53-4a34-9d7c-<wbr>8c982ee14e14] Lock Acquired to<br>
> object<br>
> 'EngineLock:{exclusiveLocks='[<wbr>localhost:data-teste2=<<wbr>STORAGE_CONNECTION,<br>
> ACTION_TYPE_FAILED_OBJECT_<wbr>LOCKED>]', sharedLocks='null'}'<br>
> 2017-02-16 21:58:24,645-03 INFO<br>
> [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>ConnectStorageServerVDSCommand<wbr>]<br>
> (default task-45) [d59bc8c0-3c53-4a34-9d7c-<wbr>8c982ee14e14] START,<br>
> ConnectStorageServerVDSCommand<wbr>(HostName = <a href="http://v5.dc0.example.com" rel="noreferrer" target="_blank">v5.dc0.example.com</a>,<br>
> StorageServerConnectionManagem<wbr>entVDSParameters:{runAsync='<wbr>true',<br>
> hostId='1a3f10f2-e4ce-44b9-<wbr>9495-06e445cfa0b0',<br>
> storagePoolId='00000000-0000-<wbr>0000-0000-000000000000',<br>
> storageType='GLUSTERFS',<br>
> connectionList='[<wbr>StorageServerConnections:{id='<wbr>null',<br>
> connection='localhost:data-<wbr>teste2', iqn='null', vfsType='glusterfs',<br>
> mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null',<br>
> iface='null', netIfaceName='null'}]'}), log id: 726df65e<br>
> 2017-02-16 21:58:26,046-03 INFO<br>
> [org.ovirt.engine.core.bll.<wbr>storage.connection.<wbr>AddStorageServerConnectionComm<wbr>and]<br>
> (default task-45) [d59bc8c0-3c53-4a34-9d7c-<wbr>8c982ee14e14] Lock freed to<br>
> object 'EngineLock:{exclusiveLocks='[<wbr>localhost:data<br>
> teste2=<STORAGE_CONNECTION, ACTION_TYPE_FAILED_OBJECT_<wbr>LOCKED>]',<br>
> sharedLocks='null'}'<br>
> 2017-02-16 21:58:26,206-03 INFO<br>
> [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>HSMGetStorageDomainsListVDSCom<wbr>mand]<br>
> (default task-52) [85548427-713f-4ffb-a385-<wbr>a97a7ee4109d] START,<br>
> HSMGetStorageDomainsListVDSCom<wbr>mand(HostName = <a href="http://v5.dc0.example.com" rel="noreferrer" target="_blank">v5.dc0.example.com</a>,<br>
> HSMGetStorageDomainsListVDSCom<wbr>mandParameters:{runAsync='<wbr>true',<br>
> hostId='1a3f10f2-e4ce-44b9-<wbr>9495-06e445cfa0b0',<br>
> storagePoolId='00000000-0000-<wbr>0000-0000-000000000000', storageType='null',<br>
> storageDomainType='Data', path='localhost:data-teste2'})<wbr>, log id: 79f6cc88<br>
> 2017-02-16 21:58:27,899-03 INFO<br>
> [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>HSMGetStorageDomainsListVDSCom<wbr>mand]<br>
> (default task-50) [38e87311-a7a5-49a8-bf18-<wbr>857dd969cd5f] START,<br>
> HSMGetStorageDomainsListVDSCom<wbr>mand(HostName = <a href="http://v5.dc0.example.com" rel="noreferrer" target="_blank">v5.dc0.example.com</a>,<br>
> HSMGetStorageDomainsListVDSCom<wbr>mandParameters:{runAsync='<wbr>true',<br>
> hostId='1a3f10f2-e4ce-44b9-<wbr>9495-06e445cfa0b0',<br>
> storagePoolId='00000000-0000-<wbr>0000-0000-000000000000', storageType='null',<br>
> storageDomainType='Data', path='localhost:data-teste2'})<wbr>, log id: 7280d13<br>
> 2017-02-16 21:58:29,156-03 INFO<br>
> [org.ovirt.engine.core.bll.<wbr>storage.connection.<wbr>RemoveStorageServerConnectionC<wbr>ommand]<br>
> (default task-56) [1b3826e4-4890-43d4-8854-<wbr>16f3c573a31f] Lock Acquired to<br>
> object<br>
> 'EngineLock:{exclusiveLocks='[<wbr>localhost:data-teste2=<<wbr>STORAGE_CONNECTION,<br>
> ACTION_TYPE_FAILED_OBJECT_<wbr>LOCKED>,<br>
> 5e5f6610-c759-448b-a53d-<wbr>9a456f513681=<STORAGE_<wbr>CONNECTION,<br>
> ACTION_TYPE_FAILED_OBJECT_<wbr>LOCKED>]', sharedLocks='null'}'<br>
> 2017-02-16 21:58:29,168-03 INFO<br>
> [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>DisconnectStorageServerVDSComm<wbr>and]<br>
> (default task-57) [5e4b20cf-60d2-4ae9-951b-<wbr>c2693603aa6f] START,<br>
> DisconnectStorageServerVDSComm<wbr>and(HostName = <a href="http://v5.dc0.example.com" rel="noreferrer" target="_blank">v5.dc0.example.com</a>,<br>
> StorageServerConnectionManagem<wbr>entVDSParameters:{runAsync='<wbr>true',<br>
> hostId='1a3f10f2-e4ce-44b9-<wbr>9495-06e445cfa0b0',<br>
> storagePoolId='00000000-0000-<wbr>0000-0000-000000000000',<br>
> storageType='GLUSTERFS',<br>
> connectionList='[<wbr>StorageServerConnections:{id='<wbr>5e5f6610-c759-448b-a53d-<wbr>9a456f513681',<br>
> connection='localhost:data-<wbr>teste2', iqn='null', vfsType='glusterfs',<br>
> mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null',<br>
> iface='null', netIfaceName='null'}]'}), log id: 6042b108<br>
> 2017-02-16 21:58:29,193-03 INFO<br>
> [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>DisconnectStorageServerVDSComm<wbr>and]<br>
> (default task-56) [1b3826e4-4890-43d4-8854-<wbr>16f3c573a31f] START,<br>
> DisconnectStorageServerVDSComm<wbr>and(HostName = <a href="http://v5.dc0.example.com" rel="noreferrer" target="_blank">v5.dc0.example.com</a>,<br>
> StorageServerConnectionManagem<wbr>entVDSParameters:{runAsync='<wbr>true',<br>
> hostId='1a3f10f2-e4ce-44b9-<wbr>9495-06e445cfa0b0',<br>
> storagePoolId='00000000-0000-<wbr>0000-0000-000000000000',<br>
> storageType='GLUSTERFS',<br>
> connectionList='[<wbr>StorageServerConnections:{id='<wbr>5e5f6610-c759-448b-a53d-<wbr>9a456f513681',<br>
> connection='localhost:data-<wbr>teste2', iqn='null', vfsType='glusterfs',<br>
> mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null',<br>
> iface='null', netIfaceName='null'}]'}), log id: 4e9421cf<br>
> 2017-02-16 21:58:31,398-03 INFO<br>
> [org.ovirt.engine.core.bll.<wbr>storage.connection.<wbr>RemoveStorageServerConnectionC<wbr>ommand]<br>
> (default task-56) [1b3826e4-4890-43d4-8854-<wbr>16f3c573a31f] Lock freed to<br>
> object<br>
> 'EngineLock:{exclusiveLocks='[<wbr>localhost:data-teste2=<<wbr>STORAGE_CONNECTION,<br>
> ACTION_TYPE_FAILED_OBJECT_<wbr>LOCKED>,<br>
> 5e5f6610-c759-448b-a53d-<wbr>9a456f513681=<STORAGE_<wbr>CONNECTION,<br>
> ACTION_TYPE_FAILED_OBJECT_<wbr>LOCKED>]', sharedLocks='null'}'<br>
><br>
> Again, many thanks!<br>
> Doug<br>
><br>
> On 16 February 2017 at 18:53, Doug Ingham <<a href="mailto:dougti@gmail.com">dougti@gmail.com</a>> wrote:<br>
>><br>
>> Hi Nir,<br>
>><br>
>> On 16 February 2017 at 13:55, Nir Soffer <<a href="mailto:nsoffer@redhat.com">nsoffer@redhat.com</a>> wrote:<br>
>>><br>
>>> On Mon, Feb 13, 2017 at 3:35 PM, Doug Ingham <<a href="mailto:dougti@gmail.com">dougti@gmail.com</a>> wrote:<br>
>>> > Hi Sahina,<br>
>>> ><br>
>>> > On 13 February 2017 at 05:45, Sahina Bose <<a href="mailto:sabose@redhat.com">sabose@redhat.com</a>> wrote:<br>
>>> >><br>
>>> >> Any errors in the gluster mount logs for this gluster volume?<br>
>>> >><br>
>>> >> How about "gluster vol heal <volname> info" - does it list any entries<br>
>>> >> to<br>
>>> >> heal?<br>
>>> ><br>
>>> ><br>
>>> > After more investigating, I found out that there is a sanlock daemon<br>
>>> > that<br>
>>> > runs with VDSM, independently of the HE, so I'd basically have to bring<br>
>>> > the<br>
>>> > volume down & wait for the leases to expire/delete them* before I can<br>
>>> > import<br>
>>> > the domain.<br>
>>> ><br>
>>> > *I understand removing /dom_md/leases/ should do the job?<br>
>>><br>
>>> No, the issue is probably dom_md/ids accessed by sanlock, but removing<br>
>>> files<br>
>>> accessed by sanlock will not help, an open file will remain open until<br>
>>> sanlock<br>
>>> close the file.<br>
>><br>
>><br>
>> I'm testing this with volume snapshots at the moment, so there are no<br>
>> processes accessing the new volume.<br>
>><br>
>><br>
>>> Did you try to reboot the host before installing it again? If you did and<br>
>>> you<br>
>>> still have these issues, you probably need to remove the previous<br>
>>> installation<br>
>>> properly before installing again.<br>
>>><br>
>>> Adding Simone to help with uninstalling and reinstalling hosted engine.<br>
>><br>
>><br>
>> The Hosted-Engine database had been corrupted and the restore wasn't<br>
>> running correctly, so I installed a new engine on a new server - no restores<br>
>> or old data. The aim is to import the old storage domain into the new Engine<br>
>> & then import the VMs into the new storage domain.<br>
>> My only problem with this is that there appear to be some file based<br>
>> leases somewhere that, unless I manage to locate & delete them, force me to<br>
>> wait for the leases to timeout before I can import the old storage domain.<br>
>> To minimise downtime, I'm trying to avoid having to wait for the leases to<br>
>> timeout.<br>
>><br>
>> Regards,<br>
>> Doug<br>
>><br>
>>><br>
>>><br>
>>> Nir<br>
>>><br>
>>> ><br>
>>> ><br>
>>> >><br>
>>> >><br>
>>> >> On Thu, Feb 9, 2017 at 11:57 PM, Doug Ingham <<a href="mailto:dougti@gmail.com">dougti@gmail.com</a>> wrote:<br>
>>> >>><br>
>>> >>> Some interesting output from the vdsm log...<br>
>>> >>><br>
>>> >>><br>
>>> >>> 2017-02-09 15:16:24,051 INFO (jsonrpc/1) [storage.StorageDomain]<br>
>>> >>> Resource namespace 01_img_60455567-ad30-42e3-<wbr>a9df-62fe86c7fd25<br>
>>> >>> already<br>
>>> >>> registered (sd:731)<br>
>>> >>> 2017-02-09 15:16:24,051 INFO (jsonrpc/1) [storage.StorageDomain]<br>
>>> >>> Resource namespace 02_vol_60455567-ad30-42e3-<wbr>a9df-62fe86c7fd25<br>
>>> >>> already<br>
>>> >>> registered (sd:740)<br>
>>> >>> 2017-02-09 15:16:24,052 INFO (jsonrpc/1) [storage.SANLock] Acquiring<br>
>>> >>> Lease(name='SDM',<br>
>>> >>><br>
>>> >>> path=u'/rhev/data-center/mnt/<wbr>glusterSD/localhost:data2/<wbr>60455567-ad30-42e3-a9df-6<br>
>>> >>> 2fe86c7fd25/dom_md/leases', offset=1048576) for host id 1<br>
>>> >>> (clusterlock:343)<br>
>>> >>> 2017-02-09 15:16:24,057 INFO (jsonrpc/1) [storage.SANLock] Releasing<br>
>>> >>> host id for domain 60455567-ad30-42e3-a9df-<wbr>62fe86c7fd25 (id: 1)<br>
>>> >>> (clusterlock:305)<br>
>>> >>> 2017-02-09 15:16:25,149 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC<br>
>>> >>> call GlusterHost.list succeeded in 0.17 seconds (__init__:515)<br>
>>> >>> 2017-02-09 15:16:25,264 INFO (Reactor thread)<br>
>>> >>> [ProtocolDetector.<wbr>AcceptorImpl] Accepted connection from<br>
>>> >>> ::ffff:<a href="http://127.0.0.1:55060" rel="noreferrer" target="_blank">127.0.0.1:55060</a> (protocoldetector:72)<br>
>>> >>> 2017-02-09 15:16:25,270 INFO (Reactor thread)<br>
>>> >>> [ProtocolDetector.Detector] Detected protocol stomp from<br>
>>> >>> ::ffff:<a href="http://127.0.0.1:55060" rel="noreferrer" target="_blank">127.0.0.1:55060</a> (protocoldetector:127)<br>
>>> >>> 2017-02-09 15:16:25,271 INFO (Reactor thread) [Broker.StompAdapter]<br>
>>> >>> Processing CONNECT request (stompreactor:102)<br>
>>> >>> 2017-02-09 15:16:25,271 INFO (JsonRpc (StompReactor))<br>
>>> >>> [Broker.StompAdapter] Subscribe command received (stompreactor:129)<br>
>>> >>> 2017-02-09 15:16:25,416 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC<br>
>>> >>> call Host.getHardwareInfo succeeded in 0.01 seconds (__init__:515)<br>
>>> >>> 2017-02-09 15:16:25,419 INFO (jsonrpc/6) [dispatcher] Run and<br>
>>> >>> protect:<br>
>>> >>> repoStats(options=None) (logUtils:49)<br>
>>> >>> 2017-02-09 15:16:25,419 INFO (jsonrpc/6) [dispatcher] Run and<br>
>>> >>> protect:<br>
>>> >>> repoStats, Return response: {u'e8d04da7-ad3d-4227-a45d-<wbr>b5a29b2f43e5':<br>
>>> >>> {'code': 0, 'actual': True<br>
>>> >>> , 'version': 4, 'acquired': True, 'delay': '0.000854128',<br>
>>> >>> 'lastCheck':<br>
>>> >>> '5.1', 'valid': True}, u'a77b8821-ff19-4d17-a3ce-<wbr>a6c3a69436d5':<br>
>>> >>> {'code': 0,<br>
>>> >>> 'actual': True, 'vers<br>
>>> >>> ion': 4, 'acquired': True, 'delay': '0.000966556', 'lastCheck':<br>
>>> >>> '2.6',<br>
>>> >>> 'valid': True}} (logUtils:52)<br>
>>> >>> 2017-02-09 15:16:25,447 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC<br>
>>> >>> call Host.getStats succeeded in 0.03 seconds (__init__:515)<br>
>>> >>> 2017-02-09 15:16:25,450 ERROR (JsonRpc (StompReactor))<br>
>>> >>> [vds.dispatcher]<br>
>>> >>> SSL error receiving from <yajsonrpc.betterAsyncore.<wbr>Dispatcher<br>
>>> >>> connected<br>
>>> >>> ('::ffff:127.0.0.1', 55060, 0, 0) at 0x7f69c0043cf8>: unexpected eof<br>
>>> >>> (betterAsyncore:113)<br>
>>> >>> 2017-02-09 15:16:25,812 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC<br>
>>> >>> call GlusterVolume.list succeeded in 0.10 seconds (__init__:515)<br>
>>> >>> 2017-02-09 15:16:25,940 INFO (Reactor thread)<br>
>>> >>> [ProtocolDetector.<wbr>AcceptorImpl] Accepted connection from<br>
>>> >>> ::ffff:<a href="http://127.0.0.1:55062" rel="noreferrer" target="_blank">127.0.0.1:55062</a> (protocoldetector:72)<br>
>>> >>> 2017-02-09 15:16:25,946 INFO (Reactor thread)<br>
>>> >>> [ProtocolDetector.Detector] Detected protocol stomp from<br>
>>> >>> ::ffff:<a href="http://127.0.0.1:55062" rel="noreferrer" target="_blank">127.0.0.1:55062</a> (protocoldetector:127)<br>
>>> >>> 2017-02-09 15:16:25,947 INFO (Reactor thread) [Broker.StompAdapter]<br>
>>> >>> Processing CONNECT request (stompreactor:102)<br>
>>> >>> 2017-02-09 15:16:25,947 INFO (JsonRpc (StompReactor))<br>
>>> >>> [Broker.StompAdapter] Subscribe command received (stompreactor:129)<br>
>>> >>> 2017-02-09 15:16:26,058 ERROR (jsonrpc/1) [storage.TaskManager.Task]<br>
>>> >>> (Task='02cad901-5fe8-4f2d-<wbr>895b-14184f67feab') Unexpected error<br>
>>> >>> (task:870)<br>
>>> >>> Traceback (most recent call last):<br>
>>> >>> File "/usr/share/vdsm/storage/task.<wbr>py", line 877, in _run<br>
>>> >>> return fn(*args, **kargs)<br>
>>> >>> File "/usr/lib/python2.7/site-<wbr>packages/vdsm/logUtils.py", line 50,<br>
>>> >>> in<br>
>>> >>> wrapper<br>
>>> >>> res = f(*args, **kwargs)<br>
>>> >>> File "/usr/share/vdsm/storage/hsm.<wbr>py", line 812, in<br>
>>> >>> forcedDetachStorageDomain<br>
>>> >>> self._<wbr>deatchStorageDomainFromOldPool<wbr>s(sdUUID)<br>
>>> >>> File "/usr/share/vdsm/storage/hsm.<wbr>py", line 790, in<br>
>>> >>> _<wbr>deatchStorageDomainFromOldPool<wbr>s<br>
>>> >>> dom.acquireClusterLock(host_<wbr>id)<br>
>>> >>> File "/usr/share/vdsm/storage/sd.<wbr>py", line 810, in<br>
>>> >>> acquireClusterLock<br>
>>> >>> self._manifest.<wbr>acquireDomainLock(hostID)<br>
>>> >>> File "/usr/share/vdsm/storage/sd.<wbr>py", line 499, in<br>
>>> >>> acquireDomainLock<br>
>>> >>> self._domainLock.acquire(<wbr>hostID, self.getDomainLease())<br>
>>> >>> File<br>
>>> >>> "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/<wbr>clusterlock.py",<br>
>>> >>> line 362, in acquire<br>
>>> >>> "Cannot acquire %s" % (lease,), str(e))<br>
>>> >>> AcquireLockFailure: Cannot obtain lock:<br>
>>> >>> u"id=60455567-ad30-42e3-a9df-<wbr>62fe86c7fd25, rc=5, out=Cannot acquire<br>
>>> >>> Lease(name='SDM',<br>
>>> >>><br>
>>> >>> path=u'/rhev/data-center/mnt/<wbr>glusterSD/localhost:data2/<wbr>60455567-ad30-42e3-a9df-<wbr>62fe86c7fd25/dom_md/leases',<br>
>>> >>> offset=1048576), err=(5, 'Sanlock resource not acquired',<br>
>>> >>> 'Input/output<br>
>>> >>> error')"<br>
>>> >>> 2017-02-09 15:16:26,058 INFO (jsonrpc/1) [storage.TaskManager.Task]<br>
>>> >>> (Task='02cad901-5fe8-4f2d-<wbr>895b-14184f67feab') aborting: Task is<br>
>>> >>> aborted:<br>
>>> >>> 'Cannot obtain lock' - code 651 (task:1175)<br>
>>> >>> 2017-02-09 15:16:26,059 ERROR (jsonrpc/1) [storage.Dispatcher]<br>
>>> >>> {'status':<br>
>>> >>> {'message': 'Cannot obtain lock:<br>
>>> >>> u"id=60455567-ad30-42e3-a9df-<wbr>62fe86c7fd25,<br>
>>> >>> rc=5, out=Cannot acquire Lease(name=\'SDM\',<br>
>>> >>><br>
>>> >>> path=u\'/rhev/data-center/mnt/<wbr>glusterSD/localhost:data2/<wbr>60455567-ad30-42e3-a9df-<wbr>62fe86c7fd25/dom_md/leases\',<br>
>>> >>> offset=1048576), err=(5, \'Sanlock resource not acquired\',<br>
>>> >>> \'Input/output<br>
>>> >>> error\')"', 'code': 651}} (dispatcher:77)<br>
>>> >>> 2017-02-09 15:16:26,059 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC<br>
>>> >>> call StorageDomain.detach failed (error 651) in 23.04 seconds<br>
>>> >>> (__init__:515)<br>
>>> >>><br>
>>> >>> --<br>
>>> >>> Doug<br>
>>> >>><br>
>>> >>> ______________________________<wbr>_________________<br>
>>> >>> Users mailing list<br>
>>> >>> <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
>>> >>> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
>>> >>><br>
>>> >><br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> > --<br>
>>> > Doug<br>
>>> ><br>
>>> > ______________________________<wbr>_________________<br>
>>> > Users mailing list<br>
>>> > <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
>>> > <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
>>> ><br>
>><br>
>><br>
>><br>
>><br>
>> --<br>
>> Doug<br>
><br>
><br>
><br>
><br>
> --<br>
> Doug<br>
</div></blockquote></div><br></div></div></div>