<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">I have reinstalled my ovirt installation using the nightly builds so that I can try out non-admin REST API access to ovirt. After installing the engine, connecting to my directory system, creating a domain, and adding a host (all successfully), I tried to add my first storage domain (NFS).<div><br></div><div>While creating the storage domain, I get an error at the end along with a couple of events that say:</div><div><br></div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div>"Failed to attach Storage Domains to Data Center DCC. (User: admin@internal)"</div></blockquote><div><br></div><div>followed by:</div><div><br></div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div>"Failed to attach Storage Domain DCVMStorage to Data Center DCC. (User: admin@internal)"</div></blockquote><div><br></div><div>I see the following in the engine.log file:</div><div><br></div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div>2012-10-22 20:17:57,617 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--127.0.0.1-8009-7) [7d1ffd97] Weird return value: Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc</div></div><div><div>mCode 661</div></div><div><div>mMessage Cannot acquire host id: ('b97019e9-bd43-46d8-afd0-421d6768271b', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))</div></div><div><div><br></div></div><div><div>2012-10-22 20:17:57,619 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--127.0.0.1-8009-7) [7d1ffd97] Weird return value: Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc</div></div><div><div>mCode 661</div></div><div><div>mMessage Cannot acquire host id: ('b97019e9-bd43-46d8-afd0-421d6768271b', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))</div></div><div><div><br></div></div><div><div>2012-10-22 20:17:57,620 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--127.0.0.1-8009-7) [7d1ffd97] Failed in CreateStoragePoolVDS method</div></div><div><div>2012-10-22 20:17:57,620 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--127.0.0.1-8009-7) [7d1ffd97] Error code unexpected and error message VDSGenericException: VDSErrorException: Failed to CreateStoragePoolVDS, error = Cannot acquire host id: ('b97019e9-bd43-46d8-afd0-421d6768271b', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))</div></div></blockquote><div><br></div><div>On the host where it tried to install from, I see the following in the vdsm.log:</div><div><br></div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div>Thread-243::INFO::2012-10-22 20:17:56,624::safelease::156::SANLock::(acquireHostId) Acquiring host id for domain b97019e9-bd43-46d8-afd0-421d6768271b (id: 250)</div></div><div><div>Thread-243::ERROR::2012-10-22 20:17:57,628::task::853::TaskManager.Task::(_setError) Task=`1ead54dc-407c-4d0b-96f4-8dc56c74d4cf`::Unexpected error</div></div><div><div>Traceback (most recent call last):</div></div><div><div> File "/usr/share/vdsm/storage/task.py", line 861, in _run</div></div><div><div> return fn(*args, **kargs)</div></div><div><div> File "/usr/share/vdsm/logUtils.py", line 38, in wrapper</div></div><div><div> res = f(*args, **kwargs)</div></div><div><div> File "/usr/share/vdsm/storage/hsm.py", line 790, in createStoragePool</div></div><div><div> return sp.StoragePool(spUUID, self.taskMng).create(poolName, masterDom, domList, masterV</div></div><div><div>ersion, safeLease)</div></div><div><div> File "/usr/share/vdsm/storage/sp.py", line 567, in create</div></div><div><div> self._acquireTemporaryClusterLock(msdUUID, safeLease)</div></div><div><div> File "/usr/share/vdsm/storage/sp.py", line 508, in _acquireTemporaryClusterLock</div></div><div><div> msd.acquireHostId(self.id)</div></div><div><div> File "/usr/share/vdsm/storage/sd.py", line 407, in acquireHostId</div></div><div><div> self._clusterLock.acquireHostId(hostId)</div></div><div><div> File "/usr/share/vdsm/storage/safelease.py", line 162, in acquireHostId</div></div><div><div> raise se.AcquireHostIdFailure(self._sdUUID, e)</div></div><div><div>AcquireHostIdFailure: Cannot acquire host id: ('b97019e9-bd43-46d8-afd0-421d6768271b', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))</div></div></blockquote><div><br></div><div>After I get this error, I logged into the host and see that the nfs mount is present:</div><div><br></div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div>eos.dcc.mobi:/home/vmstorage on /rhev/data-center/mnt/eos.dcc.mobi:_home_vmstorage type nfs4 (rw,relatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,port=0,timeo=600,retrans=6,sec=sys,clientaddr=10.1.1.12,minorversion=0,local_lock=none,addr=10.1.1.11)</div></div></blockquote><div><br></div><div>And when I look at the directory, I see the following:</div><div><br></div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div>[root@mech ~]# ls -laR /rhev/data-center/mnt/eos.dcc.mobi:_home_vmstorage</div></div><div><div>/rhev/data-center/mnt/eos.dcc.mobi:_home_vmstorage:</div></div><div><div>total 12</div></div><div><div>drwxr-xr-x. 3 vdsm kvm 4096 Oct 22 20:17 .</div></div><div><div>drwxr-xr-x. 6 vdsm kvm 4096 Oct 22 20:17 ..</div></div><div><div>drwxr-xr-x. 4 vdsm kvm 4096 Oct 22 20:17 b97019e9-bd43-46d8-afd0-421d6768271b</div></div><div><div><br></div></div><div><div>/rhev/data-center/mnt/eos.dcc.mobi:_home_vmstorage/b97019e9-bd43-46d8-afd0-421d6768271b:</div></div><div><div>total 16</div></div><div><div>drwxr-xr-x. 4 vdsm kvm 4096 Oct 22 20:17 .</div></div><div><div>drwxr-xr-x. 3 vdsm kvm 4096 Oct 22 20:17 ..</div></div><div><div>drwxr-xr-x. 2 vdsm kvm 4096 Oct 22 20:17 dom_md</div></div><div><div>drwxr-xr-x. 2 vdsm kvm 4096 Oct 22 20:17 images</div></div><div><div><br></div></div><div><div>/rhev/data-center/mnt/eos.dcc.mobi:_home_vmstorage/b97019e9-bd43-46d8-afd0-421d6768271b/dom_md:</div></div><div><div>total 2060</div></div><div><div>drwxr-xr-x. 2 vdsm kvm 4096 Oct 22 20:17 .</div></div><div><div>drwxr-xr-x. 4 vdsm kvm 4096 Oct 22 20:17 ..</div></div><div><div>-rw-rw----. 1 vdsm kvm 1048576 Oct 22 20:17 ids</div></div><div><div>-rw-rw----. 1 vdsm kvm 0 Oct 22 20:17 inbox</div></div><div><div>-rw-rw----. 1 vdsm kvm 1048576 Oct 22 20:17 leases</div></div><div><div>-rw-r--r--. 1 vdsm kvm 308 Oct 22 20:17 metadata</div></div><div><div>-rw-rw----. 1 vdsm kvm 0 Oct 22 20:17 outbox</div></div><div><div><br></div></div><div><div>/rhev/data-center/mnt/eos.dcc.mobi:_home_vmstorage/b97019e9-bd43-46d8-afd0-421d6768271b/images:</div></div><div><div>total 8</div></div><div><div>drwxr-xr-x. 2 vdsm kvm 4096 Oct 22 20:17 .</div></div><div><div>drwxr-xr-x. 4 vdsm kvm 4096 Oct 22 20:17 ..</div></div></blockquote><div><br></div><div>It looks like it was able to mount the directory and create a bunch of files and directories owned by vdsm:kvm.</div><div><br></div><div>So after all this, I was stuck with a Storage domain that wasn't assigned to my data center. When I tried to attach it to my Data Center, I got another error:</div><div><br></div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div>"Failed to attach Storage Domains to Data Center dcc. (User: admin@internal)"</div></blockquote><div><br></div><div>And I saw this in engine.log:</div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div>2012-10-22 21:30:53,788 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) [4eaa9670] Failed in CreateStoragePoolVDS method</div></div><div><div>2012-10-22 21:30:53,789 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) [4eaa9670] Error code unexpected and error message VDSGenericException: VDSErrorException: Failed to CreateStoragePoolVDS, error = Cannot acquire host id: ('b97019e9-bd43-46d8-afd0-421d6768271b', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))</div></div><div><div>2012-10-22 21:30:53,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) [4eaa9670] Command org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand return value</div></div><div><div> Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc</div></div><div><div>mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc</div></div><div><div>mCode 661</div></div><div><div>mMessage Cannot acquire host id: ('b97019e9-bd43-46d8-afd0-421d6768271b', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))</div></div><div><div><br></div></div><div><div><br></div></div><div><div>2012-10-22 21:30:53,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) [4eaa9670] Vds: mechis3</div></div><div><div>2012-10-22 21:30:53,792 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) [4eaa9670] Command CreateStoragePoolVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStoragePoolVDS, error = Cannot acquire host id: ('b97019e9-bd43-46d8-afd0-421d6768271b', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))</div></div><div><div>2012-10-22 21:30:53,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (pool-3-thread-50) [4eaa9670] FINISH, CreateStoragePoolVDSCommand, log id: 4015ca0d</div></div></blockquote><div><br></div><div>This all looks familiar - as does the vdsm.log file (not repeated).</div><div><br></div><div>Now, my system is in a different state. It now shows that the storage domain is associated with my Data Center (if I click on the data center in the ui and look at the storage tab below, I see that the nfs storage domain is listed with this data center. I also see that it reports its status in the data center as "locked". I don't see any way to "unlock" it, although I suspect that if I did, I'd get the same error as above (SanlockException).</div><div><br></div><div>If I try to destroy/delete the storage domain, I get an error that says that I can't destroy the master storage domain.</div><div><br></div><div>So how do I get out of this mess?</div><div><br></div><div>As to versions, I see the following ovirt packages when I dump the ovirt version info for my ovirt-engine system:</div><div><br></div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div>ovirt-engine.noarch 3.1.0-3.1345126685.git7649eed.fc17</div></div><div><div> @ovirt-nightly</div></div><div><div>ovirt-engine-backend.noarch 3.1.0-3.1345126685.git7649eed.fc17</div></div><div><div> @ovirt-nightly</div></div><div><div>ovirt-engine-cli.noarch 3.2.0.5-1.20121015.git4189352.fc17</div></div><div><div> @ovirt-nightly</div></div><div><div>ovirt-engine-config.noarch 3.1.0-3.1345126685.git7649eed.fc17</div></div><div><div> @ovirt-nightly</div></div><div><div>ovirt-engine-dbscripts.noarch 3.1.0-3.1345126685.git7649eed.fc17</div></div><div><div> @ovirt-nightly</div></div><div><div>ovirt-engine-genericapi.noarch 3.1.0-3.1345126685.git7649eed.fc17</div></div><div><div> @ovirt-nightly</div></div><div><div>ovirt-engine-notification-service.noarch</div></div><div><div> 3.1.0-3.1345126685.git7649eed.fc17</div></div><div><div> @ovirt-nightly</div></div><div><div>ovirt-engine-restapi.noarch 3.1.0-3.1345126685.git7649eed.fc17</div></div><div><div> @ovirt-nightly</div></div><div><div>ovirt-engine-sdk.noarch 3.2.0.2-1.20120927.git663b765.fc17</div></div><div><div> @ovirt-nightly</div></div><div><div>ovirt-engine-setup.noarch 3.1.0-3.1345126685.git7649eed.fc17</div></div><div><div> @ovirt-nightly</div></div><div><div>ovirt-engine-tools-common.noarch 3.1.0-3.1345126685.git7649eed.fc17</div></div><div><div> @ovirt-nightly</div></div><div><div>ovirt-engine-userportal.noarch 3.1.0-3.1345126685.git7649eed.fc17</div></div><div><div> @ovirt-nightly</div></div><div><div>ovirt-engine-webadmin-portal.noarch 3.1.0-3.1345126685.git7649eed.fc17</div></div><div><div> @ovirt-nightly</div></div><div><div>ovirt-image-uploader.noarch 3.1.0-0.git9c42c8.fc17 @ovirt-stable</div></div><div><div>ovirt-iso-uploader.noarch 3.1.0-0.git1841d9.fc17 @ovirt-stable</div></div><div><div>ovirt-log-collector.noarch 3.1.0-0.git10d719.fc17 @ovirt-stable</div></div><div><div>ovirt-release-fedora.noarch 4-2 @/ovirt-release-fedora.noarch</div></div></blockquote><div><br></div><div>This is a few of the packages on my vm host:</div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div>libvirt.x86_64 0.9.11.5-3.fc17 @updates</div></div><div><div><div>libvirt-client.x86_64 0.9.11.5-3.fc17 @updates</div><div>libvirt-daemon.x86_64 0.9.11.5-3.fc17 @updates</div><div>libvirt-daemon-config-network.x86_64 0.9.11.5-3.fc17 @updates</div><div>libvirt-daemon-config-nwfilter.x86_64 0.9.11.5-3.fc17 @updates</div><div>libvirt-lock-sanlock.x86_64 0.9.11.5-3.fc17 @updates</div><div>libvirt-python.x86_64 0.9.11.5-3.fc17 @updates</div></div><div><div>sanlock.x86_64 2.4-2.fc17 @updates</div><div>sanlock-lib.x86_64 2.4-2.fc17 @updates</div><div>sanlock-python.x86_64 2.4-2.fc17 @updates</div></div></div><div><div><div>vdsm.x86_64 4.10.0-10.fc17 @updates</div></div><div><div>vdsm-cli.noarch 4.10.0-10.fc17 @updates</div></div><div><div>vdsm-python.x86_64 4.10.0-10.fc17 @updates</div></div><div><div>vdsm-xmlrpc.noarch 4.10.0-10.fc17 @updates</div></div><div><div></div></div></div></blockquote><div><div><br></div></div></body></html>