[Users] Error creating the first storage domain (NFS)
Brian Vetter
bjvetter at gmail.com
Tue Oct 23 03:06:06 UTC 2012
I have reinstalled my ovirt installation using the nightly builds so that I can try out non-admin REST API access to ovirt. After installing the engine, connecting to my directory system, creating a domain, and adding a host (all successfully), I tried to add my first storage domain (NFS).
While creating the storage domain, I get an error at the end along with a couple of events that say:
"Failed to attach Storage Domains to Data Center DCC. (User: admin at internal)"
followed by:
"Failed to attach Storage Domain DCVMStorage to Data Center DCC. (User: admin at internal)"
I see the following in the engine.log file:
2012-10-22 20:17:57,617 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--127.0.0.1-8009-7) [7d1ffd97] Weird return value: Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc
mCode 661
mMessage Cannot acquire host id: ('b97019e9-bd43-46d8-afd0-421d6768271b', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))
2012-10-22 20:17:57,619 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--127.0.0.1-8009-7) [7d1ffd97] Weird return value: Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc
mCode 661
mMessage Cannot acquire host id: ('b97019e9-bd43-46d8-afd0-421d6768271b', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))
2012-10-22 20:17:57,620 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--127.0.0.1-8009-7) [7d1ffd97] Failed in CreateStoragePoolVDS method
2012-10-22 20:17:57,620 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--127.0.0.1-8009-7) [7d1ffd97] Error code unexpected and error message VDSGenericException: VDSErrorException: Failed to CreateStoragePoolVDS, error = Cannot acquire host id: ('b97019e9-bd43-46d8-afd0-421d6768271b', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))
On the host where it tried to install from, I see the following in the vdsm.log:
Thread-243::INFO::2012-10-22 20:17:56,624::safelease::156::SANLock::(acquireHostId) Acquiring host id for domain b97019e9-bd43-46d8-afd0-421d6768271b (id: 250)
Thread-243::ERROR::2012-10-22 20:17:57,628::task::853::TaskManager.Task::(_setError) Task=`1ead54dc-407c-4d0b-96f4-8dc56c74d4cf`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 861, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 38, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 790, in createStoragePool
return sp.StoragePool(spUUID, self.taskMng).create(poolName, masterDom, domList, masterV
ersion, safeLease)
File "/usr/share/vdsm/storage/sp.py", line 567, in create
self._acquireTemporaryClusterLock(msdUUID, safeLease)
File "/usr/share/vdsm/storage/sp.py", line 508, in _acquireTemporaryClusterLock
msd.acquireHostId(self.id)
File "/usr/share/vdsm/storage/sd.py", line 407, in acquireHostId
self._clusterLock.acquireHostId(hostId)
File "/usr/share/vdsm/storage/safelease.py", line 162, in acquireHostId
raise se.AcquireHostIdFailure(self._sdUUID, e)
AcquireHostIdFailure: Cannot acquire host id: ('b97019e9-bd43-46d8-afd0-421d6768271b', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))
After I get this error, I logged into the host and see that the nfs mount is present:
eos.dcc.mobi:/home/vmstorage on /rhev/data-center/mnt/eos.dcc.mobi:_home_vmstorage type nfs4 (rw,relatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,port=0,timeo=600,retrans=6,sec=sys,clientaddr=10.1.1.12,minorversion=0,local_lock=none,addr=10.1.1.11)
And when I look at the directory, I see the following:
[root at mech ~]# ls -laR /rhev/data-center/mnt/eos.dcc.mobi:_home_vmstorage
/rhev/data-center/mnt/eos.dcc.mobi:_home_vmstorage:
total 12
drwxr-xr-x. 3 vdsm kvm 4096 Oct 22 20:17 .
drwxr-xr-x. 6 vdsm kvm 4096 Oct 22 20:17 ..
drwxr-xr-x. 4 vdsm kvm 4096 Oct 22 20:17 b97019e9-bd43-46d8-afd0-421d6768271b
/rhev/data-center/mnt/eos.dcc.mobi:_home_vmstorage/b97019e9-bd43-46d8-afd0-421d6768271b:
total 16
drwxr-xr-x. 4 vdsm kvm 4096 Oct 22 20:17 .
drwxr-xr-x. 3 vdsm kvm 4096 Oct 22 20:17 ..
drwxr-xr-x. 2 vdsm kvm 4096 Oct 22 20:17 dom_md
drwxr-xr-x. 2 vdsm kvm 4096 Oct 22 20:17 images
/rhev/data-center/mnt/eos.dcc.mobi:_home_vmstorage/b97019e9-bd43-46d8-afd0-421d6768271b/dom_md:
total 2060
drwxr-xr-x. 2 vdsm kvm 4096 Oct 22 20:17 .
drwxr-xr-x. 4 vdsm kvm 4096 Oct 22 20:17 ..
-rw-rw----. 1 vdsm kvm 1048576 Oct 22 20:17 ids
-rw-rw----. 1 vdsm kvm 0 Oct 22 20:17 inbox
-rw-rw----. 1 vdsm kvm 1048576 Oct 22 20:17 leases
-rw-r--r--. 1 vdsm kvm 308 Oct 22 20:17 metadata
-rw-rw----. 1 vdsm kvm 0 Oct 22 20:17 outbox
/rhev/data-center/mnt/eos.dcc.mobi:_home_vmstorage/b97019e9-bd43-46d8-afd0-421d6768271b/images:
total 8
drwxr-xr-x. 2 vdsm kvm 4096 Oct 22 20:17 .
drwxr-xr-x. 4 vdsm kvm 4096 Oct 22 20:17 ..
It looks like it was able to mount the directory and create a bunch of files and directories owned by vdsm:kvm.
So after all this, I was stuck with a Storage domain that wasn't assigned to my data center. When I tried to attach it to my Data Center, I got another error:
"Failed to attach Storage Domains to Data Center dcc. (User: admin at internal)"
And I saw this in engine.log:
2012-10-22 21:30:53,788 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) [4eaa9670] Failed in CreateStoragePoolVDS method
2012-10-22 21:30:53,789 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) [4eaa9670] Error code unexpected and error message VDSGenericException: VDSErrorException: Failed to CreateStoragePoolVDS, error = Cannot acquire host id: ('b97019e9-bd43-46d8-afd0-421d6768271b', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))
2012-10-22 21:30:53,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) [4eaa9670] Command org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand return value
Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc
mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc
mCode 661
mMessage Cannot acquire host id: ('b97019e9-bd43-46d8-afd0-421d6768271b', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))
2012-10-22 21:30:53,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) [4eaa9670] Vds: mechis3
2012-10-22 21:30:53,792 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) [4eaa9670] Command CreateStoragePoolVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStoragePoolVDS, error = Cannot acquire host id: ('b97019e9-bd43-46d8-afd0-421d6768271b', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))
2012-10-22 21:30:53,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (pool-3-thread-50) [4eaa9670] FINISH, CreateStoragePoolVDSCommand, log id: 4015ca0d
This all looks familiar - as does the vdsm.log file (not repeated).
Now, my system is in a different state. It now shows that the storage domain is associated with my Data Center (if I click on the data center in the ui and look at the storage tab below, I see that the nfs storage domain is listed with this data center. I also see that it reports its status in the data center as "locked". I don't see any way to "unlock" it, although I suspect that if I did, I'd get the same error as above (SanlockException).
If I try to destroy/delete the storage domain, I get an error that says that I can't destroy the master storage domain.
So how do I get out of this mess?
As to versions, I see the following ovirt packages when I dump the ovirt version info for my ovirt-engine system:
ovirt-engine.noarch 3.1.0-3.1345126685.git7649eed.fc17
@ovirt-nightly
ovirt-engine-backend.noarch 3.1.0-3.1345126685.git7649eed.fc17
@ovirt-nightly
ovirt-engine-cli.noarch 3.2.0.5-1.20121015.git4189352.fc17
@ovirt-nightly
ovirt-engine-config.noarch 3.1.0-3.1345126685.git7649eed.fc17
@ovirt-nightly
ovirt-engine-dbscripts.noarch 3.1.0-3.1345126685.git7649eed.fc17
@ovirt-nightly
ovirt-engine-genericapi.noarch 3.1.0-3.1345126685.git7649eed.fc17
@ovirt-nightly
ovirt-engine-notification-service.noarch
3.1.0-3.1345126685.git7649eed.fc17
@ovirt-nightly
ovirt-engine-restapi.noarch 3.1.0-3.1345126685.git7649eed.fc17
@ovirt-nightly
ovirt-engine-sdk.noarch 3.2.0.2-1.20120927.git663b765.fc17
@ovirt-nightly
ovirt-engine-setup.noarch 3.1.0-3.1345126685.git7649eed.fc17
@ovirt-nightly
ovirt-engine-tools-common.noarch 3.1.0-3.1345126685.git7649eed.fc17
@ovirt-nightly
ovirt-engine-userportal.noarch 3.1.0-3.1345126685.git7649eed.fc17
@ovirt-nightly
ovirt-engine-webadmin-portal.noarch 3.1.0-3.1345126685.git7649eed.fc17
@ovirt-nightly
ovirt-image-uploader.noarch 3.1.0-0.git9c42c8.fc17 @ovirt-stable
ovirt-iso-uploader.noarch 3.1.0-0.git1841d9.fc17 @ovirt-stable
ovirt-log-collector.noarch 3.1.0-0.git10d719.fc17 @ovirt-stable
ovirt-release-fedora.noarch 4-2 @/ovirt-release-fedora.noarch
This is a few of the packages on my vm host:
libvirt.x86_64 0.9.11.5-3.fc17 @updates
libvirt-client.x86_64 0.9.11.5-3.fc17 @updates
libvirt-daemon.x86_64 0.9.11.5-3.fc17 @updates
libvirt-daemon-config-network.x86_64 0.9.11.5-3.fc17 @updates
libvirt-daemon-config-nwfilter.x86_64 0.9.11.5-3.fc17 @updates
libvirt-lock-sanlock.x86_64 0.9.11.5-3.fc17 @updates
libvirt-python.x86_64 0.9.11.5-3.fc17 @updates
sanlock.x86_64 2.4-2.fc17 @updates
sanlock-lib.x86_64 2.4-2.fc17 @updates
sanlock-python.x86_64 2.4-2.fc17 @updates
vdsm.x86_64 4.10.0-10.fc17 @updates
vdsm-cli.noarch 4.10.0-10.fc17 @updates
vdsm-python.x86_64 4.10.0-10.fc17 @updates
vdsm-xmlrpc.noarch 4.10.0-10.fc17 @updates
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20121022/8a5a46cb/attachment-0001.html>
More information about the Users
mailing list