On 24.01.2013 18:05, Patrick Hurrelmann wrote:

Hi list,

after rebooting one host (single host dc with local storage) the local storage domain can't be attached again. The host was set to maintenance mode and all running vms were shutdown prior the reboot.

Vdsm keeps logging the following errors:

Thread-1266::ERROR::2013-01-24 17:51:46,042::task::853::TaskManager.Task::(_setError) Task=`a0c11f61-8bcf-4f76-9923-43e8b9cc1424`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 861, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 38, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 817, in connectStoragePool return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) File "/usr/share/vdsm/storage/hsm.py", line 859, in _connectStoragePool res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) File "/usr/share/vdsm/storage/sp.py", line 641, in connect self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) File "/usr/share/vdsm/storage/sp.py", line 1109, in __rebuild self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) File "/usr/share/vdsm/storage/sp.py", line 1448, in getMasterDomain raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=c9b86219-0d51-44c3-a7de-e0fe07e2c9e6, msdUUID=00ed91f3-43be-41be-8c05-f3786588a1ad'

and

Thread-1268::ERROR::2013-01-24 17:51:49,073::task::853::TaskManager.Task::(_setError) Task=`95b7f58b-afe0-47bd-9ebd-21d3224f5165`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 861, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 38, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 528, in getSpmStatus pool = self.getPool(spUUID) File "/usr/share/vdsm/storage/hsm.py", line 265, in getPool raise se.StoragePoolUnknown(spUUID) StoragePoolUnknown: Unknown pool id, pool not connected: ('c9b86219-0d51-44c3-a7de-e0fe07e2c9e6',)

while engine logs:

2013-01-24 17:51:46,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-43) [49026692] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc mCode 304 mMessage Cannot find master domain: 'spUUID=c9b86219-0d51-44c3-a7de-e0fe07e2c9e6, msdUUID=00ed91f3-43be-41be-8c05-f3786588a1ad'

Vdsm and engine logs are also attached. I set the affected host back to maintenance. How can I recover from this and attach the storage domain again? If more information is needed, please do not hesitate to request it.

This is on CentOS 6.3 using Dreyou's rpms. Installed versions on host:

vdsm.x86_64 4.10.0-0.44.14.el6 vdsm-cli.noarch 4.10.0-0.44.14.el6 vdsm-python.x86_64 4.10.0-0.44.14.el6 vdsm-xmlrpc.noarch 4.10.0-0.44.14.el6

Engine:

ovirt-engine.noarch 3.1.0-3.19.el6 ovirt-engine-backend.noarch 3.1.0-3.19.el6 ovirt-engine-cli.noarch 3.1.0.7-1.el6 ovirt-engine-config.noarch 3.1.0-3.19.el6 ovirt-engine-dbscripts.noarch 3.1.0-3.19.el6 ovirt-engine-genericapi.noarch 3.1.0-3.19.el6 ovirt-engine-jbossas711.x86_64 1-0 ovirt-engine-notification-service.noarch 3.1.0-3.19.el6 ovirt-engine-restapi.noarch 3.1.0-3.19.el6 ovirt-engine-sdk.noarch 3.1.0.5-1.el6 ovirt-engine-setup.noarch 3.1.0-3.19.el6 ovirt-engine-tools-common.noarch 3.1.0-3.19.el6 ovirt-engine-userportal.noarch 3.1.0-3.19.el6 ovirt-engine-webadmin-portal.noarch 3.1.0-3.19.el6 ovirt-image-uploader.noarch 3.1.0-16.el6 ovirt-iso-uploader.noarch 3.1.0-16.el6 ovirt-log-collector.noarch 3.1.0-16.el6

Thanks and regards Patrick

Ok, managed to solve it. I force removed the datacenter and reinstalled the host. I added a new local storage to it and re-created the vms (disk images were moved and renamed from old non working local storage). So this host is up an running again. Regards Patrick -- Lobster LOGsuite GmbH, Münchner Straße 15a, D-82319 Starnberg HRB 178831, Amtsgericht München Geschäftsführer: Dr. Martin Fischer, Rolf Henrich