[Users] Unable to activate iSCSI domain after crash of host

Hello, Fedora 19 with 3.3.3. Only one host configured. After crash of host I'm not able to activate storage domain again. Any way to recover? Gianluca in engine.log 2014-02-07 08:11:12,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-32) [ 60d513d1] HostName = ovnode03 2014-02-07 08:11:12,602 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-32) [ 60d513d1] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IR SNonOperationalException: Not SPM: () 2014-02-07 08:11:12,613 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-32) [60d513d1] hostFr omVds::selectedVds - ovnode03, spmStatus Unknown_Pool, storage pool ISCSI 2014-02-07 08:11:12,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-32) [60d5 13d1] START, ConnectStoragePoolVDSCommand(HostName = ovnode03, HostId = b6f8f68f-4f9e-4c87-918b-aa1ff60f575a, storagePoolId = 546cd29c-7249-473 3-8fd5-317cff38ed71, vds_spm_id = 1, masterDomainId = f741671e-6480-4d7b-b357-8cf6e8d2c0f1, masterVersion = 2), log id: 3e99a2c6 2014-02-07 08:11:15,747 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-4) [465b0976] Lock Acquired to object EngineLock [exclusiveLocks= key: f741671e-6480-4d7b-b357-8cf6e8d2c0f1 value: STORAGE , sharedLocks= ] 2014-02-07 08:11:15,759 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-6-thread-49) [465b0976] Running command: A ctivateStorageDomainCommand internal: false. Entities affected : ID: f741671e-6480-4d7b-b357-8cf6e8d2c0f1 Type: Storage 2014-02-07 08:11:15,762 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-6-thread-49) [465b0976] Lock freed to obje ct EngineLock [exclusiveLocks= key: f741671e-6480-4d7b-b357-8cf6e8d2c0f1 value: STORAGE , sharedLocks= ] 2014-02-07 08:11:15,763 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-6-thread-49) [465b0976] ActivateStorage Do main. Before Connect all hosts to pool. Time:2/7/14 8:11 AM 2014-02-07 08:11:15,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-6-thread-49) [465b0976] START, ActivateStorageDomainVDSCommand( storagePoolId = 546cd29c-7249-4733-8fd5-317cff38ed71, ignoreFailoverLimit = false, storageDomainId = f741671e- 6480-4d7b-b357-8cf6e8d2c0f1), log id: da4b270 2014-02-07 08:11:16,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-32) [60d5 13d1] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=546cd29c-7249-4733-8fd5-317cff38ed7 1, msdUUID=f741671e-6480-4d7b-b357-8cf6e8d2c0f1']] 2014-02-07 08:11:16,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-32) [60d5 13d1] HostName = ovnode03 2014-02-07 08:11:16,740 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-32) [60d5 13d1] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMaste rDomainException: Cannot find master domain: 'spUUID=546cd29c-7249-4733-8fd5-317cff38ed71, msdUUID=f741671e-6480-4d7b-b357-8cf6e8d2c0f1' 2014-02-07 08:11:16,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-32) [60d5 13d1] FINISH, ConnectStoragePoolVDSCommand, log id: 3e99a2c6 In vdsm.log I get: Thread-85157::ERROR::2014-02-07 08:12:44,774::task::850::TaskManager.Task::(_setError) Task=`b9f3c2b5-18fa-4135-96f7-c152b5 ffe675`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 857, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 45, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1008, in connectStoragePool masterVersion, options) File "/usr/share/vdsm/storage/hsm.py", line 1062, in _connectStoragePool res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) File "/usr/share/vdsm/storage/sp.py", line 699, in connect self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) File "/usr/share/vdsm/storage/sp.py", line 1244, in __rebuild masterVersion=masterVersion) File "/usr/share/vdsm/storage/sp.py", line 1603, in getMasterDomain raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=546cd29c-7249-4733-8fd5-317cff38ed71, msdUUID=f741671e-6480-4 d7b-b357-8cf6e8d2c0f1' Thread-85157::DEBUG::2014-02-07 08:12:44,774::task::869::TaskManager.Task::(_run) Task=`b9f3c2b5-18fa-4135-96f7-c152b5ffe67 5`::Task._run: b9f3c2b5-18fa-4135-96f7-c152b5ffe675 ('546cd29c-7249-4733-8fd5-317cff38ed71', 1, '546cd29c-7249-4733-8fd5-31 7cff38ed71', 'f741671e-6480-4d7b-b357-8cf6e8d2c0f1', 2) {} failed - stopping task Thread-85157::DEBUG::2014-02-07 08:12:44,775::task::1194::TaskManager.Task::(stop) Task=`b9f3c2b5-18fa-4135-96f7-c152b5ffe6 75`::stopping in state preparing (force False) Thread-85157::DEBUG::2014-02-07 08:12:44,775::task::974::TaskManager.Task::(_decref) Task=`b9f3c2b5-18fa-4135-96f7-c152b5ff e675`::ref 1 aborting True Thread-85157::INFO::2014-02-07 08:12:44,775::task::1151::TaskManager.Task::(prepare) Task=`b9f3c2b5-18fa-4135-96f7-c152b5ffe675`::aborting: Task is aborted: 'Cannot find master domain' - code 304 Thread-85157::DEBUG::2014-02-07 08:12:44,775::task::1156::TaskManager.Task::(prepare) Task=`b9f3c2b5-18fa-4135-96f7-c152b5ffe675`::Prepare: aborted: Cannot find master domain Thread-85157::DEBUG::2014-02-07 08:12:44,776::task::974::TaskManager.Task::(_decref) Task=`b9f3c2b5-18fa-4135-96f7-c152b5ffe675`::ref 0 aborting True Thread-85157::DEBUG::2014-02-07 08:12:44,776::task::909::TaskManager.Task::(_doAbort) Task=`b9f3c2b5-18fa-4135-96f7-c152b5ffe675`::Task._doAbort: force False Thread-85157::DEBUG::2014-02-07 08:12:44,776::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-85157::DEBUG::2014-02-07 08:12:44,776::task::579::TaskManager.Task::(_updateState) Task=`b9f3c2b5-18fa-4135-96f7-c152b5ffe675`::moving from state preparing -> state aborting Thread-85157::DEBUG::2014-02-07 08:12:44,777::task::534::TaskManager.Task::(__state_aborting) Task=`b9f3c2b5-18fa-4135-96f7-c152b5ffe675`::_aborting: recover policy none Thread-85157::DEBUG::2014-02-07 08:12:44,777::task::579::TaskManager.Task::(_updateState) Task=`b9f3c2b5-18fa-4135-96f7-c152b5ffe675`::moving from state aborting -> state failed Thread-85157::DEBUG::2014-02-07 08:12:44,777::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-85157::DEBUG::2014-02-07 08:12:44,777::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-85157::ERROR::2014-02-07 08:12:44,777::dispatcher::67::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=546cd29c-7249-4733-8fd5-317cff38ed71, msdUUID=f741671e-6480-4d7b-b357-8cf6e8d2c0f1'", 'code ': 304}}