[Users] Unable to activate iSCSI domain after crash of host

Fri Feb 7 07:21:43 UTC 2014

Hello,
Fedora 19 with 3.3.3.
Only one host configured.
After crash of host I'm not able to activate storage domain again.

Any way to recover?
Gianluca

in engine.log
2014-02-07 08:11:12,602 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-32) [
60d513d1] HostName = ovnode03
2014-02-07 08:11:12,602 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-32) [
60d513d1] Command HSMGetAllTasksStatusesVDS execution failed.
Exception: IRSNonOperationalException: IRSGenericException:
IRSErrorException: IR
SNonOperationalException: Not SPM: ()
2014-02-07 08:11:12,613 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-32) [60d513d1] hostFr
omVds::selectedVds - ovnode03, spmStatus Unknown_Pool, storage pool ISCSI
2014-02-07 08:11:12,615 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-32) [60d5
13d1] START, ConnectStoragePoolVDSCommand(HostName = ovnode03, HostId
= b6f8f68f-4f9e-4c87-918b-aa1ff60f575a, storagePoolId =
546cd29c-7249-473
3-8fd5-317cff38ed71, vds_spm_id = 1, masterDomainId =
f741671e-6480-4d7b-b357-8cf6e8d2c0f1, masterVersion = 2), log id:
3e99a2c6
2014-02-07 08:11:15,747 INFO
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(ajp--127.0.0.1-8702-4) [465b0976] Lock Acquired
 to object EngineLock [exclusiveLocks= key:
f741671e-6480-4d7b-b357-8cf6e8d2c0f1 value: STORAGE
, sharedLocks= ]
2014-02-07 08:11:15,759 INFO
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(pool-6-thread-49) [465b0976] Running command: A
ctivateStorageDomainCommand internal: false. Entities affected :  ID:
f741671e-6480-4d7b-b357-8cf6e8d2c0f1 Type: Storage
2014-02-07 08:11:15,762 INFO
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(pool-6-thread-49) [465b0976] Lock freed to obje
ct EngineLock [exclusiveLocks= key:
f741671e-6480-4d7b-b357-8cf6e8d2c0f1 value: STORAGE
, sharedLocks= ]
2014-02-07 08:11:15,763 INFO
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand]
(pool-6-thread-49) [465b0976] ActivateStorage Do
main. Before Connect all hosts to pool. Time:2/7/14 8:11 AM
2014-02-07 08:11:15,765 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(pool-6-thread-49) [465b0976] START,
ActivateStorageDomainVDSCommand( storagePoolId =
546cd29c-7249-4733-8fd5-317cff38ed71, ignoreFailoverLimit = false,
storageDomainId = f741671e-
6480-4d7b-b357-8cf6e8d2c0f1), log id: da4b270
2014-02-07 08:11:16,739 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-32) [60d5
13d1] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand
return value
 StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304,
mMessage=Cannot find master domain:
'spUUID=546cd29c-7249-4733-8fd5-317cff38ed7
1, msdUUID=f741671e-6480-4d7b-b357-8cf6e8d2c0f1']]
2014-02-07 08:11:16,740 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-32) [60d5
13d1] HostName = ovnode03
2014-02-07 08:11:16,740 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-32) [60d5
13d1] Command ConnectStoragePoolVDS execution failed. Exception:
IRSNoMasterDomainException: IRSGenericException: IRSErrorException:
IRSNoMaste
rDomainException: Cannot find master domain:
'spUUID=546cd29c-7249-4733-8fd5-317cff38ed71,
msdUUID=f741671e-6480-4d7b-b357-8cf6e8d2c0f1'
2014-02-07 08:11:16,741 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-32) [60d5
13d1] FINISH, ConnectStoragePoolVDSCommand, log id: 3e99a2c6

In vdsm.log I get:

Thread-85157::ERROR::2014-02-07
08:12:44,774::task::850::TaskManager.Task::(_setError)
Task=`b9f3c2b5-18fa-4135-96f7-c152b5
ffe675`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 857, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1008, in connectStoragePool
    masterVersion, options)
  File "/usr/share/vdsm/storage/hsm.py", line 1062, in _connectStoragePool
    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 699, in connect
    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1244, in __rebuild
    masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1603, in getMasterDomain
    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
StoragePoolMasterNotFound: Cannot find master domain:
'spUUID=546cd29c-7249-4733-8fd5-317cff38ed71, msdUUID=f741671e-6480-4
d7b-b357-8cf6e8d2c0f1'
Thread-85157::DEBUG::2014-02-07
08:12:44,774::task::869::TaskManager.Task::(_run)
Task=`b9f3c2b5-18fa-4135-96f7-c152b5ffe67
5`::Task._run: b9f3c2b5-18fa-4135-96f7-c152b5ffe675
('546cd29c-7249-4733-8fd5-317cff38ed71', 1,
'546cd29c-7249-4733-8fd5-31
7cff38ed71', 'f741671e-6480-4d7b-b357-8cf6e8d2c0f1', 2) {} failed -
stopping task
Thread-85157::DEBUG::2014-02-07
08:12:44,775::task::1194::TaskManager.Task::(stop)
Task=`b9f3c2b5-18fa-4135-96f7-c152b5ffe6
75`::stopping in state preparing (force False)
Thread-85157::DEBUG::2014-02-07
08:12:44,775::task::974::TaskManager.Task::(_decref)
Task=`b9f3c2b5-18fa-4135-96f7-c152b5ff
e675`::ref 1 aborting True
Thread-85157::INFO::2014-02-07
08:12:44,775::task::1151::TaskManager.Task::(prepare)
Task=`b9f3c2b5-18fa-4135-96f7-c152b5ffe675`::aborting: Task is
aborted: 'Cannot find master domain' - code 304
Thread-85157::DEBUG::2014-02-07
08:12:44,775::task::1156::TaskManager.Task::(prepare)
Task=`b9f3c2b5-18fa-4135-96f7-c152b5ffe675`::Prepare: aborted: Cannot
find master domain
Thread-85157::DEBUG::2014-02-07
08:12:44,776::task::974::TaskManager.Task::(_decref)
Task=`b9f3c2b5-18fa-4135-96f7-c152b5ffe675`::ref 0 aborting True
Thread-85157::DEBUG::2014-02-07
08:12:44,776::task::909::TaskManager.Task::(_doAbort)
Task=`b9f3c2b5-18fa-4135-96f7-c152b5ffe675`::Task._doAbort: force
False
Thread-85157::DEBUG::2014-02-07
08:12:44,776::resourceManager::976::ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-85157::DEBUG::2014-02-07
08:12:44,776::task::579::TaskManager.Task::(_updateState)
Task=`b9f3c2b5-18fa-4135-96f7-c152b5ffe675`::moving from state
preparing -> state aborting
Thread-85157::DEBUG::2014-02-07
08:12:44,777::task::534::TaskManager.Task::(__state_aborting)
Task=`b9f3c2b5-18fa-4135-96f7-c152b5ffe675`::_aborting: recover policy
none
Thread-85157::DEBUG::2014-02-07
08:12:44,777::task::579::TaskManager.Task::(_updateState)
Task=`b9f3c2b5-18fa-4135-96f7-c152b5ffe675`::moving from state
aborting -> state failed
Thread-85157::DEBUG::2014-02-07
08:12:44,777::resourceManager::939::ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-85157::DEBUG::2014-02-07
08:12:44,777::resourceManager::976::ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-85157::ERROR::2014-02-07
08:12:44,777::dispatcher::67::Storage.Dispatcher.Protect::(run)
{'status': {'message': "Cannot find master domain:
'spUUID=546cd29c-7249-4733-8fd5-317cff38ed71,
msdUUID=f741671e-6480-4d7b-b357-8cf6e8d2c0f1'", 'code
': 304}}