[Users] Unable to activate data center after host reboot.

27 Apr 2013

      Hi,

I have a single machine with an older oVirt installation that was 
configured along the lines of this document: 
http://blog.jebpages.com/archives/how-to-get-up-and-running-with-ovirt/

Installed packages are:

ovirt-engine-3.0.0_0001-1.6.fc16.x86_64
vdsm-4.9.3.3-0.fc16.x86_64

It has been largely untouched since configured, and has been trouble free.

I've had to physically move the server, which involved powering it down 
Until then, it had been up for around 250 days. Upon restarting it, the 
oVirt web admin page tells me:

No valid Data Storage Domains are available in Data Center Default 
(please check your storage infrastructure).

... and I can't activate my "Default" data center, meaning I can't 
activate any storage domains and thus am unable to start my VM's.

Below is the output for both engine.log and vdsm.log, taken from the 
moment I attempt to activate the data center.

Does anybody have any insights as to what's going wrong?

Best regards,

--Steven Wiltshire.

Storage is by NFS on the single host. Here is the relevant output of 
"mount"

------------------------------------------------------------------
10.0.0.154:/home/vm/data/iso on 
/rhev/data-center/mnt/10.0.0.154:_home_vm_data_iso type nfs4 
(rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,port=0,timeo=600,retrans=6,sec=sys,clientaddr=10.0.0.154,local_lock=none,addr=10.0.0.154)
10.0.0.154:/home/vm/data/export on 
/rhev/data-center/mnt/10.0.0.154:_home_vm_data_export type nfs4 
(rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,port=0,timeo=600,retrans=6,sec=sys,clientaddr=10.0.0.154,local_lock=none,addr=10.0.0.154)
10.0.0.154:/home/vm/data/data on 
/rhev/data-center/mnt/10.0.0.154:_home_vm_data_data type nfs4 
(rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,port=0,timeo=600,retrans=6,sec=sys,clientaddr=10.0.0.154,local_lock=none,addr=10.0.0.154)
------------------------------------------------------------------

Here is the contents of the metadata file in 
/home/vm/data/data/025062dd-d69e-40ac-9031-c080b51647ae/dom_md:

------------------------------------------------------------------
CLASS=Data
DESCRIPTION=Data
IOOPTIMEOUTSEC=10
LEASERETRIES=3
LEASETIMESEC=60
LOCKPOLICY=
LOCKRENEWALINTERVALSEC=5
MASTER_VERSION=1
POOL_DESCRIPTION=Default
POOL_DOMAINS=025062dd-d69e-40ac-9031-c080b51647ae:Active,d30604e3-3ca8-470a-8da4-18c9402f7504:Active,503e6677-2d51-4668-ab17-f914bdb57c19:Active 

POOL_SPM_ID=1
POOL_SPM_LVER=30
POOL_UUID=2b633f9a-8840-11e1-9ed7-17e6c491db84
REMOTE_PATH=10.0.0.154:/home/vm/data/data
ROLE=Master
SDUUID=025062dd-d69e-40ac-9031-c080b51647ae
TYPE=NFS
VERSION=0
_SHA_CKSUM=c838aeb3b1f5180fd519ee3461e154d2bd79568d
------------------------------------------------------------------

 From engine.log:
------------------------------------------------------------------
2013-04-27 14:14:50,025 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] 
(QuartzScheduler_Worker-100) Command 
org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand 
return value
  Class Name: 
org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc
mStatus                       Class Name: 
org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc
mCode                         304
mMessage                      Cannot find master domain: 
'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, 
msdUUID=025062dd-d69e-40ac-9031-c080b51647ae'

2013-04-27 14:14:50,026 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] 
(QuartzScheduler_Worker-100) Vds: BigServer
2013-04-27 14:14:50,026 ERROR 
[org.ovirt.engine.core.vdsbroker.VDSCommandBase] 
(QuartzScheduler_Worker-100) Command ConnectStoragePoolVDS execution 
failed. Exception: IRSNoMasterDomainException: IRSGenericException: 
IRSErrorException: IRSNoMasterDomainException: Cannot find master 
domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, 
msdUUID=025062dd-d69e-40ac-9031-c080b51647ae'
2013-04-27 14:14:50,026 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] 
(QuartzScheduler_Worker-100) FINISH, ConnectStoragePoolVDSCommand, log 
id: 7fec8068
2013-04-27 14:14:50,027 INFO 
[org.ovirt.engine.core.utils.transaction.TransactionSupport] 
(QuartzScheduler_Worker-100) transaction rolled back
2013-04-27 14:14:50,028 ERROR 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-100) IrsBroker::Failed::GetStoragePoolInfoVDS
2013-04-27 14:14:50,028 ERROR 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-100) Exception: IRSGenericException: 
IRSErrorException: IRSNoMasterDomainException: Cannot find master 
domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, 
msdUUID=025062dd-d69e-40ac-9031-c080b51647ae'
2013-04-27 14:14:50,063 INFO 
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] 
(QuartzScheduler_Worker-100) Running command: 
ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 
025062dd-d69e-40ac-9031-c080b51647ae Type: Storage
2013-04-27 14:14:50,098 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] 
(QuartzScheduler_Worker-100) START, 
MarkPoolInReconstructModeVDSCommand(storagePoolId = 
2b633f9a-8840-11e1-9ed7-17e6c491db84, ignoreFailoverLimit = false, 
compatabilityVersion = null, reconstructMarkAction = ClearCache), log 
id: 31dad08a
2013-04-27 14:14:50,098 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-100) clearing cache for problematic entities in 
pool 2b633f9a-8840-11e1-9ed7-17e6c491db84
2013-04-27 14:14:50,098 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] 
(QuartzScheduler_Worker-100) FINISH, 
MarkPoolInReconstructModeVDSCommand, log id: 31dad08a
2013-04-27 14:14:57,494 WARN 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-27) domain 503e6677-2d51-4668-ab17-f914bdb57c19 
in problem. vds: BigServer
2013-04-27 14:14:57,495 WARN 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-27) domain d30604e3-3ca8-470a-8da4-18c9402f7504 
in problem. vds: BigServer
2013-04-27 14:15:00,140 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-30) hostFromVds::selectedVds - BigServer, 
spmStatus Unknown_Pool, storage pool Default
2013-04-27 14:15:00,153 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] 
(QuartzScheduler_Worker-30) START, ConnectStoragePoolVDSCommand(vdsId = 
aeac7aa6-8840-11e1-99aa-7fbd3e56e26e, storagePoolId = 
2b633f9a-8840-11e1-9ed7-17e6c491db84, vds_spm_id = 1, masterDomainId = 
025062dd-d69e-40ac-9031-c080b51647ae, masterVersion = 1), log id: 5b22a385
2013-04-27 14:15:23,608 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-13) starting ProcessDomainRecovery for domain 
d30604e3-3ca8-470a-8da4-18c9402f7504
2013-04-27 14:15:23,617 WARN 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-13) domain d30604e3-3ca8-470a-8da4-18c9402f7504 
was reported by all hosts in status UP as problematic. Moving the Domain 
to NonOperational.
2013-04-27 14:15:23,625 INFO 
[org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] 
(QuartzScheduler_Worker-13) Running command: 
HandleFailedStorageDomainCommand internal: true. Entities affected :  
ID: d30604e3-3ca8-470a-8da4-18c9402f7504 Type: Storage
2013-04-27 14:15:23,633 WARN 
[org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] 
(QuartzScheduler_Worker-13) CanDoAction of action 
DeactivateStorageDomain failed. 
Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL
2013-04-27 14:15:23,637 ERROR 
[org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] 
(QuartzScheduler_Worker-13) Transaction rolled-back for command: 
org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand.
2013-04-27 14:15:23,637 INFO 
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] 
(QuartzScheduler_Worker-13) No string for UNASSIGNED type. Use default Log
2013-04-27 14:15:23,638 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-12) starting ProcessDomainRecovery for domain 
503e6677-2d51-4668-ab17-f914bdb57c19
2013-04-27 14:15:23,646 WARN 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-12) domain 503e6677-2d51-4668-ab17-f914bdb57c19 
was reported by all hosts in status UP as problematic. Moving the Domain 
to NonOperational.
2013-04-27 14:15:23,653 INFO 
[org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] 
(QuartzScheduler_Worker-12) Running command: 
HandleFailedStorageDomainCommand internal: true. Entities affected :  
ID: 503e6677-2d51-4668-ab17-f914bdb57c19 Type: Storage
2013-04-27 14:15:23,661 WARN 
[org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] 
(QuartzScheduler_Worker-12) CanDoAction of action 
DeactivateStorageDomain failed. 
Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL
2013-04-27 14:15:23,665 ERROR 
[org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] 
(QuartzScheduler_Worker-12) Transaction rolled-back for command: 
org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand.
2013-04-27 14:15:23,665 INFO 
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] 
(QuartzScheduler_Worker-12) No string for UNASSIGNED type. Use default Log
2013-04-27 14:15:27,944 WARN 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-24) domain 503e6677-2d51-4668-ab17-f914bdb57c19 
in problem. vds: BigServer
2013-04-27 14:15:27,944 WARN 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-24) domain d30604e3-3ca8-470a-8da4-18c9402f7504 
in problem. vds: BigServer
2013-04-27 14:15:34,953 INFO 
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] 
(pool-5-thread-33) Lock Acquired to object EngineLock [exclusiveLocks= 
key: org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand 
value: 025062dd-d69e-40ac-9031-c080b51647ae
, sharedLocks= ]
2013-04-27 14:15:34,954 INFO 
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] 
(pool-5-thread-33) Running command: ActivateStorageDomainCommand 
internal: false. Entities affected :  ID: 
025062dd-d69e-40ac-9031-c080b51647ae Type: Storage
2013-04-27 14:15:34,965 INFO 
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] 
(pool-5-thread-33) Lock freed to object EngineLock [exclusiveLocks= key: 
org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand value: 
025062dd-d69e-40ac-9031-c080b51647ae
, sharedLocks= ]
2013-04-27 14:15:34,965 INFO 
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] 
(pool-5-thread-33) ActivateStorage Domain. Before Connect all hosts to 
pool. Time:4/27/13 2:15 PM
2013-04-27 14:15:35,004 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] 
(pool-5-thread-48) START, 
ValidateStorageServerConnectionVDSCommand(vdsId = 
aeac7aa6-8840-11e1-99aa-7fbd3e56e26e, storagePoolId = 
00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList 
= [{ id: 60e73855-daa2-45f5-9c59-5973ca89e830, connection: 
10.0.0.154:/home/vm/data/data };]), log id: 3f3f3015
2013-04-27 14:15:35,139 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] 
(pool-5-thread-48) FINISH, ValidateStorageServerConnectionVDSCommand, 
return: {60e73855-daa2-45f5-9c59-5973ca89e830=0}, log id: 3f3f3015
2013-04-27 14:15:35,140 INFO 
[org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] 
(pool-5-thread-48) Running command: ConnectStorageToVdsCommand internal: 
true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa 
Type: System
2013-04-27 14:15:35,149 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] 
(pool-5-thread-48) START, ConnectStorageServerVDSCommand(vdsId = 
aeac7aa6-8840-11e1-99aa-7fbd3e56e26e, storagePoolId = 
00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList 
= [{ id: 60e73855-daa2-45f5-9c59-5973ca89e830, connection: 
10.0.0.154:/home/vm/data/data };]), log id: 32a6a7fd
2013-04-27 14:15:35,183 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] 
(pool-5-thread-48) FINISH, ConnectStorageServerVDSCommand, return: 
{60e73855-daa2-45f5-9c59-5973ca89e830=0}, log id: 32a6a7fd
2013-04-27 14:15:35,186 INFO 
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] 
(pool-5-thread-33) ActivateStorage Domain. After Connect all hosts to 
pool. Time:4/27/13 2:15 PM
2013-04-27 14:15:35,187 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] 
(pool-5-thread-33) START, ActivateStorageDomainVDSCommand(storagePoolId 
= 2b633f9a-8840-11e1-9ed7-17e6c491db84, ignoreFailoverLimit = false, 
compatabilityVersion = null, storageDomainId = 
025062dd-d69e-40ac-9031-c080b51647ae), log id: 4e53941a
2013-04-27 14:16:04,042 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] 
(QuartzScheduler_Worker-30) Command 
org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand 
return value
  Class Name: 
org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc
mStatus                       Class Name: 
org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc
mCode                         304
mMessage                      Cannot find master domain: 
'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, 
msdUUID=025062dd-d69e-40ac-9031-c080b51647ae'

2013-04-27 14:16:04,043 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] 
(QuartzScheduler_Worker-30) Vds: BigServer
2013-04-27 14:16:04,043 ERROR 
[org.ovirt.engine.core.vdsbroker.VDSCommandBase] 
(QuartzScheduler_Worker-30) Command ConnectStoragePoolVDS execution 
failed. Exception: IRSNoMasterDomainException: IRSGenericException: 
IRSErrorException: IRSNoMasterDomainException: Cannot find master 
domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, 
msdUUID=025062dd-d69e-40ac-9031-c080b51647ae'
2013-04-27 14:16:04,044 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] 
(QuartzScheduler_Worker-30) FINISH, ConnectStoragePoolVDSCommand, log 
id: 5b22a385
2013-04-27 14:16:04,045 INFO 
[org.ovirt.engine.core.utils.transaction.TransactionSupport] 
(QuartzScheduler_Worker-30) transaction rolled back
2013-04-27 14:16:04,045 ERROR 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-30) IrsBroker::Failed::GetStoragePoolInfoVDS
2013-04-27 14:16:04,046 ERROR 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-30) Exception: IRSGenericException: 
IRSErrorException: IRSNoMasterDomainException: Cannot find master 
domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, 
msdUUID=025062dd-d69e-40ac-9031-c080b51647ae'
2013-04-27 14:16:04,060 WARN 
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] 
(QuartzScheduler_Worker-30) CanDoAction of action 
ReconstructMasterDomain failed. 
Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2
2013-04-27 14:16:04,092 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(pool-5-thread-33) hostFromVds::selectedVds - BigServer, spmStatus 
Unknown_Pool, storage pool Default
2013-04-27 14:16:04,103 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] 
(pool-5-thread-33) START, ConnectStoragePoolVDSCommand(vdsId = 
aeac7aa6-8840-11e1-99aa-7fbd3e56e26e, storagePoolId = 
2b633f9a-8840-11e1-9ed7-17e6c491db84, vds_spm_id = 1, masterDomainId = 
025062dd-d69e-40ac-9031-c080b51647ae, masterVersion = 1), log id: 674d0c6d
2013-04-27 14:16:41,155 WARN 
[org.ovirt.engine.core.bll.GetConfigurationValueQuery] 
(http--0.0.0.0-8080-5) calling GetConfigurationValueQuery with null 
version, using default general for version
2013-04-27 14:16:41,155 WARN 
[org.ovirt.engine.core.bll.GetConfigurationValueQuery] 
(http--0.0.0.0-8080-7) calling GetConfigurationValueQuery with null 
version, using default general for version
2013-04-27 14:16:41,156 WARN 
[org.ovirt.engine.core.bll.GetConfigurationValueQuery] 
(http--0.0.0.0-8080-8) calling GetConfigurationValueQuery with null 
version, using default general for version
2013-04-27 14:16:41,157 WARN 
[org.ovirt.engine.core.bll.GetConfigurationValueQuery] 
(http--0.0.0.0-8080-12) calling GetConfigurationValueQuery with null 
version, using default general for version
2013-04-27 14:16:41,256 WARN 
[org.ovirt.engine.core.bll.GetConfigurationValueQuery] 
(http--0.0.0.0-8080-12) calling GetConfigurationValueQuery with null 
version, using default general for version
2013-04-27 14:16:41,285 WARN 
[org.ovirt.engine.core.bll.GetConfigurationValueQuery] 
(http--0.0.0.0-8080-6) calling GetConfigurationValueQuery with null 
version, using default general for version
2013-04-27 14:17:08,014 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] 
(pool-5-thread-33) Command 
org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand 
return value
  Class Name: 
org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc
mStatus                       Class Name: 
org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc
mCode                         304
mMessage                      Cannot find master domain: 
'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, 
msdUUID=025062dd-d69e-40ac-9031-c080b51647ae'

2013-04-27 14:17:08,014 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] 
(pool-5-thread-33) Vds: BigServer
2013-04-27 14:17:08,015 ERROR 
[org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-33) 
Command ConnectStoragePoolVDS execution failed. Exception: 
IRSNoMasterDomainException: IRSGenericException: IRSErrorException: 
IRSNoMasterDomainException: Cannot find master domain: 
'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, 
msdUUID=025062dd-d69e-40ac-9031-c080b51647ae'
2013-04-27 14:17:08,015 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] 
(pool-5-thread-33) FINISH, ConnectStoragePoolVDSCommand, log id: 674d0c6d
2013-04-27 14:17:08,016 INFO 
[org.ovirt.engine.core.utils.transaction.TransactionSupport] 
(pool-5-thread-33) transaction rolled back
2013-04-27 14:17:08,016 ERROR 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(pool-5-thread-33) IrsBroker::Failed::ActivateStorageDomainVDS
2013-04-27 14:17:08,017 ERROR 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(pool-5-thread-33) Exception: IRSGenericException: IRSErrorException: 
IRSNoMasterDomainException: Cannot find master domain: 
'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, 
msdUUID=025062dd-d69e-40ac-9031-c080b51647ae'
2013-04-27 14:17:08,030 WARN 
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] 
(pool-5-thread-33) CanDoAction of action ReconstructMasterDomain failed. 
Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2
2013-04-27 14:17:08,030 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] 
(pool-5-thread-33) FINISH, ActivateStorageDomainVDSCommand, log id: 
4e53941a
2013-04-27 14:17:08,031 ERROR 
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] 
(pool-5-thread-33) Command 
org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand throw Vdc 
Bll exception. With error message VdcBLLException: 
org.ovirt.engine.core.vdsbroker.irsbroker.IRSNoMasterDomainException: 
IRSGenericException: IRSErrorException: IRSNoMasterDomainException: 
Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, 
msdUUID=025062dd-d69e-40ac-9031-c080b51647ae'
2013-04-27 14:17:08,039 INFO 
[org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] 
(pool-5-thread-33) Command [id=e01d2f93-6052-40bb-847f-1ca7fedbdb0c]: 
Compensating CHANGED_STATUS_ONLY of 
org.ovirt.engine.core.common.businessentities.storage_pool_iso_map; 
snapshot: EntityStatusSnapshot [id=storagePoolId = 
2b633f9a-8840-11e1-9ed7-17e6c491db84, storageId = 
025062dd-d69e-40ac-9031-c080b51647ae, status=InActive].
2013-04-27 14:17:08,065 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-32) hostFromVds::selectedVds - BigServer, 
spmStatus Unknown_Pool, storage pool Default
2013-04-27 14:17:08,076 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] 
(QuartzScheduler_Worker-32) START, ConnectStoragePoolVDSCommand(vdsId = 
aeac7aa6-8840-11e1-99aa-7fbd3e56e26e, storagePoolId = 
2b633f9a-8840-11e1-9ed7-17e6c491db84, vds_spm_id = 1, masterDomainId = 
025062dd-d69e-40ac-9031-c080b51647ae, masterVersion = 1), log id: 2792d3ec
2013-04-27 14:18:11,976 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] 
(QuartzScheduler_Worker-32) Command 
org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand 
return value
  Class Name: 
org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc
mStatus                       Class Name: 
org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc
mCode                         304
mMessage                      Cannot find master domain: 
'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, 
msdUUID=025062dd-d69e-40ac-9031-c080b51647ae'

2013-04-27 14:18:11,976 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] 
(QuartzScheduler_Worker-32) Vds: BigServer
2013-04-27 14:18:11,976 ERROR 
[org.ovirt.engine.core.vdsbroker.VDSCommandBase] 
(QuartzScheduler_Worker-32) Command ConnectStoragePoolVDS execution 
failed. Exception: IRSNoMasterDomainException: IRSGenericException: 
IRSErrorException: IRSNoMasterDomainException: Cannot find master 
domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, 
msdUUID=025062dd-d69e-40ac-9031-c080b51647ae'
2013-04-27 14:18:11,977 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] 
(QuartzScheduler_Worker-32) FINISH, ConnectStoragePoolVDSCommand, log 
id: 2792d3ec
2013-04-27 14:18:11,978 INFO 
[org.ovirt.engine.core.utils.transaction.TransactionSupport] 
(QuartzScheduler_Worker-32) transaction rolled back
2013-04-27 14:18:11,978 ERROR 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-32) IrsBroker::Failed::GetStoragePoolInfoVDS
2013-04-27 14:18:11,978 ERROR 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-32) Exception: IRSGenericException: 
IRSErrorException: IRSNoMasterDomainException: Cannot find master 
domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, 
msdUUID=025062dd-d69e-40ac-9031-c080b51647ae'
2013-04-27 14:18:12,017 INFO 
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] 
(QuartzScheduler_Worker-32) Running command: 
ReconstructMasterDomainCommand internal: true. Entities affected :  ID: 
025062dd-d69e-40ac-9031-c080b51647ae Type: Storage
2013-04-27 14:18:12,051 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] 
(QuartzScheduler_Worker-32) START, 
MarkPoolInReconstructModeVDSCommand(storagePoolId = 
2b633f9a-8840-11e1-9ed7-17e6c491db84, ignoreFailoverLimit = false, 
compatabilityVersion = null, reconstructMarkAction = ClearCache), log 
id: 59cf4ee8
2013-04-27 14:18:12,051 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-32) clearing cache for problematic entities in 
pool 2b633f9a-8840-11e1-9ed7-17e6c491db84
2013-04-27 14:18:12,051 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] 
(QuartzScheduler_Worker-32) FINISH, MarkPoolInReconstructModeVDSCommand, 
log id: 59cf4ee8
2013-04-27 14:18:20,632 WARN 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-83) domain 503e6677-2d51-4668-ab17-f914bdb57c19 
in problem. vds: BigServer
2013-04-27 14:18:20,633 WARN 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-83) domain d30604e3-3ca8-470a-8da4-18c9402f7504 
in problem. vds: BigServer
2013-04-27 14:18:22,093 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-85) hostFromVds::selectedVds - BigServer, 
spmStatus Unknown_Pool, storage pool Default
2013-04-27 14:18:22,104 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] 
(QuartzScheduler_Worker-85) START, ConnectStoragePoolVDSCommand(vdsId = 
aeac7aa6-8840-11e1-99aa-7fbd3e56e26e, storagePoolId = 
2b633f9a-8840-11e1-9ed7-17e6c491db84, vds_spm_id = 1, masterDomainId = 
025062dd-d69e-40ac-9031-c080b51647ae, masterVersion = 1), log id: 73e64219
2013-04-27 14:19:06,753 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-6) starting ProcessDomainRecovery for domain 
503e6677-2d51-4668-ab17-f914bdb57c19
2013-04-27 14:19:06,778 WARN 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-6) domain 503e6677-2d51-4668-ab17-f914bdb57c19 
was reported by all hosts in status UP as problematic. Moving the Domain 
to NonOperational.
2013-04-27 14:19:06,785 INFO 
[org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] 
(QuartzScheduler_Worker-6) Running command: 
HandleFailedStorageDomainCommand internal: true. Entities affected :  
ID: 503e6677-2d51-4668-ab17-f914bdb57c19 Type: Storage
2013-04-27 14:19:06,794 WARN 
[org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] 
(QuartzScheduler_Worker-6) CanDoAction of action DeactivateStorageDomain 
failed. 
Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL
2013-04-27 14:19:06,799 ERROR 
[org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] 
(QuartzScheduler_Worker-6) Transaction rolled-back for command: 
org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand.
2013-04-27 14:19:06,800 INFO 
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] 
(QuartzScheduler_Worker-6) No string for UNASSIGNED type. Use default Log
2013-04-27 14:19:06,801 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-13) starting ProcessDomainRecovery for domain 
d30604e3-3ca8-470a-8da4-18c9402f7504
2013-04-27 14:19:06,809 WARN 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-13) domain d30604e3-3ca8-470a-8da4-18c9402f7504 
was reported by all hosts in status UP as problematic. Moving the Domain 
to NonOperational.
2013-04-27 14:19:06,816 INFO 
[org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] 
(QuartzScheduler_Worker-13) Running command: 
HandleFailedStorageDomainCommand internal: true. Entities affected :  
ID: d30604e3-3ca8-470a-8da4-18c9402f7504 Type: Storage
2013-04-27 14:19:06,824 WARN 
[org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] 
(QuartzScheduler_Worker-13) CanDoAction of action 
DeactivateStorageDomain failed. 
Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL
2013-04-27 14:19:06,828 ERROR 
[org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] 
(QuartzScheduler_Worker-13) Transaction rolled-back for command: 
org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand.
2013-04-27 14:19:06,829 INFO 
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] 
(QuartzScheduler_Worker-13) No string for UNASSIGNED type. Use default Log
2013-04-27 14:19:11,388 WARN 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-17) domain 503e6677-2d51-4668-ab17-f914bdb57c19 
in problem. vds: BigServer
2013-04-27 14:19:11,389 WARN 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-17) domain d30604e3-3ca8-470a-8da4-18c9402f7504 
in problem. vds: BigServer
------------------------------------------------------------------

And from vdsm.log:

------------------------------------------------------------------
Thread-1643::ERROR::2013-04-27 
14:23:08,038::sdc::113::Storage.StorageDomainCache::(_findDomain) Error 
while looking for domain `025062dd-d69e-40ac-9031-c080b51647ae`
Traceback (most recent call last):
   File "/usr/share/vdsm/storage/sdc.py", line 109, in _findDomain
     return mod.findDomain(sdUUID)
   File "/usr/share/vdsm/storage/nfsSD.py", line 149, in findDomain
     return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
   File "/usr/share/vdsm/storage/fileSD.py", line 104, in __init__
     sdUUID = metadata[sd.DMDK_SDUUID]
   File "/usr/share/vdsm/storage/persistentDict.py", line 75, in 
__getitem__
     return dec(self._dict[key])
   File "/usr/share/vdsm/storage/persistentDict.py", line 183, in 
__getitem__
     with self._accessWrapper():
   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
     return self.gen.next()
   File "/usr/share/vdsm/storage/persistentDict.py", line 137, in 
_accessWrapper
     self.refresh()
   File "/usr/share/vdsm/storage/persistentDict.py", line 214, in refresh
     lines = self._metaRW.readlines()
   File "/usr/share/vdsm/storage/fileSD.py", line 71, in readlines
     return misc.stripNewLines(self._oop.directReadLines(self._metafile))
   File "/usr/share/vdsm/storage/processPool.py", line 53, in wrapper
     return self.runExternally(func, *args, **kwds)
   File "/usr/share/vdsm/storage/processPool.py", line 64, in runExternally
     return self._procPool.runExternally(*args, **kwargs)
   File "/usr/share/vdsm/storage/processPool.py", line 145, in 
runExternally
     raise Timeout("Operation Stuck")
Timeout: Operation Stuck
Thread-1643::DEBUG::2013-04-27 
14:23:08,039::resourceManager::535::ResourceManager::(releaseResource) 
Trying to release resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84'
Thread-1643::DEBUG::2013-04-27 
14:23:08,040::resourceManager::550::ResourceManager::(releaseResource) 
Released resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84' (0 
active users)
Thread-1643::DEBUG::2013-04-27 
14:23:08,041::resourceManager::555::ResourceManager::(releaseResource) 
Resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84' is free, finding 
out if anyone is waiting for it.
Thread-1643::DEBUG::2013-04-27 
14:23:08,041::resourceManager::562::ResourceManager::(releaseResource) 
No one is waiting for resource 
'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84', Clearing records.
Thread-1643::ERROR::2013-04-27 
14:23:08,042::task::855::TaskManager.Task::(_setError) 
Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::Unexpected error
Traceback (most recent call last):
   File "/usr/share/vdsm/storage/task.py", line 863, in _run
     return fn(*args, **kargs)
   File "/usr/share/vdsm/logUtils.py", line 38, in wrapper
     res = f(*args, **kwargs)
   File "/usr/share/vdsm/storage/hsm.py", line 721, in connectStoragePool
     return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, 
masterVersion, options)
   File "/usr/share/vdsm/storage/hsm.py", line 763, in _connectStoragePool
     res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
   File "/usr/share/vdsm/storage/sp.py", line 624, in connect
     self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
   File "/usr/share/vdsm/storage/sp.py", line 1097, in __rebuild
     self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, 
masterVersion=masterVersion)
   File "/usr/share/vdsm/storage/sp.py", line 1435, in getMasterDomain
     raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
StoragePoolMasterNotFound: Cannot find master domain: 
'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, 
msdUUID=025062dd-d69e-40ac-9031-c080b51647ae'
Thread-1643::DEBUG::2013-04-27 
14:23:08,043::task::874::TaskManager.Task::(_run) 
Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::Task._run: 
049a3ccf-e6d5-4363-b05b-90796a99fefb 
('2b633f9a-8840-11e1-9ed7-17e6c491db84', 1, '2b633f9a-8840-11e1
-9ed7-17e6c491db84', '025062dd-d69e-40ac-9031-c080b51647ae', 1) {} 
failed - stopping task
Thread-1643::DEBUG::2013-04-27 
14:23:08,044::task::1201::TaskManager.Task::(stop) 
Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::stopping in state preparing 
(force False)
Thread-1643::DEBUG::2013-04-27 
14:23:08,044::task::980::TaskManager.Task::(_decref) 
Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::ref 1 aborting True
Thread-1643::INFO::2013-04-27 
14:23:08,045::task::1159::TaskManager.Task::(prepare) 
Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::aborting: Task is aborted: 
'Cannot find master domain' - code 304
Thread-1643::DEBUG::2013-04-27 
14:23:08,045::task::1164::TaskManager.Task::(prepare) 
Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::Prepare: aborted: Cannot 
find master domain
Thread-1643::DEBUG::2013-04-27 
14:23:08,046::task::980::TaskManager.Task::(_decref) 
Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::ref 0 aborting True
Thread-1643::DEBUG::2013-04-27 
14:23:08,046::task::915::TaskManager.Task::(_doAbort) 
Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::Task._doAbort: force False
Thread-1643::DEBUG::2013-04-27 
14:23:08,047::resourceManager::841::ResourceManager.Owner::(cancelAll) 
Owner.cancelAll requests {}
Thread-1643::DEBUG::2013-04-27 
14:23:08,047::task::588::TaskManager.Task::(_updateState) 
Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::moving from state preparing 
-> state aborting
Thread-1643::DEBUG::2013-04-27 
14:23:08,048::task::537::TaskManager.Task::(__state_aborting) 
Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::_aborting: recover policy none
Thread-1643::DEBUG::2013-04-27 
14:23:08,048::task::588::TaskManager.Task::(_updateState) 
Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::moving from state aborting 
-> state failed
Thread-1643::DEBUG::2013-04-27 
14:23:08,049::resourceManager::806::ResourceManager.Owner::(releaseAll) 
Owner.releaseAll requests {} resources {}
Thread-1643::DEBUG::2013-04-27 
14:23:08,049::resourceManager::841::ResourceManager.Owner::(cancelAll) 
Owner.cancelAll requests {}
Thread-1643::ERROR::2013-04-27 
14:23:08,050::dispatcher::90::Storage.Dispatcher.Protect::(run) 
{'status': {'message': "Cannot find master domain: 
'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, 
msdUUID=025062dd-d69e-40ac-9031-c080b51647ae'", 'code': 304}}
Thread-1688::DEBUG::2013-04-27 
14:23:14,955::task::588::TaskManager.Task::(_updateState) 
Task=`5a33d6bc-3dba-4dea-8423-e242fa3809d3`::moving from state init -> 
state preparing
Thread-1688::INFO::2013-04-27 
14:23:14,956::logUtils::37::dispatcher::(wrapper) Run and protect: 
repoStats(options=None)
Thread-1688::INFO::2013-04-27 
14:23:14,956::logUtils::39::dispatcher::(wrapper) Run and protect: 
repoStats, Return response: {}
Thread-1688::DEBUG::2013-04-27 
14:23:14,957::task::1174::TaskManager.Task::(prepare) 
Task=`5a33d6bc-3dba-4dea-8423-e242fa3809d3`::finished: {}
Thread-1688::DEBUG::2013-04-27 
14:23:14,957::task::588::TaskManager.Task::(_updateState) 
Task=`5a33d6bc-3dba-4dea-8423-e242fa3809d3`::moving from state preparing 
-> state finished
Thread-1688::DEBUG::2013-04-27 
14:23:14,958::resourceManager::806::ResourceManager.Owner::(releaseAll) 
Owner.releaseAll requests {} resources {}
Thread-1688::DEBUG::2013-04-27 
14:23:14,958::resourceManager::841::ResourceManager.Owner::(cancelAll) 
Owner.cancelAll requests {}
Thread-1688::DEBUG::2013-04-27 
14:23:14,959::task::980::TaskManager.Task::(_decref) 
Task=`5a33d6bc-3dba-4dea-8423-e242fa3809d3`::ref 0 aborting False
Thread-1691::DEBUG::2013-04-27 
14:23:18,148::clientIF::261::Storage.Dispatcher.Protect::(wrapper) 
[10.0.0.154]
Thread-1691::DEBUG::2013-04-27 
14:23:18,149::task::588::TaskManager.Task::(_updateState) 
Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::moving from state init -> 
state preparing
Thread-1691::INFO::2013-04-27 
14:23:18,149::logUtils::37::dispatcher::(wrapper) Run and protect: 
getSpmStatus(spUUID='2b633f9a-8840-11e1-9ed7-17e6c491db84', options=None)
Thread-1691::ERROR::2013-04-27 
14:23:18,150::task::855::TaskManager.Task::(_setError) 
Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::Unexpected error
Traceback (most recent call last):
   File "/usr/share/vdsm/storage/task.py", line 863, in _run
     return fn(*args, **kargs)
   File "/usr/share/vdsm/logUtils.py", line 38, in wrapper
     res = f(*args, **kwargs)
   File "/usr/share/vdsm/storage/hsm.py", line 438, in getSpmStatus
     pool = self.getPool(spUUID)
   File "/usr/share/vdsm/storage/hsm.py", line 186, in getPool
     raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: 
('2b633f9a-8840-11e1-9ed7-17e6c491db84',)
Thread-1691::DEBUG::2013-04-27 
14:23:18,151::task::874::TaskManager.Task::(_run) 
Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::Task._run: 
a0eebdaf-3a1c-4c6e-acc2-c41def4ec963 
('2b633f9a-8840-11e1-9ed7-17e6c491db84',) {} failed - stopping task
Thread-1691::DEBUG::2013-04-27 
14:23:18,151::task::1201::TaskManager.Task::(stop) 
Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::stopping in state preparing 
(force False)
Thread-1691::DEBUG::2013-04-27 
14:23:18,152::task::980::TaskManager.Task::(_decref) 
Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::ref 1 aborting True
Thread-1691::INFO::2013-04-27 
14:23:18,152::task::1159::TaskManager.Task::(prepare) 
Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::aborting: Task is aborted: 
'Unknown pool id, pool not connected' - code 309
Thread-1691::DEBUG::2013-04-27 
14:23:18,153::task::1164::TaskManager.Task::(prepare) 
Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::Prepare: aborted: Unknown 
pool id, pool not connected
Thread-1691::DEBUG::2013-04-27 
14:23:18,153::task::980::TaskManager.Task::(_decref) 
Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::ref 0 aborting True
Thread-1691::DEBUG::2013-04-27 
14:23:18,154::task::915::TaskManager.Task::(_doAbort) 
Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::Task._doAbort: force False
Thread-1691::DEBUG::2013-04-27 
14:23:18,154::resourceManager::841::ResourceManager.Owner::(cancelAll) 
Owner.cancelAll requests {}
Thread-1691::DEBUG::2013-04-27 
14:23:18,155::task::588::TaskManager.Task::(_updateState) 
Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::moving from state preparing 
-> state aborting
Thread-1691::DEBUG::2013-04-27 
14:23:18,155::task::537::TaskManager.Task::(__state_aborting) 
Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::_aborting: recover policy none
Thread-1691::DEBUG::2013-04-27 
14:23:18,156::task::588::TaskManager.Task::(_updateState) 
Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::moving from state aborting 
-> state failed
Thread-1691::DEBUG::2013-04-27 
14:23:18,156::resourceManager::806::ResourceManager.Owner::(releaseAll) 
Owner.releaseAll requests {} resources {}
Thread-1691::DEBUG::2013-04-27 
14:23:18,157::resourceManager::841::ResourceManager.Owner::(cancelAll) 
Owner.cancelAll requests {}
Thread-1691::ERROR::2013-04-27 
14:23:18,158::dispatcher::90::Storage.Dispatcher.Protect::(run) 
{'status': {'message': "Unknown pool id, pool not connected: 
('2b633f9a-8840-11e1-9ed7-17e6c491db84',)", 'code': 309}}
Thread-1692::DEBUG::2013-04-27 
14:23:18,180::clientIF::261::Storage.Dispatcher.Protect::(wrapper) 
[10.0.0.154]
Thread-1692::DEBUG::2013-04-27 
14:23:18,181::task::588::TaskManager.Task::(_updateState) 
Task=`e144930f-11a4-4be3-b7ba-5ff22981e7e0`::moving from state init -> 
state preparing
Thread-1692::INFO::2013-04-27 
14:23:18,182::logUtils::37::dispatcher::(wrapper) Run and protect: 
connectStoragePool(spUUID='2b633f9a-8840-11e1-9ed7-17e6c491db84', 
hostID=1, scsiKey='2b633f9a-8840-11e1-9ed7-17e6c491db84', 
msdUUID='025062dd-d69e-40ac-9031-c080b51647ae', masterVersion=1, 
options=None)
Thread-1692::DEBUG::2013-04-27 
14:23:18,183::resourceManager::175::ResourceManager.Request::(__init__) 
ResName=`Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84`ReqID=`1e46c60e-7d09-4bcd-bcd6-4f20118a3f84`::Request 
was made in '/usr/share/vdsm/storage/hsm.py' line '747' at 
'_connectStoragePool'
Thread-1692::DEBUG::2013-04-27 
14:23:18,183::resourceManager::483::ResourceManager::(registerResource) 
Trying to register resource 
'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84' for lock type 'exclusive'
Thread-1692::DEBUG::2013-04-27 
14:23:18,184::resourceManager::525::ResourceManager::(registerResource) 
Resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84' is free. Now 
locking as 'exclusive' (1 active user)
Thread-1692::DEBUG::2013-04-27 
14:23:18,184::resourceManager::212::ResourceManager.Request::(grant) 
ResName=`Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84`ReqID=`1e46c60e-7d09-4bcd-bcd6-4f20118a3f84`::Granted 
request
Thread-1692::INFO::2013-04-27 
14:23:18,185::sp::608::Storage.StoragePool::(connect) Connect host #1 to 
the storage pool 2b633f9a-8840-11e1-9ed7-17e6c491db84 with master 
domain: 025062dd-d69e-40ac-9031-c080b51647ae (ver = 1)
Thread-1692::DEBUG::2013-04-27 
14:23:18,186::lvm::460::OperationMutex::(_invalidateAllPvs) Operation 
'lvm invalidate operation' got the operation mutex
Thread-1692::DEBUG::2013-04-27 
14:23:18,187::lvm::462::OperationMutex::(_invalidateAllPvs) Operation 
'lvm invalidate operation' released the operation mutex
Thread-1692::DEBUG::2013-04-27 
14:23:18,188::lvm::472::OperationMutex::(_invalidateAllVgs) Operation 
'lvm invalidate operation' got the operation mutex
Thread-1692::DEBUG::2013-04-27 
14:23:18,188::lvm::474::OperationMutex::(_invalidateAllVgs) Operation 
'lvm invalidate operation' released the operation mutex
Thread-1692::DEBUG::2013-04-27 
14:23:18,189::lvm::493::OperationMutex::(_invalidateAllLvs) Operation 
'lvm invalidate operation' got the operation mutex
Thread-1692::DEBUG::2013-04-27 
14:23:18,190::lvm::495::OperationMutex::(_invalidateAllLvs) Operation 
'lvm invalidate operation' released the operation mutex
Thread-1692::DEBUG::2013-04-27 
14:23:18,190::misc::1017::SamplingMethod::(__call__) Trying to enter 
sampling method (storage.sdc.refreshStorage)
Thread-1692::DEBUG::2013-04-27 
14:23:18,191::misc::1019::SamplingMethod::(__call__) Got in to sampling 
method
Thread-1692::DEBUG::2013-04-27 
14:23:18,191::misc::1017::SamplingMethod::(__call__) Trying to enter 
sampling method (storage.iscsi.rescan)
Thread-1692::DEBUG::2013-04-27 
14:23:18,192::misc::1019::SamplingMethod::(__call__) Got in to sampling 
method
Thread-1692::DEBUG::2013-04-27 
14:23:18,192::iscsi::389::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo 
-n /sbin/iscsiadm -m session -R' (cwd None)
Thread-1692::DEBUG::2013-04-27 
14:23:18,215::iscsi::389::Storage.Misc.excCmd::(rescan) FAILED: <err> = 
'iscsiadm: No session found.\n'; <rc> = 21
Thread-1692::DEBUG::2013-04-27 
14:23:18,216::misc::1027::SamplingMethod::(__call__) Returning last result
Thread-1692::DEBUG::2013-04-27 
14:23:19,820::multipath::71::Storage.Misc.excCmd::(rescan) 
'/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-1692::DEBUG::2013-04-27 
14:23:19,849::multipath::71::Storage.Misc.excCmd::(rescan) SUCCESS: 
<err> = ''; <rc> = 0
Thread-1692::DEBUG::2013-04-27 
14:23:19,850::lvm::460::OperationMutex::(_invalidateAllPvs) Operation 
'lvm invalidate operation' got the operation mutex
Thread-1692::DEBUG::2013-04-27 
14:23:19,850::lvm::462::OperationMutex::(_invalidateAllPvs) Operation 
'lvm invalidate operation' released the operation mutex
Thread-1692::DEBUG::2013-04-27 
14:23:19,851::lvm::472::OperationMutex::(_invalidateAllVgs) Operation 
'lvm invalidate operation' got the operation mutex
Thread-1692::DEBUG::2013-04-27 
14:23:19,852::lvm::474::OperationMutex::(_invalidateAllVgs) Operation 
'lvm invalidate operation' released the operation mutex
Thread-1692::DEBUG::2013-04-27 
14:23:19,852::lvm::493::OperationMutex::(_invalidateAllLvs) Operation 
'lvm invalidate operation' got the operation mutex
Thread-1692::DEBUG::2013-04-27 
14:23:19,853::lvm::495::OperationMutex::(_invalidateAllLvs) Operation 
'lvm invalidate operation' released the operation mutex
Thread-1692::DEBUG::2013-04-27 
14:23:19,853::misc::1027::SamplingMethod::(__call__) Returning last result
Thread-1692::DEBUG::2013-04-27 
14:23:19,854::lvm::352::OperationMutex::(_reloadvgs) Operation 'lvm 
reload operation' got the operation mutex
Thread-1692::DEBUG::2013-04-27 
14:23:19,857::lvm::287::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n 
/sbin/lvm vgs --config " devices { preferred_names = 
[\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 
disable_after_error_count=3 filter = [ 
\\"a%36782bcb063f75600162a25c119f14819%\\", \\"r%.*%\\" ] } global {  
locking_type=1  prioritise_write_locks=1 wait_for_locks=1 } backup {  
retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix 
--separator | -o 
uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 
025062dd-d69e-40ac-9031-c080b51647ae' (cwd None)
Thread-1692::DEBUG::2013-04-27 
14:23:19,973::lvm::287::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  
Volume group "025062dd-d69e-40ac-9031-c080b51647ae" not found\n'; <rc> = 5
Thread-1692::WARNING::2013-04-27 
14:23:19,975::lvm::356::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] 
['  Volume group "025062dd-d69e-40ac-9031-c080b51647ae" not found']
Thread-1692::DEBUG::2013-04-27 
14:23:19,976::lvm::379::OperationMutex::(_reloadvgs) Operation 'lvm 
reload operation' released the operation mutex
Thread-1692::DEBUG::2013-04-27 
14:23:19,996::fileSD::98::Storage.StorageDomain::(__init__) Reading 
domain in path 
/rhev/data-center/mnt/10.0.0.154:_home_vm_data_data/025062dd-d69e-40ac-9031-c080b51647ae
Thread-1692::DEBUG::2013-04-27 
14:23:19,998::persistentDict::175::Storage.PersistentDict::(__init__) 
Created a persistant dict with FileMetadataRW backend
Thread-1699::DEBUG::2013-04-27 
14:23:25,106::task::588::TaskManager.Task::(_updateState) 
Task=`3289175f-e99f-4625-9ee7-90b3f6cce1df`::moving from state init -> 
state preparing
Thread-1699::INFO::2013-04-27 
14:23:25,107::logUtils::37::dispatcher::(wrapper) Run and protect: 
repoStats(options=None)
Thread-1699::INFO::2013-04-27 
14:23:25,107::logUtils::39::dispatcher::(wrapper) Run and protect: 
repoStats, Return response: {}
Thread-1699::DEBUG::2013-04-27 
14:23:25,108::task::1174::TaskManager.Task::(prepare) 
Task=`3289175f-e99f-4625-9ee7-90b3f6cce1df`::finished: {}
Thread-1699::DEBUG::2013-04-27 
14:23:25,108::task::588::TaskManager.Task::(_updateState) 
Task=`3289175f-e99f-4625-9ee7-90b3f6cce1df`::moving from state preparing 
-> state finished
Thread-1699::DEBUG::2013-04-27 
14:23:25,109::resourceManager::806::ResourceManager.Owner::(releaseAll) 
Owner.releaseAll requests {} resources {}
Thread-1699::DEBUG::2013-04-27 
14:23:25,109::resourceManager::841::ResourceManager.Owner::(cancelAll) 
Owner.cancelAll requests {}
Thread-1699::DEBUG::2013-04-27 
14:23:25,110::task::980::TaskManager.Task::(_decref) 
Task=`3289175f-e99f-4625-9ee7-90b3f6cce1df`::ref 0 aborting False
Thread-1705::DEBUG::2013-04-27 
14:23:33,970::clientIF::261::Storage.Dispatcher.Protect::(wrapper) 
[10.0.0.154]
Thread-1705::DEBUG::2013-04-27 
14:23:33,971::task::588::TaskManager.Task::(_updateState) 
Task=`a421e0e3-3c83-43c0-a867-5f76f34e7619`::moving from state init -> 
state preparing
Thread-1705::INFO::2013-04-27 
14:23:33,972::logUtils::37::dispatcher::(wrapper) Run and protect: 
validateStorageServerConnection(domType=1, 
spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 
'10.0.0.154:/home/vm/data/data', 'iqn': '', 'portal': '', 'user': '', 
'password': '******', 'id': '60e73855-daa2-45f5-9c59-5973ca89e830', 
'port': ''}], options=None)
Thread-1705::INFO::2013-04-27 
14:23:33,972::storage_connection::188::Storage.ServerConnection::(validate) 
Request to validate NFS storage server
Thread-1705::DEBUG::2013-04-27 
14:23:33,973::mount::111::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo 
-n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6 
10.0.0.154:/home/vm/data/data /tmp/tmp6s__VT' (cwd None)
Thread-1705::DEBUG::2013-04-27 
14:23:34,034::mount::111::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo 
-n /bin/umount /tmp/tmp6s__VT' (cwd None)
Thread-1705::INFO::2013-04-27 
14:23:34,084::logUtils::39::dispatcher::(wrapper) Run and protect: 
validateStorageServerConnection, Return response: {'statuslist': 
[{'status': 0, 'id': '60e73855-daa2-45f5-9c59-5973ca89e830'}]}
Thread-1705::DEBUG::2013-04-27 
14:23:34,085::task::1174::TaskManager.Task::(prepare) 
Task=`a421e0e3-3c83-43c0-a867-5f76f34e7619`::finished: {'statuslist': 
[{'status': 0, 'id': '60e73855-daa2-45f5-9c59-5973ca89e830'}]}
Thread-1705::DEBUG::2013-04-27 
14:23:34,085::task::588::TaskManager.Task::(_updateState) 
Task=`a421e0e3-3c83-43c0-a867-5f76f34e7619`::moving from state preparing 
-> state finished
Thread-1705::DEBUG::2013-04-27 
14:23:34,086::resourceManager::806::ResourceManager.Owner::(releaseAll) 
Owner.releaseAll requests {} resources {}
Thread-1705::DEBUG::2013-04-27 
14:23:34,086::resourceManager::841::ResourceManager.Owner::(cancelAll) 
Owner.cancelAll requests {}
Thread-1705::DEBUG::2013-04-27 
14:23:34,087::task::980::TaskManager.Task::(_decref) 
Task=`a421e0e3-3c83-43c0-a867-5f76f34e7619`::ref 0 aborting False
Thread-1706::DEBUG::2013-04-27 
14:23:34,110::clientIF::261::Storage.Dispatcher.Protect::(wrapper) 
[10.0.0.154]
Thread-1706::DEBUG::2013-04-27 
14:23:34,111::task::588::TaskManager.Task::(_updateState) 
Task=`abbdddbc-3fa2-4799-b5d4-f902cd3a7ac7`::moving from state init -> 
state preparing
Thread-1706::INFO::2013-04-27 
14:23:34,111::logUtils::37::dispatcher::(wrapper) Run and protect: 
connectStorageServer(domType=1, 
spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 
'10.0.0.154:/home/vm/data/data', 'iqn': '', 'portal': '', 'user': '', 
'password': '******', 'id': '60e73855-daa2-45f5-9c59-5973ca89e830', 
'port': ''}], options=None)
Thread-1706::INFO::2013-04-27 
14:23:34,112::storage_connection::146::Storage.ServerConnection::(connect) 
Request to connect NFS storage server
Thread-1706::DEBUG::2013-04-27 
14:23:34,123::lvm::460::OperationMutex::(_invalidateAllPvs) Operation 
'lvm invalidate operation' got the operation mutex
Thread-1706::DEBUG::2013-04-27 
14:23:34,124::lvm::462::OperationMutex::(_invalidateAllPvs) Operation 
'lvm invalidate operation' released the operation mutex
Thread-1706::DEBUG::2013-04-27 
14:23:34,125::lvm::472::OperationMutex::(_invalidateAllVgs) Operation 
'lvm invalidate operation' got the operation mutex
Thread-1706::DEBUG::2013-04-27 
14:23:34,125::lvm::474::OperationMutex::(_invalidateAllVgs) Operation 
'lvm invalidate operation' released the operation mutex
Thread-1706::DEBUG::2013-04-27 
14:23:34,126::lvm::493::OperationMutex::(_invalidateAllLvs) Operation 
'lvm invalidate operation' got the operation mutex
Thread-1706::DEBUG::2013-04-27 
14:23:34,127::lvm::495::OperationMutex::(_invalidateAllLvs) Operation 
'lvm invalidate operation' released the operation mutex
Thread-1706::INFO::2013-04-27 
14:23:34,127::logUtils::39::dispatcher::(wrapper) Run and protect: 
connectStorageServer, Return response: {'statuslist': [{'status': 0, 
'id': '60e73855-daa2-45f5-9c59-5973ca89e830'}]}
Thread-1706::DEBUG::2013-04-27 
14:23:34,128::task::1174::TaskManager.Task::(prepare) 
Task=`abbdddbc-3fa2-4799-b5d4-f902cd3a7ac7`::finished: {'statuslist': 
[{'status': 0, 'id': '60e73855-daa2-45f5-9c59-5973ca89e830'}]}
Thread-1706::DEBUG::2013-04-27 
14:23:34,129::task::588::TaskManager.Task::(_updateState) 
Task=`abbdddbc-3fa2-4799-b5d4-f902cd3a7ac7`::moving from state preparing 
-> state finished
Thread-1706::DEBUG::2013-04-27 
14:23:34,129::resourceManager::806::ResourceManager.Owner::(releaseAll) 
Owner.releaseAll requests {} resources {}
Thread-1706::DEBUG::2013-04-27 
14:23:34,130::resourceManager::841::ResourceManager.Owner::(cancelAll) 
Owner.cancelAll requests {}
Thread-1706::DEBUG::2013-04-27 
14:23:34,130::task::980::TaskManager.Task::(_decref) 
Task=`abbdddbc-3fa2-4799-b5d4-f902cd3a7ac7`::ref 0 aborting False
Thread-1707::DEBUG::2013-04-27 
14:23:35,251::task::588::TaskManager.Task::(_updateState) 
Task=`f6026160-a4cf-48a1-9f08-2c9da1cd6184`::moving from state init -> 
state preparing
Thread-1707::INFO::2013-04-27 
14:23:35,252::logUtils::37::dispatcher::(wrapper) Run and protect: 
repoStats(options=None)
Thread-1707::INFO::2013-04-27 
14:23:35,252::logUtils::39::dispatcher::(wrapper) Run and protect: 
repoStats, Return response: {}
Thread-1707::DEBUG::2013-04-27 
14:23:35,253::task::1174::TaskManager.Task::(prepare) 
Task=`f6026160-a4cf-48a1-9f08-2c9da1cd6184`::finished: {}
Thread-1707::DEBUG::2013-04-27 
14:23:35,253::task::588::TaskManager.Task::(_updateState) 
Task=`f6026160-a4cf-48a1-9f08-2c9da1cd6184`::moving from state preparing 
-> state finished
Thread-1707::DEBUG::2013-04-27 
14:23:35,254::resourceManager::806::ResourceManager.Owner::(releaseAll) 
Owner.releaseAll requests {} resources {}
------------------------------------------------------------------