[Users] SPM is always contending

[Andy Singleton]

After the node acting as spm was (accidentally) put into maintenance 
mode, the spm role is always contending. The original node doesnt take 
the role back either.
About 10 instances were still running on the node when it was put into 
maintenance.

I have ovirt 3.1.0-4 with a targetcli iscsi store.

Here are the contents of the engine.log.

Andy

2013-04-04 15:25:23,979 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] 
(QuartzScheduler_Worker-96) [5449512a] spmStart polling ended. spm status:
Free
2013-04-04 15:25:23,993 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] 
(QuartzScheduler_Worker-96) [5449512a] START, HSMClearTaskVDSCommand(vd
sId = d265245e-3cc5-11e2-bce7-001018fc3b14, 
taskId=7c1cf7c6-21d4-4504-882d-3d465742b6ef), log id: d2da8ba
2013-04-04 15:25:24,011 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] 
(QuartzScheduler_Worker-96) [5449512a] FINISH, HSMClearTaskVDSCommand,
log id: d2da8ba
2013-04-04 15:25:24,012 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] 
(QuartzScheduler_Worker-96) [5449512a] FINISH, SpmStartVDSCommand, return:
org.ovirt.engine.core.common.businessentities.SpmStatusResult at 30a05218, 
log id: 5980d6f4
2013-04-04 15:25:24,014 INFO 
[org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] 
(QuartzScheduler_Worker-96) [6d6dbe98] Running command: SetStoragePoolStat
usCommand internal: true. Entities affected :  ID: 
0ba15357-9b3b-4a76-8dac-b2b66b922174 Type: StoragePool
2013-04-04 15:25:24,843 ERROR 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-96) [6d6dbe98] 
IrsBroker::Failed::GetStoragePoolInfoV
DS due to: IrsSpmStartFailedException: IRSGenericException: 
IRSErrorException: SpmStart failed
2013-04-04 15:25:24,900 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-96) [6d6dbe98] Irs placed on server null failed. 
Proc
eed Failover
2013-04-04 15:25:34,977 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-96) [6d6dbe98] hostFromVds::selectedVds - 
moon-palace
, spmStatus Free, storage pool Primary
2013-04-04 15:25:35,806 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(QuartzScheduler_Worker-96) [6d6dbe98] starting spm on vds moon-palace, 
stora
ge pool Primary, prevId -1, LVER 2520
2013-04-04 15:25:35,851 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] 
(QuartzScheduler_Worker-96) [6d6dbe98] START, SpmStartVDSCommand(vdsId = 3d
88c8b0-84bc-11e2-96b1-001018fc3b14, storagePoolId = 
0ba15357-9b3b-4a76-8dac-b2b66b922174, prevId=-1, prevLVER=2520, 
storagePoolFormatType=V2, recoveryMode=Manual, SCSIF
encing=false), log id: 30f4f64e
2013-04-04 15:25:45,901 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] 
(QuartzScheduler_Worker-96) [6d6dbe98] spmStart polling started: taskId = 6
8002cff-478d-4c2c-8839-b935d8a858b1
2013-04-04 15:26:11,903 INFO [org.ovirt.engine.core.bll.VdsLoadBalancer] 
(QuartzScheduler_Worker-81) [45db3aaa] VdsLoadBalancer: Starting load 
balance for cluster: BP_
Primary, algorithm: EvenlyDistribute.
2013-04-04 15:26:12,807 INFO [org.ovirt.engine.core.bll.VdsLoadBalancer] 
(QuartzScheduler_Worker-81) [45db3aaa] VdsLoadBalancer: high util: 51, 
low util: 0, duration:
2, threashold: 80
2013-04-04 15:26:12,845 INFO 
[org.ovirt.engine.core.bll.VdsLoadBalancingAlgorithm] 
(QuartzScheduler_Worker-81) [45db3aaa] VdsLoadBalancer: number of 
relevant vdss (no
migration, no pending): 3.
2013-04-04 15:26:12,846 INFO 
[org.ovirt.engine.core.bll.VdsCpuVdsLoadBalancingAlgorithm] 
(QuartzScheduler_Worker-81) [45db3aaa] VdsLoadBalancer: number of over 
utilize
d vdss found: 0.
2013-04-04 15:26:12,846 INFO 
[org.ovirt.engine.core.bll.VdsCpuVdsLoadBalancingAlgorithm] 
(QuartzScheduler_Worker-81) [45db3aaa] VdsLoadBalancer: max cpu limit: 
40, num
ber of ready to migration vdss: 3
2013-04-04 15:26:19,024 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] 
(QuartzScheduler_Worker-96) [6d6dbe98] spmStart polling ended: taskId = 680
02cff-478d-4c2c-8839-b935d8a858b1 task status = finished
2013-04-04 15:26:19,024 ERROR 
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] 
(QuartzScheduler_Worker-96) [6d6dbe98] Start SPM Task failed - result: clea
nSuccess, message: VDSGenericException: VDSErrorException: Failed in 
vdscommand to HSMGetTaskStatusVDS, error = BlockSD master file system 
FSCK error

This log sequence repeats indefinitely