[Users] Data Center stuck between "Non Responsive" and "Contending"

[Ted Miller]

My Data Center is down, and won't come back up.

Data Center Status on the GUI flips between "Non Responsive" and "Contending"

Also noted:
Host sometimes seen flipping between "Low" and "Contending" in SPM column.
Storage VM2 "Data (Master)" is in "Cross Data-Center Status" = Unknown
VM2 is "up" under "Volumes" tab

Created another volume for VM storage.  It shows up in "volumes" tab, but 
when I try to add "New Domain" in storage tab, says that "There are No Data 
Centers to which the Storage Domain can be attached"

Setup:
2 hosts w/ glusterfs storage
1 engine
all 3 computers Centos 6.5, just updated
ovirt-engine                       3.3.0.1-1.el6
ovirt-engine-lib                 3.3.2-1.el6
ovirt-host-deploy.noarch  1.1.3-1.el6
glusterfs.x86_64               3.4.2-1.el6

This loop seems to repeat in the ovirt-engine log (grep of log showing only 
DefaultQuartzScheduler_Worker-79 thread:

2014-01-26 14:44:58,416 INFO  
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(DefaultQuartzScheduler_Worker-79) Irs placed on server 
9a591103-83be-4ca9-b207-06929223b541 failed. Proceed Failover
2014-01-26 14:44:58,511 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(DefaultQuartzScheduler_Worker-79) hostFromVds::selectedVds - office4a, 
spmStatus Free, storage pool mill
2014-01-26 14:44:58,550 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(DefaultQuartzScheduler_Worker-79) SpmStatus on vds 
127ed939-34af-41a8-87a0-e2f6174b1877: Free
2014-01-26 14:44:58,571 INFO 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(DefaultQuartzScheduler_Worker-79) starting spm on vds office4a, storage pool 
mill, prevId 2, LVER 15
2014-01-26 14:44:58,579 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] 
(DefaultQuartzScheduler_Worker-79) START, SpmStartVDSCommand(HostName = 
office4a, HostId = 127ed939-34af-41a8-87a0-e2f6174b1877, storagePoolId = 
536a864d-83aa-473a-a675-e38aafdd9071, prevId=2, prevLVER=15, 
storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 
74c38eb7
2014-01-26 14:44:58,617 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] 
(DefaultQuartzScheduler_Worker-79) spmStart polling started: taskId = 
e8986753-fc80-4b11-a11d-6d3470b1728c
2014-01-26 14:45:00,662 ERROR 
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] 
(DefaultQuartzScheduler_Worker-79) Failed in HSMGetTaskStatusVDS method
2014-01-26 14:45:00,664 ERROR 
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] 
(DefaultQuartzScheduler_Worker-79) Error code AcquireHostIdFailure and error 
message VDSGenericException: VDSErrorException: Failed to 
HSMGetTaskStatusVDS, error = Cannot acquire host id
2014-01-26 14:45:00,665 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] 
(DefaultQuartzScheduler_Worker-79) spmStart polling ended: taskId = 
e8986753-fc80-4b11-a11d-6d3470b1728c task status = finished
2014-01-26 14:45:00,666 ERROR 
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] 
(DefaultQuartzScheduler_Worker-79) Start SPM Task failed - result: 
cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to 
HSMGetTaskStatusVDS, error = Cannot acquire host id
2014-01-26 14:45:00,695 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] 
(DefaultQuartzScheduler_Worker-79) spmStart polling ended, spm status: Free
2014-01-26 14:45:00,702 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] 
(DefaultQuartzScheduler_Worker-79) START, HSMClearTaskVDSCommand(HostName = 
office4a, HostId = 127ed939-34af-41a8-87a0-e2f6174b1877, 
taskId=e8986753-fc80-4b11-a11d-6d3470b1728c), log id: 336ec5a6
2014-01-26 14:45:00,722 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] 
(DefaultQuartzScheduler_Worker-79) FINISH, HSMClearTaskVDSCommand, log id: 
336ec5a6
2014-01-26 14:45:00,724 INFO 
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] 
(DefaultQuartzScheduler_Worker-79) FINISH, SpmStartVDSCommand, return: 
org.ovirt.engine.core.common.businessentities.SpmStatusResult at 13652652, log 
id: 74c38eb7
2014-01-26 14:45:00,733 INFO 
[org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] 
(DefaultQuartzScheduler_Worker-79) Running command: 
SetStoragePoolStatusCommand internal: true. Entities affected : ID: 
536a864d-83aa-473a-a675-e38aafdd9071 Type: StoragePool
2014-01-26 14:45:00,778 ERROR 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] 
(DefaultQuartzScheduler_Worker-79) IrsBroker::Failed::GetStoragePoolInfoVDS 
due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: 
SpmStart failed

Ted Miller
Elkhart, IN, USA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20140126/87209dfb/attachment-0001.html>