[Users] Data Center stuck between "Non Responsive" and "Contending"

[Itamar Heim]

On 01/26/2014 10:08 PM, Ted Miller wrote:
> My Data Center is down, and won't come back up.
>
> Data Center Status on the GUI flips between "Non Responsive" and
> "Contending"
>
> Also noted:
> Host sometimes seen flipping between "Low" and "Contending" in SPM column.
> Storage VM2 "Data (Master)" is in "Cross Data-Center Status" = Unknown
> VM2 is "up" under "Volumes" tab
>
> Created another volume for VM storage.  It shows up in "volumes" tab,
> but when I try to add "New Domain" in storage tab, says that "There are
> No Data Centers to which the Storage Domain can be attached"
>
> Setup:
> 2 hosts w/ glusterfs storage
> 1 engine
> all 3 computers Centos 6.5, just updated
> ovirt-engine                       3.3.0.1-1.el6
> ovirt-engine-lib                 3.3.2-1.el6
> ovirt-host-deploy.noarch  1.1.3-1.el6
> glusterfs.x86_64               3.4.2-1.el6
>
> This loop seems to repeat in the ovirt-engine log (grep of log showing
> only DefaultQuartzScheduler_Worker-79 thread:
>
> 2014-01-26 14:44:58,416 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
> (DefaultQuartzScheduler_Worker-79) Irs placed on server
> 9a591103-83be-4ca9-b207-06929223b541 failed. Proceed Failover
> 2014-01-26 14:44:58,511 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
> (DefaultQuartzScheduler_Worker-79) hostFromVds::selectedVds - office4a,
> spmStatus Free, storage pool mill
> 2014-01-26 14:44:58,550 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
> (DefaultQuartzScheduler_Worker-79) SpmStatus on vds
> 127ed939-34af-41a8-87a0-e2f6174b1877: Free
> 2014-01-26 14:44:58,571 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
> (DefaultQuartzScheduler_Worker-79) starting spm on vds office4a, storage
> pool mill, prevId 2, LVER 15
> 2014-01-26 14:44:58,579 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
> (DefaultQuartzScheduler_Worker-79) START, SpmStartVDSCommand(HostName =
> office4a, HostId = 127ed939-34af-41a8-87a0-e2f6174b1877, storagePoolId =
> 536a864d-83aa-473a-a675-e38aafdd9071, prevId=2, prevLVER=15,
> storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log
> id: 74c38eb7
> 2014-01-26 14:44:58,617 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
> (DefaultQuartzScheduler_Worker-79) spmStart polling started: taskId =
> e8986753-fc80-4b11-a11d-6d3470b1728c
> 2014-01-26 14:45:00,662 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand]
> (DefaultQuartzScheduler_Worker-79) Failed in HSMGetTaskStatusVDS method
> 2014-01-26 14:45:00,664 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand]
> (DefaultQuartzScheduler_Worker-79) Error code AcquireHostIdFailure and
> error message VDSGenericException: VDSErrorException: Failed to
> HSMGetTaskStatusVDS, error = Cannot acquire host id
> 2014-01-26 14:45:00,665 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
> (DefaultQuartzScheduler_Worker-79) spmStart polling ended: taskId =
> e8986753-fc80-4b11-a11d-6d3470b1728c task status = finished
> 2014-01-26 14:45:00,666 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
> (DefaultQuartzScheduler_Worker-79) Start SPM Task failed - result:
> cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to
> HSMGetTaskStatusVDS, error = Cannot acquire host id
> 2014-01-26 14:45:00,695 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
> (DefaultQuartzScheduler_Worker-79) spmStart polling ended, spm status: Free
> 2014-01-26 14:45:00,702 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand]
> (DefaultQuartzScheduler_Worker-79) START,
> HSMClearTaskVDSCommand(HostName = office4a, HostId =
> 127ed939-34af-41a8-87a0-e2f6174b1877,
> taskId=e8986753-fc80-4b11-a11d-6d3470b1728c), log id: 336ec5a6
> 2014-01-26 14:45:00,722 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand]
> (DefaultQuartzScheduler_Worker-79) FINISH, HSMClearTaskVDSCommand, log
> id: 336ec5a6
> 2014-01-26 14:45:00,724 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
> (DefaultQuartzScheduler_Worker-79) FINISH, SpmStartVDSCommand, return:
> org.ovirt.engine.core.common.businessentities.SpmStatusResult at 13652652,
> log id: 74c38eb7
> 2014-01-26 14:45:00,733 INFO
> [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand]
> (DefaultQuartzScheduler_Worker-79) Running command:
> SetStoragePoolStatusCommand internal: true. Entities affected : ID:
> 536a864d-83aa-473a-a675-e38aafdd9071 Type: StoragePool
> 2014-01-26 14:45:00,778 ERROR
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
> (DefaultQuartzScheduler_Worker-79)
> IrsBroker::Failed::GetStoragePoolInfoVDS due to:
> IrsSpmStartFailedException: IRSGenericException: IRSErrorException:
> SpmStart failed
>
> Ted Miller
> Elkhart, IN, USA
>
>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>

is this gluster storage (guessing sunce you mentioned a 'volume')
does it have a quorum?
(there were reports of split brain on the domain metadata before when no 
quorum exist for gluster)