10:08 p.m.
--------------090008040302090307060003
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: 7bit
My Data Center is down, and won't come back up.
Data Center Status on the GUI flips between "Non Responsive" and
"Contending"
Also noted:
Host sometimes seen flipping between "Low" and "Contending" in SPM
column.
Storage VM2 "Data (Master)" is in "Cross Data-Center Status" =
Unknown
VM2 is "up" under "Volumes" tab
Created another volume for VM storage. It shows up in "volumes" tab, but
when I try to add "New Domain" in storage tab, says that "There are No Data
Centers to which the Storage Domain can be attached"
Setup:
2 hosts w/ glusterfs storage
1 engine
all 3 computers Centos 6.5, just updated
ovirt-engine 3.3.0.1-1.el6
ovirt-engine-lib 3.3.2-1.el6
ovirt-host-deploy.noarch 1.1.3-1.el6
glusterfs.x86_64 3.4.2-1.el6
This loop seems to repeat in the ovirt-engine log (grep of log showing only
DefaultQuartzScheduler_Worker-79 thread:
2014-01-26 14:44:58,416 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-79) Irs placed on server
9a591103-83be-4ca9-b207-06929223b541 failed. Proceed Failover
2014-01-26 14:44:58,511 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-79) hostFromVds::selectedVds - office4a,
spmStatus Free, storage pool mill
2014-01-26 14:44:58,550 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-79) SpmStatus on vds
127ed939-34af-41a8-87a0-e2f6174b1877: Free
2014-01-26 14:44:58,571 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-79) starting spm on vds office4a, storage pool
mill, prevId 2, LVER 15
2014-01-26 14:44:58,579 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
(DefaultQuartzScheduler_Worker-79) START, SpmStartVDSCommand(HostName =
office4a, HostId = 127ed939-34af-41a8-87a0-e2f6174b1877, storagePoolId =
536a864d-83aa-473a-a675-e38aafdd9071, prevId=2, prevLVER=15,
storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id:
74c38eb7
2014-01-26 14:44:58,617 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
(DefaultQuartzScheduler_Worker-79) spmStart polling started: taskId =
e8986753-fc80-4b11-a11d-6d3470b1728c
2014-01-26 14:45:00,662 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand]
(DefaultQuartzScheduler_Worker-79) Failed in HSMGetTaskStatusVDS method
2014-01-26 14:45:00,664 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand]
(DefaultQuartzScheduler_Worker-79) Error code AcquireHostIdFailure and error
message VDSGenericException: VDSErrorException: Failed to
HSMGetTaskStatusVDS, error = Cannot acquire host id
2014-01-26 14:45:00,665 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
(DefaultQuartzScheduler_Worker-79) spmStart polling ended: taskId =
e8986753-fc80-4b11-a11d-6d3470b1728c task status = finished
2014-01-26 14:45:00,666 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
(DefaultQuartzScheduler_Worker-79) Start SPM Task failed - result:
cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to
HSMGetTaskStatusVDS, error = Cannot acquire host id
2014-01-26 14:45:00,695 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
(DefaultQuartzScheduler_Worker-79) spmStart polling ended, spm status: Free
2014-01-26 14:45:00,702 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand]
(DefaultQuartzScheduler_Worker-79) START, HSMClearTaskVDSCommand(HostName =
office4a, HostId = 127ed939-34af-41a8-87a0-e2f6174b1877,
taskId=e8986753-fc80-4b11-a11d-6d3470b1728c), log id: 336ec5a6
2014-01-26 14:45:00,722 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand]
(DefaultQuartzScheduler_Worker-79) FINISH, HSMClearTaskVDSCommand, log id:
336ec5a6
2014-01-26 14:45:00,724 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
(DefaultQuartzScheduler_Worker-79) FINISH, SpmStartVDSCommand, return:
org.ovirt.engine.core.common.businessentities.SpmStatusResult@13652652, log
id: 74c38eb7
2014-01-26 14:45:00,733 INFO
[org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand]
(DefaultQuartzScheduler_Worker-79) Running command:
SetStoragePoolStatusCommand internal: true. Entities affected : ID:
536a864d-83aa-473a-a675-e38aafdd9071 Type: StoragePool
2014-01-26 14:45:00,778 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-79) IrsBroker::Failed::GetStoragePoolInfoVDS
due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException:
SpmStart failed
Ted Miller
Elkhart, IN, USA
--------------090008040302090307060003
Content-Type: text/html; charset="ISO-8859-1"
Content-Transfer-Encoding: 7bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
</head>
<body text="#000000" bgcolor="#FFFFFF">
My Data Center is down, and won't come back up.<br>
<br>
Data Center Status on the GUI flips between "Non Responsive" and
"Contending"<br>
<br>
Also noted:<br>
Host sometimes seen flipping between "Low" and "Contending" in
SPM
column.<br>
Storage VM2 "Data (Master)" is in "Cross Data-Center Status" =
Unknown<br>
VM2 is "up" under "Volumes" tab<br>
<br>
Created another volume for VM storage. It shows up in "volumes"
tab, but when I try to add "New Domain" in storage tab, says that
"There are No Data Centers to which the Storage Domain can be
attached"<br>
<br>
Setup: <br>
2 hosts w/ glusterfs storage<br>
1 engine<br>
all 3 computers Centos 6.5, just updated<br>
ovirt-engine
3.3.0.1-1.el6<br>
ovirt-engine-lib
3.3.2-1.el6<br>
ovirt-host-deploy.noarch 1.1.3-1.el6<br>
glusterfs.x86_64
3.4.2-1.el6<br>
<br>
This loop seems to repeat in the ovirt-engine log (grep of log
showing only <font face="Courier New, Courier,
monospace">DefaultQuartzScheduler_Worker-79</font>
thread:<br>
<br>
<font face="Courier New, Courier, monospace">2014-01-26 14:44:58,416
INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-79) Irs placed on server
9a591103-83be-4ca9-b207-06929223b541 failed. Proceed Failover<br>
2014-01-26 14:44:58,511 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-79) hostFromVds::selectedVds -
office4a, spmStatus Free, storage pool mill<br>
2014-01-26 14:44:58,550 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-79) SpmStatus on vds
127ed939-34af-41a8-87a0-e2f6174b1877: Free<br>
2014-01-26 14:44:58,571 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-79) starting spm on vds office4a,
storage pool mill, prevId 2, LVER 15<br>
2014-01-26 14:44:58,579 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
(DefaultQuartzScheduler_Worker-79) START,
SpmStartVDSCommand(HostName = office4a, HostId =
127ed939-34af-41a8-87a0-e2f6174b1877, storagePoolId =
536a864d-83aa-473a-a675-e38aafdd9071, prevId=2, prevLVER=15,
storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false),
log id: 74c38eb7<br>
2014-01-26 14:44:58,617 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
(DefaultQuartzScheduler_Worker-79) spmStart polling started:
taskId = e8986753-fc80-4b11-a11d-6d3470b1728c<br>
2014-01-26 14:45:00,662 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand]
(DefaultQuartzScheduler_Worker-79) Failed in HSMGetTaskStatusVDS
method<br>
2014-01-26 14:45:00,664 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand]
(DefaultQuartzScheduler_Worker-79) Error code AcquireHostIdFailure
and error message VDSGenericException: VDSErrorException: Failed
to HSMGetTaskStatusVDS, error = Cannot acquire host id<br>
2014-01-26 14:45:00,665 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
(DefaultQuartzScheduler_Worker-79) spmStart polling ended: taskId
= e8986753-fc80-4b11-a11d-6d3470b1728c task status = finished<br>
2014-01-26 14:45:00,666 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
(DefaultQuartzScheduler_Worker-79) Start SPM Task failed - result:
cleanSuccess, message: VDSGenericException: VDSErrorException:
Failed to HSMGetTaskStatusVDS, error = Cannot acquire host id<br>
2014-01-26 14:45:00,695 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
(DefaultQuartzScheduler_Worker-79) spmStart polling ended, spm
status: Free<br>
2014-01-26 14:45:00,702 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand]
(DefaultQuartzScheduler_Worker-79) START,
HSMClearTaskVDSCommand(HostName = office4a, HostId =
127ed939-34af-41a8-87a0-e2f6174b1877,
taskId=e8986753-fc80-4b11-a11d-6d3470b1728c), log id: 336ec5a6<br>
2014-01-26 14:45:00,722 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand]
(DefaultQuartzScheduler_Worker-79) FINISH, HSMClearTaskVDSCommand,
log id: 336ec5a6<br>
2014-01-26 14:45:00,724 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
(DefaultQuartzScheduler_Worker-79) FINISH, SpmStartVDSCommand,
return:
org.ovirt.engine.core.common.businessentities.SpmStatusResult@13652652,
log id: 74c38eb7<br>
2014-01-26 14:45:00,733 INFO
[org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand]
(DefaultQuartzScheduler_Worker-79) Running command:
SetStoragePoolStatusCommand internal: true. Entities affected :
ID: 536a864d-83aa-473a-a675-e38aafdd9071 Type: StoragePool<br>
2014-01-26 14:45:00,778 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-79)
IrsBroker::Failed::GetStoragePoolInfoVDS due to:
IrsSpmStartFailedException: IRSGenericException:
IRSErrorException: SpmStart failed<br>
</font><br>
Ted Miller<br>
Elkhart, IN, USA<br>
<br>
</body>
</html>
--------------090008040302090307060003--