<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html;
      charset=ISO-8859-1">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    My Data Center is down, and won't come back up.<br>
    <br>
    Data Center Status on the GUI flips between "Non Responsive" and
    "Contending"<br>
    <br>
    Also noted:<br>
    Host sometimes seen flipping between "Low" and "Contending" in SPM
    column.<br>
    Storage VM2 "Data (Master)" is in "Cross Data-Center Status" =
    Unknown<br>
    VM2 is "up" under "Volumes" tab<br>
    <br>
    Created another volume for VM storage.&nbsp; It shows up in "volumes"
    tab, but when I try to add "New Domain" in storage tab, says that
    "There are No Data Centers to which the Storage Domain can be
    attached"<br>
    <br>
    Setup: <br>
    2 hosts w/ glusterfs storage<br>
    1 engine<br>
    all 3 computers Centos 6.5, just updated<br>
    ovirt-engine&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3.3.0.1-1.el6<br>
    ovirt-engine-lib&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3.3.2-1.el6<br>
    ovirt-host-deploy.noarch&nbsp; 1.1.3-1.el6<br>
    glusterfs.x86_64&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3.4.2-1.el6<br>
    <br>
    This loop seems to repeat in the ovirt-engine log (grep of log
    showing only <font face="Courier New, Courier, monospace">DefaultQuartzScheduler_Worker-79</font>
    thread:<br>
    <br>
    <font face="Courier New, Courier, monospace">2014-01-26 14:44:58,416
      INFO&nbsp; [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
      (DefaultQuartzScheduler_Worker-79) Irs placed on server
      9a591103-83be-4ca9-b207-06929223b541 failed. Proceed Failover<br>
      2014-01-26 14:44:58,511 INFO&nbsp;
      [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
      (DefaultQuartzScheduler_Worker-79) hostFromVds::selectedVds -
      office4a, spmStatus Free, storage pool mill<br>
      2014-01-26 14:44:58,550 INFO&nbsp;
      [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
      (DefaultQuartzScheduler_Worker-79) SpmStatus on vds
      127ed939-34af-41a8-87a0-e2f6174b1877: Free<br>
      2014-01-26 14:44:58,571 INFO&nbsp;
      [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
      (DefaultQuartzScheduler_Worker-79) starting spm on vds office4a,
      storage pool mill, prevId 2, LVER 15<br>
      2014-01-26 14:44:58,579 INFO&nbsp;
      [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
      (DefaultQuartzScheduler_Worker-79) START,
      SpmStartVDSCommand(HostName = office4a, HostId =
      127ed939-34af-41a8-87a0-e2f6174b1877, storagePoolId =
      536a864d-83aa-473a-a675-e38aafdd9071, prevId=2, prevLVER=15,
      storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false),
      log id: 74c38eb7<br>
      2014-01-26 14:44:58,617 INFO&nbsp;
      [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
      (DefaultQuartzScheduler_Worker-79) spmStart polling started:
      taskId = e8986753-fc80-4b11-a11d-6d3470b1728c<br>
      2014-01-26 14:45:00,662 ERROR
      [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand]
      (DefaultQuartzScheduler_Worker-79) Failed in HSMGetTaskStatusVDS
      method<br>
      2014-01-26 14:45:00,664 ERROR
      [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand]
      (DefaultQuartzScheduler_Worker-79) Error code AcquireHostIdFailure
      and error message VDSGenericException: VDSErrorException: Failed
      to HSMGetTaskStatusVDS, error = Cannot acquire host id<br>
      2014-01-26 14:45:00,665 INFO&nbsp;
      [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
      (DefaultQuartzScheduler_Worker-79) spmStart polling ended: taskId
      = e8986753-fc80-4b11-a11d-6d3470b1728c task status = finished<br>
      2014-01-26 14:45:00,666 ERROR
      [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
      (DefaultQuartzScheduler_Worker-79) Start SPM Task failed - result:
      cleanSuccess, message: VDSGenericException: VDSErrorException:
      Failed to HSMGetTaskStatusVDS, error = Cannot acquire host id<br>
      2014-01-26 14:45:00,695 INFO&nbsp;
      [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
      (DefaultQuartzScheduler_Worker-79) spmStart polling ended, spm
      status: Free<br>
      2014-01-26 14:45:00,702 INFO&nbsp;
      [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand]
      (DefaultQuartzScheduler_Worker-79) START,
      HSMClearTaskVDSCommand(HostName = office4a, HostId =
      127ed939-34af-41a8-87a0-e2f6174b1877,
      taskId=e8986753-fc80-4b11-a11d-6d3470b1728c), log id: 336ec5a6<br>
      2014-01-26 14:45:00,722 INFO&nbsp;
      [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand]
      (DefaultQuartzScheduler_Worker-79) FINISH, HSMClearTaskVDSCommand,
      log id: 336ec5a6<br>
      2014-01-26 14:45:00,724 INFO&nbsp;
      [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
      (DefaultQuartzScheduler_Worker-79) FINISH, SpmStartVDSCommand,
      return:
      org.ovirt.engine.core.common.businessentities.SpmStatusResult@13652652,
      log id: 74c38eb7<br>
      2014-01-26 14:45:00,733 INFO&nbsp;
      [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand]
      (DefaultQuartzScheduler_Worker-79) Running command:
      SetStoragePoolStatusCommand internal: true. Entities affected :&nbsp;
      ID: 536a864d-83aa-473a-a675-e38aafdd9071 Type: StoragePool<br>
      2014-01-26 14:45:00,778 ERROR
      [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
      (DefaultQuartzScheduler_Worker-79)
      IrsBroker::Failed::GetStoragePoolInfoVDS due to:
      IrsSpmStartFailedException: IRSGenericException:
      IRSErrorException: SpmStart failed<br>
    </font><br>
    Ted Miller<br>
    Elkhart, IN, USA<br>
    <br>
  </body>
</html>