<html>
  <head>
    <meta content="text/html; charset=windows-1252"
      http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    Are the ovirt-ha-agent and ovirt-ha-broker services running on all
    the nodes? If they are, check the agent.log and broker.log for
    errors.<br>
    <br>
    <div class="moz-cite-prefix">On 06/14/2016 05:29 PM, Charles Kozler
      wrote:<br>
    </div>
    <blockquote
cite="mid:CACiW5VJhRkJ8ZsWvm9u97Kp+5_iXmR6B4MkPoysiMgPvLPtRDA@mail.gmail.com"
      type="cite">
      <div dir="ltr">Anyone have any other possible information on this?
        I've noticed this issue before and usually it just takes a bit
        of time for the cluster to 'settle' after some node reboots but
        its been a few days and its still marked as stale. 
        <div><br>
        </div>
        <div>
          <div><br>
          </div>
          <div><br>
          </div>
          <div>--== Host 1 status ==--</div>
          <div><br>
          </div>
          <div>Status up-to-date                  : False</div>
          <div>Hostname                           : njsevcnp01</div>
          <div>Host ID                            : 1</div>
          <div>Engine status                      : unknown stale-data</div>
          <div>Score                              : 0</div>
          <div>stopped                            : True</div>
          <div>Local maintenance                  : False</div>
          <div>crc32                              : 260dbf06</div>
          <div>Host timestamp                     : 327</div>
          <div><br>
          </div>
          <div><br>
          </div>
          <div>--== Host 2 status ==--</div>
          <div><br>
          </div>
          <div>Status up-to-date                  : False</div>
          <div>Hostname                           : njsevcnp02</div>
          <div>Host ID                            : 2</div>
          <div>Engine status                      : unknown stale-data</div>
          <div>Score                              : 0</div>
          <div>stopped                            : True</div>
          <div>Local maintenance                  : False</div>
          <div>crc32                              : 25da07df</div>
          <div>Host timestamp                     : 3030</div>
          <div><br>
          </div>
          <div><br>
          </div>
          <div>--== Host 3 status ==--</div>
          <div><br>
          </div>
          <div>Status up-to-date                  : False</div>
          <div>Hostname                           : njsevcnp03</div>
          <div>Host ID                            : 3</div>
          <div>Engine status                      : unknown stale-data</div>
          <div>Score                              : 0</div>
          <div>stopped                            : True</div>
          <div>Local maintenance                  : False</div>
          <div>crc32                              : c67818cb</div>
          <div>Host timestamp                     : 10877406</div>
        </div>
        <div><br>
        </div>
        <div><br>
        </div>
        <div>&amp;&amp; vdsClient on node2 showing hosted engine is up
          on node 2</div>
        <div><br>
        </div>
        <div>
          <div>48207078-8cb0-413c-8984-40aa772f4d94</div>
          <div><span class="" style="white-space:pre">        </span>Status =
            Up</div>
          <div><span class="" style="white-space:pre">        </span>nicModel =
            rtl8139,pv</div>
          <div><span class="" style="white-space:pre">        </span>statusTime
            = 4540044460</div>
          <div><span class="" style="white-space:pre">        </span>emulatedMachine
            = pc</div>
          <div><span class="" style="white-space:pre">        </span>pid =
            30571</div>
          <div><span class="" style="white-space:pre">        </span>vmName =
            HostedEngine</div>
          <div><span class="" style="white-space:pre">        </span>devices =
            [{'device': 'memballoon', 'specParams': {'model': 'none'},
            'type': 'balloon', 'alias': 'balloon0'}, {'alias': 'scsi0',
            'deviceId': '17f10db1-2e9e-4422-9ea5-61a628072e29',
            'address': {'slot': '0x04', 'bus': '0x00', 'domain':
            '0x0000', 'type': 'pci', 'function': '0x0'}, 'device':
            'scsi', 'model': 'virtio-scsi', 'type': 'controller'},
            {'device': 'usb', 'alias': 'usb', 'type': 'controller',
            'deviceId': '9be34ac0-7d00-4a95-bdfe-5b328fc1355b',
            'address': {'slot': '0x01', 'bus': '0x00', 'domain':
            '0x0000', 'type': 'pci', 'function': '0x2'}}, {'device':
            'ide', 'alias': 'ide', 'type': 'controller', 'deviceId':
            '222629a8-0dd6-4e8e-9b42-43aac314c0c2', 'address': {'slot':
            '0x01', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci',
            'function': '0x1'}}, {'device': 'virtio-serial', 'alias':
            'virtio-serial0', 'type': 'controller', 'deviceId':
            '7cbccd04-853a-408f-94c2-5b10b641b7af', 'address': {'slot':
            '0x05', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci',
            'function': '0x0'}}, {'device': 'vnc', 'specParams':
            {'spiceSecureChannels':
            'smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir',
            'displayIp': '0'}, 'type': 'graphics', 'port': '5900'},
            {'nicModel': 'pv', 'macAddr': '00:16:3e:16:83:91',
            'linkActive': True, 'network': 'ovirtmgmt', 'alias': 'net0',
            'deviceId': '3f679659-142c-41f3-a69d-4264d7234fbc',
            'address': {'slot': '0x03', 'bus': '0x00', 'domain':
            '0x0000', 'type': 'pci', 'function': '0x0'}, 'device':
            'bridge', 'type': 'interface', 'name': 'vnet0'}, {'address':
            {'slot': '0x06', 'bus': '0x00', 'domain': '0x0000', 'type':
            'pci', 'function': '0x0'}, 'volumeInfo': {'domainID':
            'c6323975-2966-409d-b9e0-48370a513a98', 'volType': 'path',
            'leaseOffset': 0, 'volumeID':
            'aa66d378-5a5f-490c-b0ab-993b79838d95', 'leasePath':
            '/rhev/data-center/mnt/nas01:_volume1_vm__os_ovirt36__engine/c6323975-2966-409d-b9e0-48370a513a98/images/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95.lease',
            'imageID': '8518ef4a-7b17-4291-856c-81875ba4e264', 'path':
            '/rhev/data-center/mnt/nas01:_volume1_vm__os_ovirt36__engine/c6323975-2966-409d-b9e0-48370a513a98/images/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95'},
            'index': '0', 'iface': 'virtio', 'apparentsize':
            '10737418240', 'imageID':
            '8518ef4a-7b17-4291-856c-81875ba4e264', 'readonly': 'False',
            'shared': 'exclusive', 'truesize': '6899802112', 'type':
            'disk', 'domainID': 'c6323975-2966-409d-b9e0-48370a513a98',
            'reqsize': '0', 'format': 'raw', 'deviceId':
            '8518ef4a-7b17-4291-856c-81875ba4e264', 'poolID':
            '00000000-0000-0000-0000-000000000000', 'device': 'disk',
            'path':
            '/var/run/vdsm/storage/c6323975-2966-409d-b9e0-48370a513a98/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95',
            'propagateErrors': 'off', 'name': 'vda', 'bootOrder': '1',
            'volumeID': 'aa66d378-5a5f-490c-b0ab-993b79838d95', 'alias':
            'virtio-disk0', 'volumeChain': [{'domainID':
            'c6323975-2966-409d-b9e0-48370a513a98', 'volType': 'path',
            'leaseOffset': 0, 'volumeID':
            'aa66d378-5a5f-490c-b0ab-993b79838d95', 'leasePath':
            '/rhev/data-center/mnt/nas01:_volume1_vm__os_ovirt36__engine/c6323975-2966-409d-b9e0-48370a513a98/images/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95.lease',
            'imageID': '8518ef4a-7b17-4291-856c-81875ba4e264', 'path':
            '/rhev/data-center/mnt/nas01:_volume1_vm__os_ovirt36__engine/c6323975-2966-409d-b9e0-48370a513a98/images/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95'}]},
            {'index': '2', 'iface': 'ide', 'name': 'hdc', 'alias':
            'ide0-1-0', 'readonly': 'True', 'deviceId':
            '8c3179ac-b322-4f5c-9449-c52e3665e0ae', 'address': {'bus':
            '1', 'controller': '0', 'type': 'drive', 'target': '0',
            'unit': '0'}, 'device': 'cdrom', 'shared': 'false', 'path':
            '', 'type': 'disk'}, {'device': 'unix', 'alias': 'channel0',
            'type': 'channel', 'address': {'bus': '0', 'controller':
            '0', 'type': 'virtio-serial', 'port': '1'}}, {'device':
            'unix', 'alias': 'channel1', 'type': 'channel', 'address':
            {'bus': '0', 'controller': '0', 'type': 'virtio-serial',
            'port': '2'}}, {'device': 'unix', 'alias': 'channel2',
            'type': 'channel', 'address': {'bus': '0', 'controller':
            '0', 'type': 'virtio-serial', 'port': '3'}}, {'device': '',
            'alias': 'video0', 'type': 'video', 'address': {'slot':
            '0x02', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci',
            'function': '0x0'}}]</div>
          <div><span class="" style="white-space:pre">        </span>guestDiskMapping
            = {'8518ef4a-7b17-4291-8': {'name': '/dev/vda'},
            'QEMU_DVD-ROM_QM00003': {'name': '/dev/sr0'}}</div>
          <div><span class="" style="white-space:pre">        </span>vmType =
            kvm</div>
          <div><span class="" style="white-space:pre">        </span>displaySecurePort
            = -1</div>
          <div><span class="" style="white-space:pre">        </span>memSize =
            4096</div>
          <div><span class="" style="white-space:pre">        </span>displayPort
            = 5900</div>
          <div><span class="" style="white-space:pre">        </span>clientIp
            = </div>
          <div><span class="" style="white-space:pre">        </span>spiceSecureChannels
            =
            smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir</div>
          <div><span class="" style="white-space:pre">        </span>smp = 4</div>
          <div><span class="" style="white-space:pre">        </span>displayIp
            = 0</div>
          <div><span class="" style="white-space:pre">        </span>display =
            vnc</div>
          <div><span class="" style="white-space:pre">        </span>pauseCode
            = NOERR</div>
        </div>
        <div><br>
        </div>
        <div><br>
        </div>
        <div><br>
        </div>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On Mon, Jun 13, 2016 at 8:25 AM,
          Charles Kozler <span dir="ltr">&lt;<a moz-do-not-send="true"
              href="mailto:charles@fixflyer.com" target="_blank">charles@fixflyer.com</a>&gt;</span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div dir="ltr">It is up. I can do "ps -Aef | grep -i
              qemu-kvm | grep -i hosted" and see it running. I also
              forcefully shut it down with hosted-engine --vm-stop when
              it was on node1 and then did --vm-start on node 2 and it
              came up. Also the Web UI is reachable so thats how I also
              know the hosted engine VM is running</div>
            <div class="gmail_extra">
              <div>
                <div class="h5"><br>
                  <div class="gmail_quote">On Mon, Jun 13, 2016 at 8:24
                    AM, Alexis HAUSER <span dir="ltr">&lt;<a
                        moz-do-not-send="true"
                        href="mailto:alexis.hauser@telecom-bretagne.eu"
                        target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:alexis.hauser@telecom-bretagne.eu">alexis.hauser@telecom-bretagne.eu</a></a>&gt;</span>
                    wrote:<br>
                    <blockquote class="gmail_quote" style="margin:0 0 0
                      .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
                      &gt; <a moz-do-not-send="true"
                        href="http://imgur.com/a/6xkaS" rel="noreferrer"
                        target="_blank">http://imgur.com/a/6xkaS</a><br>
                      <br>
                      I had similar errors with one single host and a
                      hosted-engine VM.<br>
                      My case should be totally different, but one thing
                      you could try first is to check VM is really up.<br>
                      In my issues, VM was shown by hosted-engine
                      command as up, but was down. with vdsClient
                      command, you can check if it's status with more
                      details.<br>
                      <br>
                      What is the result for you of the following
                      command ?<br>
                      <br>
                       vdsClient -s 0 list<br>
                    </blockquote>
                  </div>
                  <br>
                  <br clear="all">
                  <div><br>
                  </div>
                </div>
              </div>
              <span class="">-- <br>
                <div data-smartmail="gmail_signature">
                  <div dir="ltr">
                    <div>
                      <div dir="ltr">
                        <div>
                          <div dir="ltr">
                            <div><br
                                style="color:rgb(0,0,0);font-family:'Times
                                New Roman';font-size:medium">
                              <div
                                style="color:rgb(0,0,0);font-family:'Times
                                New Roman'"><strong><span
                                    style="font-family:'times new
                                    roman',times,serif"><font size="2">Charles
                                      Kozler</font></span></strong></div>
                              <div
                                style="color:rgb(0,0,0);font-family:'Times
                                New Roman'"><em><span
                                    style="font-family:'times new
                                    roman',times,serif"><font size="2">Vice
                                      President, IT Operations</font></span></em></div>
                              <div
                                style="color:rgb(0,0,0);font-family:'Times
                                New Roman';font-size:medium"> </div>
                              <div
                                style="color:rgb(0,0,0);font-family:'Times
                                New Roman'"><span
                                  style="font-family:'times new
                                  roman',times,serif"><font size="2">FIX
                                    Flyer, LLC</font></span></div>
                              <div
                                style="color:rgb(0,0,0);font-family:'Times
                                New Roman'"><span
                                  style="font-family:'times new
                                  roman',times,serif"><font size="2">225
                                    Broadway | Suite 1600 | New York, NY
                                    10007</font></span></div>
                              <div
                                style="color:rgb(0,0,0);font-family:'Times
                                New Roman'"><span
                                  style="font-family:'times new
                                  roman',times,serif"><font size="2"><a
                                      moz-do-not-send="true"
                                      href="tel:1-888-349-3593"
                                      value="+18883493593"
                                      target="_blank">1-888-349-3593</a></font></span></div>
                              <div
                                style="color:rgb(0,0,0);font-family:'Times
                                New Roman'"><a moz-do-not-send="true"
                                  href="http://fixflyer.com"
                                  target="_blank"><span
                                    style="font-family:'times new
                                    roman',times,serif"><font size="2">http://www.fixflyer.com</font></span></a></div>
                              <div
                                style="color:rgb(0,0,0);font-family:'Times
                                New Roman'"><font size="2"> </font></div>
                              <div
                                style="color:rgb(0,0,0);font-family:'Times
                                New Roman'"><font size="1">NOTICE TO
                                  RECIPIENT: THIS E-MAIL IS MEANT ONLY
                                  FOR THE INTENDED RECIPIENT(S) OF THE
                                  TRANSMISSION, AND CONTAINS
                                  CONFIDENTIAL INFORMATION WHICH IS
                                  PROPRIETARY TO FIX FLYER LLC.  ANY
                                  UNAUTHORIZED USE, COPYING,
                                  DISTRIBUTION, OR DISSEMINATION IS
                                  STRICTLY PROHIBITED.  ALL RIGHTS TO
                                  THIS INFORMATION IS RESERVED BY FIX
                                  FLYER LLC.  IF YOU ARE NOT THE
                                  INTENDED RECIPIENT, PLEASE CONTACT THE
                                  SENDER BY REPLY E-MAIL AND PLEASE
                                  DELETE THIS E-MAIL FROM YOUR SYSTEM
                                  AND DESTROY ANY COPIES.</font></div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </div>
              </span></div>
          </blockquote>
        </div>
        <br>
        <br clear="all">
        <div><br>
        </div>
        -- <br>
        <div class="gmail_signature" data-smartmail="gmail_signature">
          <div dir="ltr">
            <div>
              <div dir="ltr">
                <div>
                  <div dir="ltr">
                    <div><br style="color:rgb(0,0,0);font-family:'Times
                        New Roman';font-size:medium">
                      <div style="color:rgb(0,0,0);font-family:'Times
                        New Roman'"><strong><span
                            style="font-family:'times new
                            roman',times,serif"><font size="2">Charles
                              Kozler</font></span></strong></div>
                      <div style="color:rgb(0,0,0);font-family:'Times
                        New Roman'"><em><span style="font-family:'times
                            new roman',times,serif"><font size="2">Vice
                              President, IT Operations</font></span></em></div>
                      <div style="color:rgb(0,0,0);font-family:'Times
                        New Roman';font-size:medium"> </div>
                      <div style="color:rgb(0,0,0);font-family:'Times
                        New Roman'"><span style="font-family:'times new
                          roman',times,serif"><font size="2">FIX Flyer,
                            LLC</font></span></div>
                      <div style="color:rgb(0,0,0);font-family:'Times
                        New Roman'"><span style="font-family:'times new
                          roman',times,serif"><font size="2">225
                            Broadway | Suite 1600 | New York, NY 10007</font></span></div>
                      <div style="color:rgb(0,0,0);font-family:'Times
                        New Roman'"><span style="font-family:'times new
                          roman',times,serif"><font size="2">1-888-349-3593</font></span></div>
                      <div style="color:rgb(0,0,0);font-family:'Times
                        New Roman'"><a moz-do-not-send="true"
                          href="http://fixflyer.com" target="_blank"><span
                            style="font-family:'times new
                            roman',times,serif"><font size="2">http://www.fixflyer.com</font></span></a></div>
                      <div style="color:rgb(0,0,0);font-family:'Times
                        New Roman'"><font size="2"> </font></div>
                      <div style="color:rgb(0,0,0);font-family:'Times
                        New Roman'"><font size="1">NOTICE TO RECIPIENT:
                          THIS E-MAIL IS MEANT ONLY FOR THE INTENDED
                          RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS
                          CONFIDENTIAL INFORMATION WHICH IS PROPRIETARY
                          TO FIX FLYER LLC.  ANY UNAUTHORIZED USE,
                          COPYING, DISTRIBUTION, OR DISSEMINATION IS
                          STRICTLY PROHIBITED.  ALL RIGHTS TO THIS
                          INFORMATION IS RESERVED BY FIX FLYER LLC.  IF
                          YOU ARE NOT THE INTENDED RECIPIENT, PLEASE
                          CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE
                          DELETE THIS E-MAIL FROM YOUR SYSTEM AND
                          DESTROY ANY COPIES.</font></div>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </div>
        </div>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
    </blockquote>
    <br>
  </body>
</html>