<div dir="ltr">Anyone have any other possible information on this? I&#39;ve noticed this issue before and usually it just takes a bit of time for the cluster to &#39;settle&#39; after some node reboots but its been a few days and its still marked as stale. <div><br></div><div><div><br></div><div><br></div><div>--== Host 1 status ==--</div><div><br></div><div>Status up-to-date                  : False</div><div>Hostname                           : njsevcnp01</div><div>Host ID                            : 1</div><div>Engine status                      : unknown stale-data</div><div>Score                              : 0</div><div>stopped                            : True</div><div>Local maintenance                  : False</div><div>crc32                              : 260dbf06</div><div>Host timestamp                     : 327</div><div><br></div><div><br></div><div>--== Host 2 status ==--</div><div><br></div><div>Status up-to-date                  : False</div><div>Hostname                           : njsevcnp02</div><div>Host ID                            : 2</div><div>Engine status                      : unknown stale-data</div><div>Score                              : 0</div><div>stopped                            : True</div><div>Local maintenance                  : False</div><div>crc32                              : 25da07df</div><div>Host timestamp                     : 3030</div><div><br></div><div><br></div><div>--== Host 3 status ==--</div><div><br></div><div>Status up-to-date                  : False</div><div>Hostname                           : njsevcnp03</div><div>Host ID                            : 3</div><div>Engine status                      : unknown stale-data</div><div>Score                              : 0</div><div>stopped                            : True</div><div>Local maintenance                  : False</div><div>crc32                              : c67818cb</div><div>Host timestamp                     : 10877406</div></div><div><br></div><div><br></div><div>&amp;&amp; vdsClient on node2 showing hosted engine is up on node 2</div><div><br></div><div><div>48207078-8cb0-413c-8984-40aa772f4d94</div><div><span class="" style="white-space:pre">        </span>Status = Up</div><div><span class="" style="white-space:pre">        </span>nicModel = rtl8139,pv</div><div><span class="" style="white-space:pre">        </span>statusTime = 4540044460</div><div><span class="" style="white-space:pre">        </span>emulatedMachine = pc</div><div><span class="" style="white-space:pre">        </span>pid = 30571</div><div><span class="" style="white-space:pre">        </span>vmName = HostedEngine</div><div><span class="" style="white-space:pre">        </span>devices = [{&#39;device&#39;: &#39;memballoon&#39;, &#39;specParams&#39;: {&#39;model&#39;: &#39;none&#39;}, &#39;type&#39;: &#39;balloon&#39;, &#39;alias&#39;: &#39;balloon0&#39;}, {&#39;alias&#39;: &#39;scsi0&#39;, &#39;deviceId&#39;: &#39;17f10db1-2e9e-4422-9ea5-61a628072e29&#39;, &#39;address&#39;: {&#39;slot&#39;: &#39;0x04&#39;, &#39;bus&#39;: &#39;0x00&#39;, &#39;domain&#39;: &#39;0x0000&#39;, &#39;type&#39;: &#39;pci&#39;, &#39;function&#39;: &#39;0x0&#39;}, &#39;device&#39;: &#39;scsi&#39;, &#39;model&#39;: &#39;virtio-scsi&#39;, &#39;type&#39;: &#39;controller&#39;}, {&#39;device&#39;: &#39;usb&#39;, &#39;alias&#39;: &#39;usb&#39;, &#39;type&#39;: &#39;controller&#39;, &#39;deviceId&#39;: &#39;9be34ac0-7d00-4a95-bdfe-5b328fc1355b&#39;, &#39;address&#39;: {&#39;slot&#39;: &#39;0x01&#39;, &#39;bus&#39;: &#39;0x00&#39;, &#39;domain&#39;: &#39;0x0000&#39;, &#39;type&#39;: &#39;pci&#39;, &#39;function&#39;: &#39;0x2&#39;}}, {&#39;device&#39;: &#39;ide&#39;, &#39;alias&#39;: &#39;ide&#39;, &#39;type&#39;: &#39;controller&#39;, &#39;deviceId&#39;: &#39;222629a8-0dd6-4e8e-9b42-43aac314c0c2&#39;, &#39;address&#39;: {&#39;slot&#39;: &#39;0x01&#39;, &#39;bus&#39;: &#39;0x00&#39;, &#39;domain&#39;: &#39;0x0000&#39;, &#39;type&#39;: &#39;pci&#39;, &#39;function&#39;: &#39;0x1&#39;}}, {&#39;device&#39;: &#39;virtio-serial&#39;, &#39;alias&#39;: &#39;virtio-serial0&#39;, &#39;type&#39;: &#39;controller&#39;, &#39;deviceId&#39;: &#39;7cbccd04-853a-408f-94c2-5b10b641b7af&#39;, &#39;address&#39;: {&#39;slot&#39;: &#39;0x05&#39;, &#39;bus&#39;: &#39;0x00&#39;, &#39;domain&#39;: &#39;0x0000&#39;, &#39;type&#39;: &#39;pci&#39;, &#39;function&#39;: &#39;0x0&#39;}}, {&#39;device&#39;: &#39;vnc&#39;, &#39;specParams&#39;: {&#39;spiceSecureChannels&#39;: &#39;smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir&#39;, &#39;displayIp&#39;: &#39;0&#39;}, &#39;type&#39;: &#39;graphics&#39;, &#39;port&#39;: &#39;5900&#39;}, {&#39;nicModel&#39;: &#39;pv&#39;, &#39;macAddr&#39;: &#39;00:16:3e:16:83:91&#39;, &#39;linkActive&#39;: True, &#39;network&#39;: &#39;ovirtmgmt&#39;, &#39;alias&#39;: &#39;net0&#39;, &#39;deviceId&#39;: &#39;3f679659-142c-41f3-a69d-4264d7234fbc&#39;, &#39;address&#39;: {&#39;slot&#39;: &#39;0x03&#39;, &#39;bus&#39;: &#39;0x00&#39;, &#39;domain&#39;: &#39;0x0000&#39;, &#39;type&#39;: &#39;pci&#39;, &#39;function&#39;: &#39;0x0&#39;}, &#39;device&#39;: &#39;bridge&#39;, &#39;type&#39;: &#39;interface&#39;, &#39;name&#39;: &#39;vnet0&#39;}, {&#39;address&#39;: {&#39;slot&#39;: &#39;0x06&#39;, &#39;bus&#39;: &#39;0x00&#39;, &#39;domain&#39;: &#39;0x0000&#39;, &#39;type&#39;: &#39;pci&#39;, &#39;function&#39;: &#39;0x0&#39;}, &#39;volumeInfo&#39;: {&#39;domainID&#39;: &#39;c6323975-2966-409d-b9e0-48370a513a98&#39;, &#39;volType&#39;: &#39;path&#39;, &#39;leaseOffset&#39;: 0, &#39;volumeID&#39;: &#39;aa66d378-5a5f-490c-b0ab-993b79838d95&#39;, &#39;leasePath&#39;: &#39;/rhev/data-center/mnt/nas01:_volume1_vm__os_ovirt36__engine/c6323975-2966-409d-b9e0-48370a513a98/images/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95.lease&#39;, &#39;imageID&#39;: &#39;8518ef4a-7b17-4291-856c-81875ba4e264&#39;, &#39;path&#39;: &#39;/rhev/data-center/mnt/nas01:_volume1_vm__os_ovirt36__engine/c6323975-2966-409d-b9e0-48370a513a98/images/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95&#39;}, &#39;index&#39;: &#39;0&#39;, &#39;iface&#39;: &#39;virtio&#39;, &#39;apparentsize&#39;: &#39;10737418240&#39;, &#39;imageID&#39;: &#39;8518ef4a-7b17-4291-856c-81875ba4e264&#39;, &#39;readonly&#39;: &#39;False&#39;, &#39;shared&#39;: &#39;exclusive&#39;, &#39;truesize&#39;: &#39;6899802112&#39;, &#39;type&#39;: &#39;disk&#39;, &#39;domainID&#39;: &#39;c6323975-2966-409d-b9e0-48370a513a98&#39;, &#39;reqsize&#39;: &#39;0&#39;, &#39;format&#39;: &#39;raw&#39;, &#39;deviceId&#39;: &#39;8518ef4a-7b17-4291-856c-81875ba4e264&#39;, &#39;poolID&#39;: &#39;00000000-0000-0000-0000-000000000000&#39;, &#39;device&#39;: &#39;disk&#39;, &#39;path&#39;: &#39;/var/run/vdsm/storage/c6323975-2966-409d-b9e0-48370a513a98/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95&#39;, &#39;propagateErrors&#39;: &#39;off&#39;, &#39;name&#39;: &#39;vda&#39;, &#39;bootOrder&#39;: &#39;1&#39;, &#39;volumeID&#39;: &#39;aa66d378-5a5f-490c-b0ab-993b79838d95&#39;, &#39;alias&#39;: &#39;virtio-disk0&#39;, &#39;volumeChain&#39;: [{&#39;domainID&#39;: &#39;c6323975-2966-409d-b9e0-48370a513a98&#39;, &#39;volType&#39;: &#39;path&#39;, &#39;leaseOffset&#39;: 0, &#39;volumeID&#39;: &#39;aa66d378-5a5f-490c-b0ab-993b79838d95&#39;, &#39;leasePath&#39;: &#39;/rhev/data-center/mnt/nas01:_volume1_vm__os_ovirt36__engine/c6323975-2966-409d-b9e0-48370a513a98/images/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95.lease&#39;, &#39;imageID&#39;: &#39;8518ef4a-7b17-4291-856c-81875ba4e264&#39;, &#39;path&#39;: &#39;/rhev/data-center/mnt/nas01:_volume1_vm__os_ovirt36__engine/c6323975-2966-409d-b9e0-48370a513a98/images/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95&#39;}]}, {&#39;index&#39;: &#39;2&#39;, &#39;iface&#39;: &#39;ide&#39;, &#39;name&#39;: &#39;hdc&#39;, &#39;alias&#39;: &#39;ide0-1-0&#39;, &#39;readonly&#39;: &#39;True&#39;, &#39;deviceId&#39;: &#39;8c3179ac-b322-4f5c-9449-c52e3665e0ae&#39;, &#39;address&#39;: {&#39;bus&#39;: &#39;1&#39;, &#39;controller&#39;: &#39;0&#39;, &#39;type&#39;: &#39;drive&#39;, &#39;target&#39;: &#39;0&#39;, &#39;unit&#39;: &#39;0&#39;}, &#39;device&#39;: &#39;cdrom&#39;, &#39;shared&#39;: &#39;false&#39;, &#39;path&#39;: &#39;&#39;, &#39;type&#39;: &#39;disk&#39;}, {&#39;device&#39;: &#39;unix&#39;, &#39;alias&#39;: &#39;channel0&#39;, &#39;type&#39;: &#39;channel&#39;, &#39;address&#39;: {&#39;bus&#39;: &#39;0&#39;, &#39;controller&#39;: &#39;0&#39;, &#39;type&#39;: &#39;virtio-serial&#39;, &#39;port&#39;: &#39;1&#39;}}, {&#39;device&#39;: &#39;unix&#39;, &#39;alias&#39;: &#39;channel1&#39;, &#39;type&#39;: &#39;channel&#39;, &#39;address&#39;: {&#39;bus&#39;: &#39;0&#39;, &#39;controller&#39;: &#39;0&#39;, &#39;type&#39;: &#39;virtio-serial&#39;, &#39;port&#39;: &#39;2&#39;}}, {&#39;device&#39;: &#39;unix&#39;, &#39;alias&#39;: &#39;channel2&#39;, &#39;type&#39;: &#39;channel&#39;, &#39;address&#39;: {&#39;bus&#39;: &#39;0&#39;, &#39;controller&#39;: &#39;0&#39;, &#39;type&#39;: &#39;virtio-serial&#39;, &#39;port&#39;: &#39;3&#39;}}, {&#39;device&#39;: &#39;&#39;, &#39;alias&#39;: &#39;video0&#39;, &#39;type&#39;: &#39;video&#39;, &#39;address&#39;: {&#39;slot&#39;: &#39;0x02&#39;, &#39;bus&#39;: &#39;0x00&#39;, &#39;domain&#39;: &#39;0x0000&#39;, &#39;type&#39;: &#39;pci&#39;, &#39;function&#39;: &#39;0x0&#39;}}]</div><div><span class="" style="white-space:pre">        </span>guestDiskMapping = {&#39;8518ef4a-7b17-4291-8&#39;: {&#39;name&#39;: &#39;/dev/vda&#39;}, &#39;QEMU_DVD-ROM_QM00003&#39;: {&#39;name&#39;: &#39;/dev/sr0&#39;}}</div><div><span class="" style="white-space:pre">        </span>vmType = kvm</div><div><span class="" style="white-space:pre">        </span>displaySecurePort = -1</div><div><span class="" style="white-space:pre">        </span>memSize = 4096</div><div><span class="" style="white-space:pre">        </span>displayPort = 5900</div><div><span class="" style="white-space:pre">        </span>clientIp = </div><div><span class="" style="white-space:pre">        </span>spiceSecureChannels = smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir</div><div><span class="" style="white-space:pre">        </span>smp = 4</div><div><span class="" style="white-space:pre">        </span>displayIp = 0</div><div><span class="" style="white-space:pre">        </span>display = vnc</div><div><span class="" style="white-space:pre">        </span>pauseCode = NOERR</div></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 13, 2016 at 8:25 AM, Charles Kozler <span dir="ltr">&lt;<a href="mailto:charles@fixflyer.com" target="_blank">charles@fixflyer.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">It is up. I can do &quot;ps -Aef | grep -i qemu-kvm | grep -i hosted&quot; and see it running. I also forcefully shut it down with hosted-engine --vm-stop when it was on node1 and then did --vm-start on node 2 and it came up. Also the Web UI is reachable so thats how I also know the hosted engine VM is running</div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Mon, Jun 13, 2016 at 8:24 AM, Alexis HAUSER <span dir="ltr">&lt;<a href="mailto:alexis.hauser@telecom-bretagne.eu" target="_blank">alexis.hauser@telecom-bretagne.eu</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
&gt; <a href="http://imgur.com/a/6xkaS" rel="noreferrer" target="_blank">http://imgur.com/a/6xkaS</a><br>
<br>
I had similar errors with one single host and a hosted-engine VM.<br>
My case should be totally different, but one thing you could try first is to check VM is really up.<br>
In my issues, VM was shown by hosted-engine command as up, but was down. with vdsClient command, you can check if it&#39;s status with more details.<br>
<br>
What is the result for you of the following command ?<br>
<br>
 vdsClient -s 0 list<br>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span class="">-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><br style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;;font-size:medium"><div style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;"><strong><span style="font-family:&#39;times new roman&#39;,times,serif"><font size="2">Charles Kozler</font></span></strong></div><div style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;"><em><span style="font-family:&#39;times new roman&#39;,times,serif"><font size="2">Vice President, IT Operations</font></span></em></div><div style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;;font-size:medium"> </div><div style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;"><span style="font-family:&#39;times new roman&#39;,times,serif"><font size="2">FIX Flyer, LLC</font></span></div><div style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;"><span style="font-family:&#39;times new roman&#39;,times,serif"><font size="2">225 Broadway | Suite 1600 | New York, NY 10007</font></span></div><div style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;"><span style="font-family:&#39;times new roman&#39;,times,serif"><font size="2"><a href="tel:1-888-349-3593" value="+18883493593" target="_blank">1-888-349-3593</a></font></span></div><div style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;"><a href="http://fixflyer.com" target="_blank"><span style="font-family:&#39;times new roman&#39;,times,serif"><font size="2">http://www.fixflyer.com</font></span></a></div><div style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;"><font size="2"> </font></div><div style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;"><font size="1">NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION WHICH IS PROPRIETARY TO FIX FLYER LLC.  ANY UNAUTHORIZED USE, COPYING, DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED.  ALL RIGHTS TO THIS INFORMATION IS RESERVED BY FIX FLYER LLC.  IF YOU ARE NOT THE INTENDED RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.</font></div></div></div></div></div></div></div></div>
</span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><br style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;;font-size:medium"><div style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;"><strong><span style="font-family:&#39;times new roman&#39;,times,serif"><font size="2">Charles Kozler</font></span></strong></div><div style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;"><em><span style="font-family:&#39;times new roman&#39;,times,serif"><font size="2">Vice President, IT Operations</font></span></em></div><div style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;;font-size:medium"> </div><div style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;"><span style="font-family:&#39;times new roman&#39;,times,serif"><font size="2">FIX Flyer, LLC</font></span></div><div style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;"><span style="font-family:&#39;times new roman&#39;,times,serif"><font size="2">225 Broadway | Suite 1600 | New York, NY 10007</font></span></div><div style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;"><span style="font-family:&#39;times new roman&#39;,times,serif"><font size="2">1-888-349-3593</font></span></div><div style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;"><a href="http://fixflyer.com" target="_blank"><span style="font-family:&#39;times new roman&#39;,times,serif"><font size="2">http://www.fixflyer.com</font></span></a></div><div style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;"><font size="2"> </font></div><div style="color:rgb(0,0,0);font-family:&#39;Times New Roman&#39;"><font size="1">NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION WHICH IS PROPRIETARY TO FIX FLYER LLC.  ANY UNAUTHORIZED USE, COPYING, DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED.  ALL RIGHTS TO THIS INFORMATION IS RESERVED BY FIX FLYER LLC.  IF YOU ARE NOT THE INTENDED RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.</font></div></div></div></div></div></div></div></div>
</div>