Anyone have any other possible information on this? I've noticed this issue before and usually it just takes a bit of time for the cluster to 'settle' after some node reboots but its been a few days and its still marked as stale. 



--== Host 1 status ==--

Status up-to-date                  : False
Hostname                           : njsevcnp01
Host ID                            : 1
Engine status                      : unknown stale-data
Score                              : 0
stopped                            : True
Local maintenance                  : False
crc32                              : 260dbf06
Host timestamp                     : 327


--== Host 2 status ==--

Status up-to-date                  : False
Hostname                           : njsevcnp02
Host ID                            : 2
Engine status                      : unknown stale-data
Score                              : 0
stopped                            : True
Local maintenance                  : False
crc32                              : 25da07df
Host timestamp                     : 3030


--== Host 3 status ==--

Status up-to-date                  : False
Hostname                           : njsevcnp03
Host ID                            : 3
Engine status                      : unknown stale-data
Score                              : 0
stopped                            : True
Local maintenance                  : False
crc32                              : c67818cb
Host timestamp                     : 10877406


&& vdsClient on node2 showing hosted engine is up on node 2

48207078-8cb0-413c-8984-40aa772f4d94
Status = Up
nicModel = rtl8139,pv
statusTime = 4540044460
emulatedMachine = pc
pid = 30571
vmName = HostedEngine
devices = [{'device': 'memballoon', 'specParams': {'model': 'none'}, 'type': 'balloon', 'alias': 'balloon0'}, {'alias': 'scsi0', 'deviceId': '17f10db1-2e9e-4422-9ea5-61a628072e29', 'address': {'slot': '0x04', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'scsi', 'model': 'virtio-scsi', 'type': 'controller'}, {'device': 'usb', 'alias': 'usb', 'type': 'controller', 'deviceId': '9be34ac0-7d00-4a95-bdfe-5b328fc1355b', 'address': {'slot': '0x01', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x2'}}, {'device': 'ide', 'alias': 'ide', 'type': 'controller', 'deviceId': '222629a8-0dd6-4e8e-9b42-43aac314c0c2', 'address': {'slot': '0x01', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x1'}}, {'device': 'virtio-serial', 'alias': 'virtio-serial0', 'type': 'controller', 'deviceId': '7cbccd04-853a-408f-94c2-5b10b641b7af', 'address': {'slot': '0x05', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}}, {'device': 'vnc', 'specParams': {'spiceSecureChannels': 'smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir', 'displayIp': '0'}, 'type': 'graphics', 'port': '5900'}, {'nicModel': 'pv', 'macAddr': '00:16:3e:16:83:91', 'linkActive': True, 'network': 'ovirtmgmt', 'alias': 'net0', 'deviceId': '3f679659-142c-41f3-a69d-4264d7234fbc', 'address': {'slot': '0x03', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'bridge', 'type': 'interface', 'name': 'vnet0'}, {'address': {'slot': '0x06', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'volumeInfo': {'domainID': 'c6323975-2966-409d-b9e0-48370a513a98', 'volType': 'path', 'leaseOffset': 0, 'volumeID': 'aa66d378-5a5f-490c-b0ab-993b79838d95', 'leasePath': '/rhev/data-center/mnt/nas01:_volume1_vm__os_ovirt36__engine/c6323975-2966-409d-b9e0-48370a513a98/images/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95.lease', 'imageID': '8518ef4a-7b17-4291-856c-81875ba4e264', 'path': '/rhev/data-center/mnt/nas01:_volume1_vm__os_ovirt36__engine/c6323975-2966-409d-b9e0-48370a513a98/images/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95'}, 'index': '0', 'iface': 'virtio', 'apparentsize': '10737418240', 'imageID': '8518ef4a-7b17-4291-856c-81875ba4e264', 'readonly': 'False', 'shared': 'exclusive', 'truesize': '6899802112', 'type': 'disk', 'domainID': 'c6323975-2966-409d-b9e0-48370a513a98', 'reqsize': '0', 'format': 'raw', 'deviceId': '8518ef4a-7b17-4291-856c-81875ba4e264', 'poolID': '00000000-0000-0000-0000-000000000000', 'device': 'disk', 'path': '/var/run/vdsm/storage/c6323975-2966-409d-b9e0-48370a513a98/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95', 'propagateErrors': 'off', 'name': 'vda', 'bootOrder': '1', 'volumeID': 'aa66d378-5a5f-490c-b0ab-993b79838d95', 'alias': 'virtio-disk0', 'volumeChain': [{'domainID': 'c6323975-2966-409d-b9e0-48370a513a98', 'volType': 'path', 'leaseOffset': 0, 'volumeID': 'aa66d378-5a5f-490c-b0ab-993b79838d95', 'leasePath': '/rhev/data-center/mnt/nas01:_volume1_vm__os_ovirt36__engine/c6323975-2966-409d-b9e0-48370a513a98/images/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95.lease', 'imageID': '8518ef4a-7b17-4291-856c-81875ba4e264', 'path': '/rhev/data-center/mnt/nas01:_volume1_vm__os_ovirt36__engine/c6323975-2966-409d-b9e0-48370a513a98/images/8518ef4a-7b17-4291-856c-81875ba4e264/aa66d378-5a5f-490c-b0ab-993b79838d95'}]}, {'index': '2', 'iface': 'ide', 'name': 'hdc', 'alias': 'ide0-1-0', 'readonly': 'True', 'deviceId': '8c3179ac-b322-4f5c-9449-c52e3665e0ae', 'address': {'bus': '1', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'device': 'cdrom', 'shared': 'false', 'path': '', 'type': 'disk'}, {'device': 'unix', 'alias': 'channel0', 'type': 'channel', 'address': {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '1'}}, {'device': 'unix', 'alias': 'channel1', 'type': 'channel', 'address': {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '2'}}, {'device': 'unix', 'alias': 'channel2', 'type': 'channel', 'address': {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '3'}}, {'device': '', 'alias': 'video0', 'type': 'video', 'address': {'slot': '0x02', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}}]
guestDiskMapping = {'8518ef4a-7b17-4291-8': {'name': '/dev/vda'}, 'QEMU_DVD-ROM_QM00003': {'name': '/dev/sr0'}}
vmType = kvm
displaySecurePort = -1
memSize = 4096
displayPort = 5900
clientIp = 
spiceSecureChannels = smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir
smp = 4
displayIp = 0
display = vnc
pauseCode = NOERR




On Mon, Jun 13, 2016 at 8:25 AM, Charles Kozler <charles@fixflyer.com> wrote:
It is up. I can do "ps -Aef | grep -i qemu-kvm | grep -i hosted" and see it running. I also forcefully shut it down with hosted-engine --vm-stop when it was on node1 and then did --vm-start on node 2 and it came up. Also the Web UI is reachable so thats how I also know the hosted engine VM is running

On Mon, Jun 13, 2016 at 8:24 AM, Alexis HAUSER <alexis.hauser@telecom-bretagne.eu> wrote:

> http://imgur.com/a/6xkaS

I had similar errors with one single host and a hosted-engine VM.
My case should be totally different, but one thing you could try first is to check VM is really up.
In my issues, VM was shown by hosted-engine command as up, but was down. with vdsClient command, you can check if it's status with more details.

What is the result for you of the following command ?

 vdsClient -s 0 list



--

Charles Kozler
Vice President, IT Operations
 
FIX Flyer, LLC
225 Broadway | Suite 1600 | New York, NY 10007
 
NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION WHICH IS PROPRIETARY TO FIX FLYER LLC.  ANY UNAUTHORIZED USE, COPYING, DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED.  ALL RIGHTS TO THIS INFORMATION IS RESERVED BY FIX FLYER LLC.  IF YOU ARE NOT THE INTENDED RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.



--

Charles Kozler
Vice President, IT Operations
 
FIX Flyer, LLC
225 Broadway | Suite 1600 | New York, NY 10007
1-888-349-3593
 
NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION WHICH IS PROPRIETARY TO FIX FLYER LLC.  ANY UNAUTHORIZED USE, COPYING, DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED.  ALL RIGHTS TO THIS INFORMATION IS RESERVED BY FIX FLYER LLC.  IF YOU ARE NOT THE INTENDED RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.