I checked the database using the same command that engine-setup uses:

engine=> select vds_id, ha_global_maintenance from vds_statistics ;  
               vds_id                | ha_global_maintenance  
--------------------------------------+----------------------- 
3e7dbbf6-a1f3-490b-8d66-74618c29366b | f 
1057a613-7ebd-4789-bc5f-25e6ba5d6a40 | f 
378c0ee8-e74f-4052-9dc7-347eb3f9d42b | f 
03f6ec4d-2738-41fe-b914-a3ebcc1e3c48 | f 
(4 rows)

And it sure isn't in global maintenance even though all the hosted engine hosts show that it is in global maintenance.  Do I manually change those flags, that seems dangerous...

Logan

On Mon, Mar 20, 2017 at 8:14 AM, Logan Kuhn <support@jac-properties.com> wrote:
Starting at 1:09am on Saturday the Hosted Engine has been rebooting because it failed it's liveliness check.  This is due to the webadmin not loading.  Nothing changed as far as I can tell on the engine since it's last successful reboot on Friday afternoon.  

The engine, dwhd and httpd are all up and do not seem to be reporting anything unusual in their respective logs.  The engine can talk to the database as I can login using the credentials in /etc/ovirt-engine/engine.conf.d/10-setup-database.conf and the logs on the postgres server are showing activity.

I tried to run engine-setup but it says it's not in global maintenance even though the hosted engine hosts agree that it is.  We are on version 4.0.6.3

Server, engine and agent logs are attached

Regards,
Logan