Two other symptoms:

- Some of the HV's respond 'Destination Host Prohibited' on ping, but SSH works fine

- Seeing this in the log:

2020-04-01 10:34:31,839 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) Unable to process messages: javax.net.ssl.SSLException: Received fatal alert: certificate_expired

***
Mark Steele
CIO / VP Technical Operations | TelVue Corporation
TelVue - We Share Your Vision
16000 Horizon Way, Suite 100 | Mt. Laurel, NJ 08054


On Wed, Apr 1, 2020 at 10:25 AM Mark Steele <msteele@telvue.com> wrote:
Yesterday we had a storage crash that caused our hosted engine to stop and not auto-restart. We manually restarted it and this morning, all elements are showing a status of either question mark '?' or a red down chevron.

We are running version oVirt Engine Version: 3.5.0.1-1.el6.

We are chasing it down and see this in the log:

2020-04-01 10:15:53,060 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Failed to refresh VDS , vds = e98f8f1d-dd1d-47c4-b039-d6eaaf578317 : hv-06, VDS Network Error, continuing

Any insight into what might be causing this or the fix would be appreciated.

***
Mark Steele