Hope you are well. Got a bit of an issue, actually a big issue. We had some sort of dip of some sort. All the VM's is still running, but some of the hosts is show "Unassigned" or "NonResponsive". So all the hosts was showing UP and was fine before our dip. So I did increase  vdsHeartbeatInSecond to 240, no luck.

I still get a timeout on the engine lock even thou I can connect to that host from the engine using nc to test to port 54321. I also did restart vdsmd and also rebooted the host with no luck.

 nc -v someserver 54321
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 172.40.2.172:54321.

2020-08-05 20:20:34,256+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM someserver command Get Host Capabilities failed: Message timeout which can be caused by communication issues

Any troubleshoot ideas will be gladly appreciated.

Regards

Nar