[ovirt-users] Debugging why hosted engine flips between EngineUp and EngineBadHealth

[Johan Bernhardsson]

We have had a similar issue that has been resolved with restarting the
engine vps. 

Not ideal but it solves the problem for a about a month.

/JohanOn Fri, 2017-12-01 at 10:50 +0100, Luca 'remix_tj' Lorenzetto wrote:
> Hi all,
> 
> since some days my hosted-engine environments (one RHEV 4.0.7, one
> ovirt 4.1.7) continue to send mails about changes between EngineUp
> and
> EngineBadHealth.
> 
> This is pretty annoying and i'm not able to find out the root cause.
> 
> The only issue i've seen on hosts is this error appearing sometimes
> randomly about sending mails.
> 
> Thread-1::ERROR::2017-12-01
> 03:05:05,084::notifications::39::ovirt_hosted_engine_ha.broker.notifi
> cations.Notifications::(send_email)
> [Errno -2] Name or service not known
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-
> packages/ovirt_hosted_engine_ha/broker/notifications.py",
> line 26, in send_email
>     timeout=float(cfg["smtp-timeout"]))
>   File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__
>     (code, msg) = self.connect(host, port)
>   File "/usr/lib64/python2.7/smtplib.py", line 315, in connect
>     self.sock = self._get_socket(host, port, self.timeout)
>   File "/usr/lib64/python2.7/smtplib.py", line 290, in _get_socket
>     return socket.create_connection((host, port), timeout)
>   File "/usr/lib64/python2.7/socket.py", line 553, in
> create_connection
>     for res in getaddrinfo(host, port, 0, SOCK_STREAM):
> gaierror: [Errno -2] Name or service not known
> Thread-6::WARNING::2017-12-01
> 03:05:05,427::engine_health::130::engine_health.CpuLoadNoEngine::(act
> ion)
> bad health status: Hosted Engine is not up!
> 
> There are no errors on engine logs and all the api queries done by
> ovirt-hosted-engine-ha returns HTTP code 200.
> 
> I suspect the switch between EngineUP and EngineBadHealth status
> could
> be due to some dns resolution issues, but there is no clear message
> on
> the log showing this and this doesn't help our netadmins to make some
> traces.
> 
> Is there a way to increase the verbosity of broker.log and agent.log?
> 
> Luca
>