<div dir="ltr">Martin -<div><br></div><div>One thing I noticed on all of the nodes is this:</div><div><br></div><div><div>Jun 14 08:11:11 njsevcnp01 ovirt-ha-agent[15713]: ovirt-ha-agent ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Connection closed: Connection timed out</div><div>Jun 14 08:11:11 <a href="http://njsevcnp01.fixflyer.com">njsevcnp01.fixflyer.com</a> ovirt-ha-agent[15713]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Failed to start monitor <type 'type'>, options {'hostname': 'njsevcnp01'}: Connection timed out' - trying to restart agent</div></div><div><br></div><div>Then the agent is restarted</div><div><br></div><div><div>[root@njsevcnp01 ~]# ps -Aef | grep -i ovirt-ha-agent | grep -iv grep</div><div>vdsm 15713 1 0 08:09 ? 00:00:01 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent --no-daemon</div></div><div><br></div><div>I dont know why the connection would time out because as you can see that log is from node01 and I cant figure out why its timing out on the connection</div><div><br></div><div>The other interesting thing is this log from node01. The odd thing is that it seems there is some split brain somewhere in oVirt because this log is from node02 but it is asking the engine and its getting back "vm not running on this host' rather than 'stale data'. But I dont know engine internals</div><div><br></div><div><div>MainThread::INFO::2016-06-14 08:13:05,163::state_machine::171::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host njsevcnp02 (id 2): {hostname: njsevcnp02, host-id: 2, engine-status: {reason: vm not running on this host, health: bad, vm: down, detail: unknown}, score: 0, stopped: True, maintenance: False, crc32: 25da07df, host-ts: 3030}</div><div>MainThread::INFO::2016-06-14 08:13:05,163::state_machine::171::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host njsevcnp03 (id 3): {hostname: njsevcnp03, host-id: 3, engine-status: {reason: vm not running on this host, health: bad, vm: down, detail: unknown}, score: 0, stopped: True, maintenance: False, crc32: c67818cb, host-ts: 10877406}</div><div><br></div></div><div><br></div><div>And that same log on node02 where the engine is running</div><div><br></div><div><br></div><div><div>MainThread::INFO::2016-06-14 08:15:44,451::state_machine::171::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host njsevcnp01 (id 1): {hostname: njsevcnp01, host-id: 1, engine-status: {reason: vm not running on this host, health: bad, vm: down, detail: unknown}, score: 0, stopped: True, maintenance: False, crc32: 260dbf06, host-ts: 327}</div><div>MainThread::INFO::2016-06-14 08:15:44,451::state_machine::171::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host njsevcnp03 (id 3): {hostname: njsevcnp03, host-id: 3, engine-status: {reason: vm not running on this host, health: bad, vm: down, detail: unknown}, score: 0, stopped: True, maintenance: False, crc32: c67818cb, host-ts: 10877406}</div><div>MainThread::INFO::2016-06-14 08:15:44,451::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 2): {engine-health: {health: good, vm: up, detail: up}, bridge: True, mem-free: 20702.0, maintenance: False, cpu-load: None, gateway: True}</div><div>MainThread::INFO::2016-06-14 08:15:44,452::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1465906544.45 type=state_transition detail=StartState-ReinitializeFSM hostname=njsevcnp02</div></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 14, 2016 at 7:59 AM, Martin Sivak <span dir="ltr"><<a href="mailto:msivak@redhat.com" target="_blank">msivak@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
is there anything interesting in the hosted engine log files?<br>
/var/log/ovirt-hosted-engine-ha/agent.log<br>
<br>
There should be something appearing there every 10 seconds or faster.<br>
<br>
What is the status of the hosted engine services? systemctl status<br>
ovirt-ha-agent ovirt-ha-broker<br>
<br>
<br>
Best regards<br>
<br>
--<br>
Martin Sivak<br>
SLA / oVirt<br>
<div><div class="h5"><br>
On Sat, Jun 11, 2016 at 8:53 PM, Charles Kozler <<a href="mailto:charles@fixflyer.com">charles@fixflyer.com</a>> wrote:<br>
> See linked images please. As you can see all three nodes are reporting stale<br>
> data. The results of this are:<br>
><br>
> 1. Not all VM's migrate seamlessly in the cluster. Sometimes I have to shut<br>
> them down to get them to be able to migrate again<br>
><br>
> 2. Hosted engine refuses to move due to constraints (image). This part<br>
> doesnt make sense to me because I can forcefully shut it down and then go<br>
> directly on a hosted engine node and bring it back up. Also, the Web UI<br>
> shows all nodes under the cluster except then it thinks its not apart of the<br>
> cluster<br>
><br>
> 3. Time is in sync (image)<br>
><br>
> 4. Storage is 100% fine. Gluster back end reports mirroring and status<br>
> 'started'. No split brain has occurred and ovirt nodes have never lost<br>
> connectivity to storage<br>
><br>
> 5. I reinstalled all three nodes. For some reason only node 3 still shows as<br>
> having updates available. (image). For clarity, I did not click "upgrade" I<br>
> simply did 'reinstall' from the Web UI. Having looked at the output and<br>
> yum.log from /var/log it almost looks like it did do an update. All package<br>
> versions across all three nodes are the same (respective to ovirt/vdsm)<br>
> (image). For some reason though<br>
> ovirt-engine-appliance-3.6-20160126.1.el7.centos.noarch exists on node 1 but<br>
> not on node 2 or 3. Could this be relative? I dont recall installing that<br>
> specifically on node 1 but I may have<br>
><br>
> Been slamming my head on this so I am hoping you can provide some assistance<br>
><br>
> <a href="http://imgur.com/a/6xkaS" rel="noreferrer" target="_blank">http://imgur.com/a/6xkaS</a><br>
><br>
> Thanks!<br>
><br>
> --<br>
><br>
> Charles Kozler<br>
> Vice President, IT Operations<br>
><br>
> FIX Flyer, LLC<br>
> 225 Broadway | Suite 1600 | New York, NY 10007<br>
> <a href="tel:1-888-349-3593" value="+18883493593">1-888-349-3593</a><br>
> <a href="http://www.fixflyer.com" rel="noreferrer" target="_blank">http://www.fixflyer.com</a><br>
><br>
> NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED RECIPIENT(S)<br>
> OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION WHICH IS<br>
> PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, DISTRIBUTION,<br>
> OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS INFORMATION IS<br>
> RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED RECIPIENT, PLEASE<br>
> CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS E-MAIL FROM YOUR<br>
> SYSTEM AND DESTROY ANY COPIES.<br>
><br>
</div></div>> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><br style="color:rgb(0,0,0);font-family:'Times New Roman';font-size:medium"><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><strong><span style="font-family:'times new roman',times,serif"><font size="2">Charles Kozler</font></span></strong></div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><em><span style="font-family:'times new roman',times,serif"><font size="2">Vice President, IT Operations</font></span></em></div><div style="color:rgb(0,0,0);font-family:'Times New Roman';font-size:medium"> </div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><span style="font-family:'times new roman',times,serif"><font size="2">FIX Flyer, LLC</font></span></div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><span style="font-family:'times new roman',times,serif"><font size="2">225 Broadway | Suite 1600 | New York, NY 10007</font></span></div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><span style="font-family:'times new roman',times,serif"><font size="2">1-888-349-3593</font></span></div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><a href="http://fixflyer.com" target="_blank"><span style="font-family:'times new roman',times,serif"><font size="2">http://www.fixflyer.com</font></span></a></div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><font size="2"> </font></div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><font size="1">NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION WHICH IS PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.</font></div></div></div></div></div></div></div></div>
</div>