<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">The issue was a split-brain issue on the dom_md/ids file causing an input/output error, thanks!</div><div class="gmail_extra"><br><div class="gmail_quote">

On Mon, Feb 3, 2014 at 10:43 PM, Andrew Lau <span dir="ltr">&lt;<a href="mailto:andrew@andrewklau.com" target="_blank">andrew@andrewklau.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

<div dir="ltr"><div><div class="h5"><div><div>On Mon, Feb 3, 2014 at 10:40 PM, Doron Fediuck <span dir="ltr">&lt;<a href="mailto:dfediuck@redhat.com" target="_blank">dfediuck@redhat.com</a>&gt;</span> wrote:<br>
</div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div class="h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">


<div><br>
<br>
----- Original Message -----<br>
&gt; From: &quot;Andrew Lau&quot; &lt;<a href="mailto:andrew@andrewklau.com" target="_blank">andrew@andrewklau.com</a>&gt;<br>
</div><div><div>&gt; To: &quot;Doron Fediuck&quot; &lt;<a href="mailto:dfediuck@redhat.com" target="_blank">dfediuck@redhat.com</a>&gt;<br>
&gt; Cc: &quot;users&quot; &lt;<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>&gt;, &quot;Jiri Moskovcak&quot; &lt;<a href="mailto:jmoskovc@redhat.com" target="_blank">jmoskovc@redhat.com</a>&gt;, &quot;Greg Padgett&quot; &lt;<a href="mailto:gpadgett@redhat.com" target="_blank">gpadgett@redhat.com</a>&gt;<br>



&gt; Sent: Monday, February 3, 2014 1:35:01 PM<br>
&gt; Subject: Re: [Users] Hosted Engine always reports &quot;unknown stale-data&quot;<br>
&gt;<br>
&gt; On Mon, Feb 3, 2014 at 9:53 PM, Doron Fediuck &lt;<a href="mailto:dfediuck@redhat.com" target="_blank">dfediuck@redhat.com</a>&gt; wrote:<br>
&gt;<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; ----- Original Message -----<br>
&gt; &gt; &gt; From: &quot;Andrew Lau&quot; &lt;<a href="mailto:andrew@andrewklau.com" target="_blank">andrew@andrewklau.com</a>&gt;<br>
&gt; &gt; &gt; To: &quot;users&quot; &lt;<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>&gt;<br>
&gt; &gt; &gt; Sent: Monday, February 3, 2014 12:32:45 PM<br>
&gt; &gt; &gt; Subject: [Users] Hosted Engine always reports &quot;unknown stale-data&quot;<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; Hi,<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; I was wondering if anyone has this same notice when they run:<br>
&gt; &gt; &gt; hosted-engine --vm-status<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; The &quot;engine status&quot; will always be &quot;unknown stale-data&quot; even when the VM<br>
&gt; &gt; is<br>
&gt; &gt; &gt; powered on and the engine is online. engine-health will actually report<br>
&gt; &gt; the<br>
&gt; &gt; &gt; correct status.<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; eg.<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; --== Host 1 status ==--<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; Status up-to-date : False<br>
&gt; &gt; &gt; Hostname : 172.16.0.11<br>
&gt; &gt; &gt; Host ID : 1<br>
&gt; &gt; &gt; Engine status : unknown stale-data<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; Is it some sort of blocked port causing this or is this by design?<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; Thanks,<br>
&gt; &gt; &gt; Andrew<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; _______________________________________________<br>
&gt; &gt; &gt; Users mailing list<br>
&gt; &gt; &gt; <a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
&gt; &gt; &gt; <a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
&gt; &gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; Hi Andrew,<br>
&gt; &gt; it looks like an issue with the time stamp.<br>
&gt; &gt; Which time stamp do you have? How relevant is it?<br>
&gt; &gt;<br>
&gt;<br>
&gt; timestamps seem to be outdated by a lot, interesting error in the broker.log<br>
&gt;<br>
&gt; Thread-24::INFO::2014-02-03<br>
&gt; 22:33:14,801::engine_health::90::engine_health.CpuLoadNoEngine::(action) VM<br>
&gt; not running on this host, status down<br>
&gt; Thread-22::INFO::2014-02-03<br>
&gt; 22:33:14,834::mem_free::53::mem_free.MemFree::(action) memFree: 27382<br>
&gt; Thread-23::ERROR::2014-02-03<br>
&gt; 22:33:14,922::cpu_load_no_engine::156::cpu_load_no_engine.EngineHealth::(update_stat_file)<br>
&gt; Failed to getVmStats: &#39;pid&#39;<br>
&gt; Thread-23::INFO::2014-02-03<br>
&gt; 22:33:14,923::cpu_load_no_engine::121::cpu_load_no_engine.EngineHealth::(calculate_load)<br>
&gt; System load total=0.0124, engine=0.0000, non-engine=0.0124<br>
&gt;<br>
&gt; I&#39;m assuming that update_stat_file is the metadata file the vm-status is<br>
&gt; getting pulled from?<br>
&gt;<br>
<br>
</div></div>Yep.<br>
Can you please verify the time your host actually has?<br>
ie- we have a known issue with time, since we assume all<br>
hosts are in sync. So if one of your hosts has a time sync<br>
issue, this can explain the problem you see.<br></blockquote><div><br></div><div><font face="tahoma, sans-serif">--== Host 1 status ==--</font></div><div><font face="tahoma, sans-serif"><br>
</font></div><div><font face="tahoma, sans-serif">Status up-to-date                  : False</font></div><div><font face="tahoma, sans-serif">Hostname                           : 172.16.0.11</font></div>
<div><font face="tahoma, sans-serif">Host ID                            : 1</font></div><div><font face="tahoma, sans-serif">Engine status                      : unknown stale-data</font></div>
</div></div><div><font face="tahoma, sans-serif">Score                              : 0</font></div><div><font face="tahoma, sans-serif">Local maintenance                  : False</font></div>
<div><font face="tahoma, sans-serif">Host timestamp                     : 1391417611</font></div><div><font face="tahoma, sans-serif"><br></font></div><div>
<font face="tahoma, sans-serif"><div class="im"><div>--== Host 2 status ==--</div><div><br></div><div>Status up-to-date                  : False</div></div><div>
Hostname                           : 172.16.0.12</div><div>Host ID                            : 2</div><div>Engine status                      : unknown stale-data</div><div>
Score                              : 0</div><div>Local maintenance                  : False</div><div>Host timestamp                     : 1391417171</div><div><br></div></font></div>
<div style="font-family:tahoma,sans-serif"><br></div><div><div><font face="tahoma, sans-serif"><div style="font-family:tahoma,sans-serif;display:inline">​</div></font>[root@hv01 ~]# date +%s                                                                     │[root@hv02 ~]# date +%s</div>


<div><div style="font-family:tahoma,sans-serif;display:inline">​​</div>1391427754                                                                                  │139142775<div style="font-family:tahoma,sans-serif;display:inline">


​5​</div></div></div><div><br></div><div style="font-family:tahoma,sans-serif">​​</div></div><br></div></div>
</blockquote></div><br></div></div>