Re: [ovirt-users] hosted-engine unknow stale-data

This is a multi-part message in MIME format. ------------160fee5d54b636727eac37286c Content-Type: text/plain; format=flowed; charset="UTF-8" Content-Transfer-Encoding: 8bit Why are both hosts reporting as ovirt 1? Look at the hostname fields to see what mean. -derek Sent using my mobile device. Please excuse any typos. On January 16, 2018 7:11:09 AM Artem Tambovskiy <artem.tambovskiy(a)gmail.com&gt; wrote: ------------160fee5d54b636727eac37286c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <html<head</head<body<div style=3D"color: black;"<div style=3D"color: black;"<p style=3D"margin: 0 0 1em 0; color: black;">Why are both hosts reporting = as ovirt 1? <brLook at the hostname fields to see what mean. </p<p style=3D"margin: 0 0 1em 0; color: black;">-derek<brSent using my mobile device. Please excuse any typos. <br</p</div<div style=3D"color: black;"<p style=3D"color: black; font-size: 10pt; font-family: Arial, sans-serif; = margin: 10pt 0;">On January 16, 2018 7:11:09 AM Artem Tambovskiy &lt;artem.= tambovskiy(a)gmail.com&amp;gt; wrote:</p<blockquote type=3D"cite" class=3D"gmail_quote" style=3D"margin: 0 0 0 0.75= ex; border-left: 1px solid #808080; padding-left: 0.75ex;"<div dir=3D"ltr">Hello,<br><br>Yes, I followed exactly the same procedure w= hile reinstalling the hosts (the only difference that I have SSH key config= ured instead of the password).=C2=A0<div><br></div><div>Just reinstalled th= e second host one more time, after 20 min the host still haven&#39;t reache= d active score of 3400 (Hosted Engine HA:Not Active) and I still don&#39;t = see crown icon for this host.=C2=A0<br><br>hosted-engine --vm-status =C2=A0= from ovirt1 host=C2=A0<br><br><div>[root@ovirt1 ~]# hosted-engine --vm-stat= us</div><div><br></div><div><br></div><div>--=3D=3D Host 1 status =3D=3D--<= /div><div><br></div><div>conf_on_shared_storage =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 : True</div><div>Status up-to-date =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: True</div><div>Hostname =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 : <a href=3D"http://ovirt1.telia.ru">ovirt1.telia.ru</a></div><div>H= ost ID =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 1</div><div>Engine status =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: {&quot;heal= th&quot;: &quot;good&quot;, &quot;vm&quot;: &quot;up&quot;, &quot;detail&qu= ot;: &quot;up&quot;}</div><div>Score =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 3400</d= iv><div>stopped =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: False</div><div>Local maintenance = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: False</div>= <div>crc32 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 3f94156a</div><div>local_conf_ti= mestamp =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 349144</div><div= =A0 =C2=A0 : 349144</div><div>Extra metadata (valid at timestamp):</div><di= v>=C2=A0 =C2=A0 =C2=A0 =C2=A0 metadata_parse_version=3D1</div><div>=C2=A0 = =C2=A0 =C2=A0 =C2=A0 metadata_feature_version=3D1</div><div>=C2=A0 =C2=A0 = =C2=A0 =C2=A0 timestamp=3D349144 (Tue Jan 16 15:03:45 2018)</div><div>=C2= =A0 =C2=A0 =C2=A0 =C2=A0 host-id=3D1</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 = score=3D3400</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 vm_conf_refresh_time=3D3= 49144 (Tue Jan 16 15:03:45 2018)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 conf= _on_shared_storage=3DTrue</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 maintenance= =3DFalse</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 state=3DEngineUp</div><div>= =C2=A0 =C2=A0 =C2=A0 =C2=A0 stopped=3DFalse</div><div><br></div><div><br></= div><div>--=3D=3D Host 2 status =3D=3D--</div><div><br></div><div>conf_on_s= hared_storage =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : True</div><div>St= atus up-to-date =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0: False</div><div>Hostname =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : <a href=3D"http://ovirt1.te= lia.ru">ovirt1.telia.ru</a></div><div>Host ID =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 2</d= iv><div>Engine status =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0: unknown stale-data</div><div>Score =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0: 0</div><div>stopped =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: True</div><= div>Local maintenance =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0: False</div><div>crc32 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: c7037c03</= div><div>local_conf_timestamp =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 : 7530</div><div>Host timestamp =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 7530</div><div>Extra metadata (valid at t= imestamp):</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 metadata_parse_version=3D1= </div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 metadata_feature_version=3D1</div><d= iv>=C2=A0 =C2=A0 =C2=A0 =C2=A0 timestamp=3D7530 (Fri Jan 12 16:10:12 2018)<= /div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 host-id=3D2</div><div>=C2=A0 =C2=A0 = =C2=A0 =C2=A0 score=3D0</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 vm_conf_refre= sh_time=3D7530 (Fri Jan 12 16:10:12 2018)</div><div>=C2=A0 =C2=A0 =C2=A0 = =C2=A0 conf_on_shared_storage=3DTrue</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 = maintenance=3DFalse</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 state=3DAgentStop= ped</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 stopped=3DTrue</div><br><br>hoste= d-engine --vm-status output from ovirt2 host=C2=A0</div><div><br><div>[root= @ovirt2 ovirt-hosted-engine-ha]# hosted-engine --vm-status</div><div><br></= div><div><br></div><div>--=3D=3D Host 1 status =3D=3D--</div><div><br></div= ue</div><div>Status up-to-date =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0: False</div><div>Hostname =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : <a href=3D= "http://ovirt1.telia.ru">ovirt1.telia.ru</a></div><div>Host ID =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0: 1</div><div>Engine status =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: unknown stale-data</div><div>Sc= ore =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 3400</div><div>stopped =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0: False</div><div>Local maintenance =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: False</div><div>crc32 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0: 6d3606f1</div><div>local_conf_timestamp =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 349264</div><div>Host timestamp =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 349264</div><= div>Extra metadata (valid at timestamp):</div><div>=C2=A0 =C2=A0 =C2=A0 =C2= =A0 metadata_parse_version=3D1</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 metada= ta_feature_version=3D1</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 timestamp=3D34= 9264 (Tue Jan 16 15:05:45 2018)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 host-= id=3D1</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 score=3D3400</div><div>=C2=A0 = =C2=A0 =C2=A0 =C2=A0 vm_conf_refresh_time=3D349264 (Tue Jan 16 15:05:45 201= 8)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 conf_on_shared_storage=3DTrue</div= =A0 =C2=A0 =C2=A0 state=3DEngineUp</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 st= opped=3DFalse</div><div><br></div><div><br></div><div>--=3D=3D Host 2 statu= s =3D=3D--</div><div><br></div><div>conf_on_shared_storage =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 : True</div><div>Status up-to-date =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: False</div><div>Hostn= ame =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 : <a href=3D"http://ovirt1.telia.ru">ovirt1.telia.ru</= a></div><div>Host ID =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 2</div><div>Engine status = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0: unknown stale-data</div><div>Score =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 0</d= iv><div>stopped =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: True</div><div>Local maintenance = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: False</div>= <div>crc32 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: c7037c03</div><div>local_conf_ti= mestamp =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 7530</div><div>H= ost timestamp =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 : 7530</div><div>Extra metadata (valid at timestamp):</div><div>= =C2=A0 =C2=A0 =C2=A0 =C2=A0 metadata_parse_version=3D1</div><div>=C2=A0 =C2= =A0 =C2=A0 =C2=A0 metadata_feature_version=3D1</div><div>=C2=A0 =C2=A0 =C2= =A0 =C2=A0 timestamp=3D7530 (Fri Jan 12 16:10:12 2018)</div><div>=C2=A0 =C2= =A0 =C2=A0 =C2=A0 host-id=3D2</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 score= =3D0</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 vm_conf_refresh_time=3D7530 (Fri= Jan 12 16:10:12 2018)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 conf_on_shared= _storage=3DTrue</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 maintenance=3DFalse</= div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 state=3DAgentStopped</div><div>=C2=A0 = =C2=A0 =C2=A0 =C2=A0 stopped=3DTrue</div><div><br></div><div><br></div><div= &quot;Host <a href=3D"http://ovirt2.telia.ru">ovirt2.telia.ru</a> has time-= drift of 5305 seconds while maximum configured value is 300 seconds.&quot; = that is a bit weird as haven&#39;t touched any time settings since I instal= led the cluster.=C2=A0<br>both host have the same time and timezone (MSK) b= ut hosted engine lives in UTC timezone. Is it mandatory to have everything = in sync and in the same timezone?</div><div><br></div>Regards,</div><div>Ar= tem<br><br><br><br><br><br></div><div class=3D"gmail_extra"><br><div class= =3D"gmail_quote">On Tue, Jan 16, 2018 at 2:20 PM, Kasturi Narra <span dir= =3D"ltr">&lt;<a href=3D"mailto:knarra@redhat.com" target=3D"_blank">knarra@= redhat.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style= =3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding= -left:1ex"><div dir=3D"ltr">Hello,<div>=C2=A0=C2=A0</div><div>=C2=A0 =C2=A0= =C2=A0I now see that your hosted engine is up and running. Can you let me = know how did you try reinstalling the host? Below is the procedure which is= used and hope you did not miss any step while reinstalling. If no, can you= try reinstalling again and see if that works ?</div><div><br></div><div>1)= Move the host to maintenance</div><div>2) click on reinstall</div><div>3) = provide the password</div><div>4) uncheck &#39;automatically configure host= firewall&#39;</div><div>5) click on &#39;Deploy&#39; tab</div><div>6) clic= k Hosted Engine deployment as &#39;Deploy&#39;</div><div><br></div><div>And= once the host installation is done, wait till the active score of the host= shows 3400 in the general tab then check hosted-engine --vm-status.=C2=A0<= /div><div><br></div><div>Thanks</div><span class=3D"gmail-HOEnZb"><font col= or=3D"#888888"><div>kasturi</div></font></span></div><div class=3D"gmail-HO= EnZb"><div class=3D"gmail-h5"><div class=3D"gmail_extra"><br><div class=3D"= gmail_quote">On Mon, Jan 15, 2018 at 4:57 PM, Artem Tambovskiy <span dir=3D= "ltr">&lt;<a href=3D"mailto:artem.tambovskiy@gmail.com" target=3D"_blank">a= rtem.tambovskiy(a)gmail.com&lt;/a&gt;&amp;gt;&lt;/span&gt; wrote:<br><blockquote class=3D"gma= il_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,2= 04,204);padding-left:1ex"><div dir=3D"ltr">Hello,<br><br>I have uploaded 2 = archives with all relevant logs to shared hosting<br>files from host 1 =C2= =A0(which is currently running all VM&#39;s including hosted_engine) =C2=A0= - =C2=A0<span style=3D"color:rgb(0,0,0);font-family:Arial,sans-serif;font-s= ize:15.0016px"><a href=3D"https://yadi.sk/d/PttRoYV63RTvhK" target=3D"_blan= k">https://yadi.sk/d/PttRoYV63RT<wbr>vhK</a><br>files from second host -=C2= =A0</span><span style=3D"color:rgb(0,0,0);font-family:Arial,sans-serif;font= -size:15.0016px"><a href=3D"https://yadi.sk/d/UBducEsV3RTvhc" target=3D"_bl= ank">https://yadi.sk/d/UBducEsV3R<wbr>Tvhc</a>=C2=A0<br></span><span style= =3D"color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:15.0016px"><br>= I have tried to restart both=C2=A0</span>ovirt-ha-agent and ovirt-ha-broker= but it gives no effect. I have also tried to shutdown hosted_engine VM, st= op ovirt-ha-agent and ovirt-ha-broker =C2=A0services disconnect storage and= connect it again =C2=A0- no effect as well.=C2=A0<br>Also I tried to reins= tall second host from WebGUI - this lead to the interesting situation - now= =C2=A0=C2=A0hosted-engine --vm-status =C2=A0shows that both hosts have the = same address.=C2=A0<div><br></div><div><span><div>[root@ovirt1 ~]# hosted-e= ngine --vm-status =C2=A0 =C2=A0</div><div><br></div><div>--=3D=3D Host 1 st= atus =3D=3D--</div><div><br></div><div>conf_on_shared_storage =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 : True</div></span><div>Status up-to-date =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: True</div><div= =C2=A0 =C2=A0 =C2=A0 =C2=A0 : <a href=3D"http://ovirt1.telia.ru" target=3D"= _blank">ovirt1.telia.ru</a></div><div>Host ID =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 1</d= iv><div>Engine status =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0: {&quot;health&quot;: &quot;good&quot;, &quot;vm&q= uot;: &quot;up&quot;, &quot;detail&quot;: &quot;up&quot;}</div><div>Score = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 3400</div><span><div>stopped =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0: False</div><div>Local maintenance =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: False</div></span><div>crc32 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0: a7758085</div><div>local_conf_timestamp =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 259327</div><div>Host timestamp =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 259327= </div><span><div>Extra metadata (valid at timestamp):</div><div>=C2=A0 =C2= =A0 =C2=A0 =C2=A0 metadata_parse_version=3D1</div><div>=C2=A0 =C2=A0 =C2=A0= =C2=A0 metadata_feature_version=3D1</div></span><div>=C2=A0 =C2=A0 =C2=A0 = =C2=A0 timestamp=3D259327 (Mon Jan 15 14:06:48 2018)</div><div>=C2=A0 =C2= =A0 =C2=A0 =C2=A0 host-id=3D1</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 score= =3D3400</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 vm_conf_refresh_time=3D259327= (Mon Jan 15 14:06:48 2018)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 conf_on_s= hared_storage=3DTrue</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 maintenance=3DFa= lse</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 state=3DEngineUp</div><div>=C2=A0= =C2=A0 =C2=A0 =C2=A0 stopped=3DFalse</div><span><div><br></div><div><br></= div><div>--=3D=3D Host 2 status =3D=3D--</div><div><br></div><div>conf_on_s= hared_storage =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : True</div><div>St= atus up-to-date =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0: False</div><div>Hostname =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : <a href=3D"http://ovirt1.te= lia.ru" target=3D"_blank">ovirt1.telia.ru</a></div><div>Host ID =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0: 2</div><div>Engine status =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: unknown stale-data</div><div>Sc= ore =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 0</div><div>stopped =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0: True</div><div>Local maintenance =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0: False</div><div>crc32 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0: c7037c03</div><div>local_conf_timestamp =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 7530</div><div>Host timestamp =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 7530</div><div>Ex= tra metadata (valid at timestamp):</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 me= tadata_parse_version=3D1</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 metadata_fea= ture_version=3D1</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 timestamp=3D7530 (Fr= i Jan 12 16:10:12 2018)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 host-id=3D2</= div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 score=3D0</div><div>=C2=A0 =C2=A0 =C2= =A0 =C2=A0 vm_conf_refresh_time=3D7530 (Fri Jan 12 16:10:12 2018)</div><div= =C2=A0 =C2=A0 =C2=A0 maintenance=3DFalse</div><div>=C2=A0 =C2=A0 =C2=A0 = =C2=A0 state=3DAgentStopped</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 stopped= =3DTrue</div><div><br></div></span><div>Gluster seems working fine. all glu= ster nodes showing connected state.</div><div><br></div><div>Any advises on= how to resolve this situation are highly appreciated!<br><br></div><div>Re= gards,</div><div>Artem</div><div><br></div></div></div><div class=3D"gmail-= m_4720594609300657870HOEnZb"><div class=3D"gmail-m_4720594609300657870h5"><= div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Mon, Jan 15, 20= 18 at 11:45 AM, Kasturi Narra <span dir=3D"ltr">&lt;<a href=3D"mailto:knarr= a(a)redhat.com&quot; target=3D&quot;_blank&quot;&gt;knarra(a)redhat.com&lt;/a&gt;&amp;gt;&lt;/span&gt; wrote:<br>= <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr">Hello Ar= tem,<div><br></div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 Can you check if gluste= rd service is running on host1 and all the peers are in connected state ? I= f yes, can you restart ovirt-ha-agent and broker services and check if thin= gs are working fine ?</div><div><br></div><div>Thanks</div><span class=3D"g= mail-m_4720594609300657870m_-3199546786741141787HOEnZb"><font color=3D"#888= 888"><div>kasturi</div></font></span></div><div class=3D"gmail-m_4720594609= 300657870m_-3199546786741141787HOEnZb"><div class=3D"gmail-m_47205946093006= 57870m_-3199546786741141787h5"><div class=3D"gmail_extra"><br><div class=3D= "gmail_quote">On Sat, Jan 13, 2018 at 12:33 AM, Artem Tambovskiy <span dir= =3D"ltr">&lt;<a href=3D"mailto:artem.tambovskiy@gmail.com" target=3D"_blank= &quot;&gt;artem.tambovskiy(a)gmail.com&lt;/a&gt;&amp;gt;&lt;/span&gt; wrote:<br><blockquote class=3D"= gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(20= 4,204,204);padding-left:1ex"><div dir=3D"ltr">Explored logs on both hosts.= =C2=A0<div>broker.log shows no errors.</div><div><br></div><div>agent.log l= ooking not good:<br><br>on host1 (which running hosted engine) :</div><div>= <br></div><div><div>MainThread::ERROR::2018-01-12 21:51:03,883::agent::205:= :ovir<wbr>t_hosted_engine_ha.agent.agent<wbr>.Agent::(_run_agent) Traceback= (most recent call last):</div><div>=C2=A0 File &quot;/usr/lib/python2.7/si= te-packa<wbr>ges/ovirt_hosted_engine_ha/age<wbr>nt/agent.py&quot;, line 191= , in _run_agent</div><div>=C2=A0 =C2=A0 return action(he)</div><div>=C2=A0 = File &quot;/usr/lib/python2.7/site-packa<wbr>ges/ovirt_hosted_engine_ha/age= <wbr>nt/agent.py&quot;, line 64, in action_proper</div><div>=C2=A0 =C2=A0 r= eturn he.start_monitoring()</div><div>=C2=A0 File &quot;/usr/lib/python2.7/= site-packa<wbr>ges/ovirt_hosted_engine_ha/age<wbr>nt/hosted_engine.py&quot;= , line 411, in start_monitoring</div><div>=C2=A0 =C2=A0 self._initialize_sa= nlock()</div><div>=C2=A0 File &quot;/usr/lib/python2.7/site-packa<wbr>ges/o= virt_hosted_engine_ha/age<wbr>nt/hosted_engine.py&quot;, line 749, in _init= ialize_sanlock</div><div>=C2=A0 =C2=A0 &quot;Failed to initialize sanlock, = the number of errors has&quot;</div><div>SanlockInitializationError: Failed= to initialize sanlock, the number of errors has exceeded the limit</div><d= iv><br></div><div>MainThread::ERROR::2018-01-12 21:51:03,884::agent::206::o= vir<wbr>t_hosted_engine_ha.agent.agent<wbr>.Agent::(_run_agent) Trying to r= estart agent</div><div>MainThread::WARNING::2018-01-1<wbr>2 21:51:08,889::a= gent::209::ovir<wbr>t_hosted_engine_ha.agent.agent<wbr>.Agent::(_run_agent)= Restarting agent, attempt &#39;1&#39;</div><div>MainThread::INFO::2018-01-= 12 21:51:08,919::hosted_engine::2<wbr>42::ovirt_hosted_engine_ha.age<wbr>nt= hosted_engine.HostedEngine:<wbr>:(_get_hostname) Found certificate common = name: <a href=3D"http://ovirt1.telia.ru" target=3D"_blank">ovirt1.telia.ru<= /a></div><div>MainThread::INFO::2018-01-12 21:51:08,921::hosted_engine::6<w= br>04::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(= _initialize_vdsm) Initializing VDSM</div><div>MainThread::INFO::2018-01-12 = 21:51:11,398::hosted_engine::6<wbr>30::ovirt_hosted_engine_ha.age<wbr>nt.ho= sted_engine.HostedEngine:<wbr>:(_initialize_storage_images) Connecting the = storage</div><div>MainThread::INFO::2018-01-12 21:51:11,399::storage_server= ::<wbr>220::<a href=3D"http://ovirt_hosted_engine_ha.li" target=3D"_blank">= ovirt_hosted_engine_ha.li</a><wbr>b.storage_server.StorageServer<wbr>::(val= idate_storage_server) Validating storage server</div><div>MainThread::INFO:= :2018-01-12 21:51:13,725::storage_server::<wbr>239::<a href=3D"http://ovirt= _hosted_engine_ha.li" target=3D"_blank">ovirt_hosted_engine_ha.li</a><wbr>b= storage_server.StorageServer<wbr>::(connect_storage_server) Connecting sto= rage server</div><div>MainThread::INFO::2018-01-12 21:51:18,390::storage_se= rver::<wbr>246::<a href=3D"http://ovirt_hosted_engine_ha.li" target=3D"_bla= nk">ovirt_hosted_engine_ha.li</a><wbr>b.storage_server.StorageServer<wbr>::= (connect_storage_server) Connecting storage server</div><div>MainThread::IN= FO::2018-01-12 21:51:18,423::storage_server::<wbr>253::<a href=3D"http://ov= irt_hosted_engine_ha.li" target=3D"_blank">ovirt_hosted_engine_ha.li</a><wb= r>b.storage_server.StorageServer<wbr>::(connect_storage_server) Refreshing = the storage domain</div><div>MainThread::INFO::2018-01-12 21:51:18,689::hos= ted_engine::6<wbr>63::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.Hoste= dEngine:<wbr>:(_initialize_storage_images) Preparing images</div><div>MainT= hread::INFO::2018-01-12 21:51:18,690::image::126::ovir<wbr>t_hosted_engine_= ha.lib.image.I<wbr>mage::(prepare_images) Preparing images</div><div>MainTh= read::INFO::2018-01-12 21:51:21,895::hosted_engine::6<wbr>66::ovirt_hosted_= engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_initialize_storage_= images) Refreshing vm.conf</div><div>MainThread::INFO::2018-01-12 21:51:21,= 895::config::493::ovi<wbr>rt_hosted_engine_ha.agent.host<wbr>ed_engine.Host= edEngine.config:<wbr>:(refresh_vm_conf) Reloading vm.conf from the shared s= torage domain</div><div>MainThread::INFO::2018-01-12 21:51:21,896::config::= 416::ovi<wbr>rt_hosted_engine_ha.agent.host<wbr>ed_engine.HostedEngine.conf= ig:<wbr>:(_get_vm_conf_content_from_ov<wbr>f_store) Trying to get a fresher= copy of vm configuration from the OVF_STORE</div><div>MainThread::INFO::20= 18-01-12 21:51:21,896::ovf_store::132::<wbr>ovirt_hosted_engine_ha.lib.ovf<= wbr>.ovf_store.OVFStore::(getEngin<wbr>eVMOVF) Extracting Engine VM OVF fro= m the OVF_STORE</div><div>MainThread::INFO::2018-01-12 21:51:21,897::ovf_st= ore::134::<wbr>ovirt_hosted_engine_ha.lib.ovf<wbr>.ovf_store.OVFStore::(get= Engin<wbr>eVMOVF) OVF_STORE volume path: /var/run/vdsm/storage/4a7f8717<wbr= c/8048cbd7-77e2-4805-9af4-<wbr>d109fa36dfcf=C2=A0</div><div>MainThread::INF= O::2018-01-12 21:51:21,915::config::435::ovi<wbr>rt_hosted_engine_ha.agent.= host<wbr>ed_engine.HostedEngine.config:<wbr>:(_get_vm_conf_content_from_ov<= wbr>f_store) Found an OVF for HE VM, trying to convert</div><div>MainThread= ::INFO::2018-01-12 21:51:21,918::config::440::ovi<wbr>rt_hosted_engine_ha.a= gent.host<wbr>ed_engine.HostedEngine.config:<wbr>:(_get_vm_conf_content_fro= m_ov<wbr>f_store) Got vm.conf from OVF_STORE</div><div>MainThread::INFO::20= 18-01-12 21:51:21,919::hosted_engine::5<wbr>09::ovirt_hosted_engine_ha.age<= wbr>nt.hosted_engine.HostedEngine:<wbr>:(_initialize_broker) Initializing h= a-broker connection</div><div>MainThread::INFO::2018-01-12 21:51:21,919::br= okerlink::130:<wbr>:<a href=3D"http://ovirt_hosted_engine_ha.lib.br" target= =3D"_blank">ovirt_hosted_engine_ha.lib.br</a><wbr>okerlink.BrokerLink::(sta= rt_mo<wbr>nitor) Starting monitor ping, options {&#39;addr&#39;: &#39;80.23= 9.162.97&#39;}</div><div>MainThread::INFO::2018-01-12 21:51:21,922::brokerl= ink::141:<wbr>:<a href=3D"http://ovirt_hosted_engine_ha.lib.br" target=3D"_= blank">ovirt_hosted_engine_ha.lib.br</a><wbr>okerlink.BrokerLink::(start_mo= <wbr>nitor) Success, id 140547104457680</div><div>MainThread::INFO::2018-01= -12 21:51:21,922::brokerlink::130:<wbr>:<a href=3D"http://ovirt_hosted_engi= ne_ha.lib.br" target=3D"_blank">ovirt_hosted_engine_ha.lib.br</a><wbr>okerl= ink.BrokerLink::(start_mo<wbr>nitor) Starting monitor mgmt-bridge, options = {&#39;use_ssl&#39;: &#39;true&#39;, &#39;bridge_name&#39;: &#39;ovirtmgmt&#= 39;, &#39;address&#39;: &#39;0&#39;}</div><div>MainThread::INFO::2018-01-12= 21:51:21,936::brokerlink::141:<wbr>:<a href=3D"http://ovirt_hosted_engine_= ha.lib.br" target=3D"_blank">ovirt_hosted_engine_ha.lib.br</a><wbr>okerlink= BrokerLink::(start_mo<wbr>nitor) Success, id 140547104458064</div><div>Mai= nThread::INFO::2018-01-12 21:51:21,936::brokerlink::130:<wbr>:<a href=3D"ht= tp://ovirt_hosted_engine_ha.lib.br" target=3D"_blank">ovirt_hosted_engine_h= a.lib.br</a><wbr>okerlink.BrokerLink::(start_mo<wbr>nitor) Starting monitor= mem-free, options {&#39;use_ssl&#39;: &#39;true&#39;, &#39;address&#39;: &= #39;0&#39;}</div><div>MainThread::INFO::2018-01-12 21:51:21,938::brokerlink= ::141:<wbr>:<a href=3D"http://ovirt_hosted_engine_ha.lib.br" target=3D"_bla= nk">ovirt_hosted_engine_ha.lib.br</a><wbr>okerlink.BrokerLink::(start_mo<wb= r>nitor) Success, id 140547104458448</div><div>MainThread::INFO::2018-01-12= 21:51:21,939::brokerlink::130:<wbr>:<a href=3D"http://ovirt_hosted_engine_= ha.lib.br" target=3D"_blank">ovirt_hosted_engine_ha.lib.br</a><wbr>okerlink= BrokerLink::(start_mo<wbr>nitor) Starting monitor cpu-load-no-engine, opti= ons {&#39;use_ssl&#39;: &#39;true&#39;, &#39;vm_uuid&#39;: &#39;b366e466-b0= ea-4a09-866b-d0248<wbr>d7523a6&#39;, &#39;address&#39;: &#39;0&#39;}</div><= div>MainThread::INFO::2018-01-12 21:51:21,940::brokerlink::141:<wbr>:<a hre= f=3D"http://ovirt_hosted_engine_ha.lib.br" target=3D"_blank">ovirt_hosted_e= ngine_ha.lib.br</a><wbr>okerlink.BrokerLink::(start_mo<wbr>nitor) Success, = id 140547104457552</div><div>MainThread::INFO::2018-01-12 21:51:21,941::bro= kerlink::130:<wbr>:<a href=3D"http://ovirt_hosted_engine_ha.lib.br" target= =3D"_blank">ovirt_hosted_engine_ha.lib.br</a><wbr>okerlink.BrokerLink::(sta= rt_mo<wbr>nitor) Starting monitor engine-health, options {&#39;use_ssl&#39;= : &#39;true&#39;, &#39;vm_uuid&#39;: &#39;b366e466-b0ea-4a09-866b-d0248<wbr= 2018-01-12 21:51:21,942::brokerlink::141:<wbr>:<a href=3D"http://ovirt_host= ed_engine_ha.lib.br" target=3D"_blank">ovirt_hosted_engine_ha.lib.br</a><wb= r>okerlink.BrokerLink::(start_mo<wbr>nitor) Success, id 140547104459792</di= v><div>MainThread::INFO::2018-01-12 21:51:26,951::brokerlink::179:<wbr>:<a = href=3D"http://ovirt_hosted_engine_ha.lib.br" target=3D"_blank">ovirt_hoste= d_engine_ha.lib.br</a><wbr>okerlink.BrokerLink::(set_stor<wbr>age_domain) S= uccess, id 140546772847056</div><div>MainThread::INFO::2018-01-12 21:51:26,= 952::hosted_engine::6<wbr>01::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engi= ne.HostedEngine:<wbr>:(_initialize_broker) Broker initialized, all submonit= ors started</div><div>MainThread::INFO::2018-01-12 21:51:27,049::hosted_eng= ine::7<wbr>04::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine= :<wbr>:(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, ho= st id 1 is acquired (file: /var/run/vdsm/storage/4a7f8717<wbr>-9bb0-4d80-80= 16-498fa4b88162/0<wbr>93faa75-5e33-4559-84fa-1f1f8d4<wbr>8153b/911c7637-b49= d-463e-b186-<wbr>23b404e50769)</div><div>MainThread::INFO::2018-01-12 21:53= :48,067::hosted_engine::7<wbr>45::ovirt_hosted_engine_ha.age<wbr>nt.hosted_= engine.HostedEngine:<wbr>:(_initialize_sanlock) Failed to acquire the lock.= Waiting &#39;5&#39;s before the next attempt</div><div>MainThread::INFO::2= 018-01-12 21:56:14,088::hosted_engine::7<wbr>45::ovirt_hosted_engine_ha.age= <wbr>nt.hosted_engine.HostedEngine:<wbr>:(_initialize_sanlock) Failed to ac= quire the lock. Waiting &#39;5&#39;s before the next attempt</div><div>Main= Thread::INFO::2018-01-12 21:58:40,111::hosted_engine::7<wbr>45::ovirt_hoste= d_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_initialize_sanloc= k) Failed to acquire the lock. Waiting &#39;5&#39;s before the next attempt= </div><div>MainThread::INFO::2018-01-12 22:01:06,133::hosted_engine::7<wbr>= 45::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_in= itialize_sanlock) Failed to acquire the lock. Waiting &#39;5&#39;s before t= he next attempt</div></div><div><br><br>agent.log from second host=C2=A0<br= 0::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_ini= tialize_storage_images) Connecting the storage</div><div>MainThread::INFO::= 2018-01-12 22:01:37,242::storage_server::<wbr>220::<a href=3D"http://ovirt_= hosted_engine_ha.li" target=3D"_blank">ovirt_hosted_engine_ha.li</a><wbr>b.= storage_server.StorageServer<wbr>::(validate_storage_server) Validating sto= rage server</div><div>MainThread::INFO::2018-01-12 22:01:39,540::hosted_eng= ine::6<wbr>39::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine= :<wbr>:(_initialize_storage_images) Storage domain reported as valid and re= connect is not forced.</div><div>MainThread::INFO::2018-01-12 22:01:41,939:= :hosted_engine::4<wbr>53::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.H= ostedEngine:<wbr>:(start_monitoring) Current state EngineUnexpectedlyDown (= score: 0)</div><div>MainThread::INFO::2018-01-12 22:01:52,150::config::493:= :ovi<wbr>rt_hosted_engine_ha.agent.host<wbr>ed_engine.HostedEngine.config:<= wbr>:(refresh_vm_conf) Reloading vm.conf from the shared storage domain</di= v><div>MainThread::INFO::2018-01-12 22:01:52,150::config::416::ovi<wbr>rt_h= osted_engine_ha.agent.host<wbr>ed_engine.HostedEngine.config:<wbr>:(_get_vm= _conf_content_from_ov<wbr>f_store) Trying to get a fresher copy of vm confi= guration from the OVF_STORE</div><div>MainThread::INFO::2018-01-12 22:01:52= ,151::ovf_store::132::<wbr>ovirt_hosted_engine_ha.lib.ovf<wbr>.ovf_store.OV= FStore::(getEngin<wbr>eVMOVF) Extracting Engine VM OVF from the OVF_STORE</= div><div>MainThread::INFO::2018-01-12 22:01:52,153::ovf_store::134::<wbr>ov= irt_hosted_engine_ha.lib.ovf<wbr>.ovf_store.OVFStore::(getEngin<wbr>eVMOVF)= OVF_STORE volume path: /var/run/vdsm/storage/4a7f8717<wbr>-9bb0-4d80-8016-= 498fa4b88162/5<wbr>cabd8e1-5f4b-469e-becc-227469e<wbr>03f5c/8048cbd7-77e2-4= 805-9af4-<wbr>d109fa36dfcf=C2=A0</div><div>MainThread::INFO::2018-01-12 22:= 01:52,174::config::435::ovi<wbr>rt_hosted_engine_ha.agent.host<wbr>ed_engin= e.HostedEngine.config:<wbr>:(_get_vm_conf_content_from_ov<wbr>f_store) Foun= d an OVF for HE VM, trying to convert</div><div>MainThread::INFO::2018-01-1= 2 22:01:52,179::config::440::ovi<wbr>rt_hosted_engine_ha.agent.host<wbr>ed_= engine.HostedEngine.config:<wbr>:(_get_vm_conf_content_from_ov<wbr>f_store)= Got vm.conf from OVF_STORE</div><div>MainThread::INFO::2018-01-12 22:01:52= ,189::hosted_engine::6<wbr>04::ovirt_hosted_engine_ha.age<wbr>nt.hosted_eng= ine.HostedEngine:<wbr>:(_initialize_vdsm) Initializing VDSM</div><div>MainT= hread::INFO::2018-01-12 22:01:54,586::hosted_engine::6<wbr>30::ovirt_hosted= _engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_initialize_storage= _images) Connecting the storage</div><div>MainThread::INFO::2018-01-12 22:0= 1:54,587::storage_server::<wbr>220::<a href=3D"http://ovirt_hosted_engine_h= a.li" target=3D"_blank">ovirt_hosted_engine_ha.li</a><wbr>b.storage_server.= StorageServer<wbr>::(validate_storage_server) Validating storage server</di= v><div>MainThread::INFO::2018-01-12 22:01:56,903::hosted_engine::6<wbr>39::= ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_initia= lize_storage_images) Storage domain reported as valid and reconnect is not = forced.</div><div>MainThread::INFO::2018-01-12 22:01:59,299::states::682::o= vi<wbr>rt_hosted_engine_ha.agent.host<wbr>ed_engine.HostedEngine::(score<wb= r>) Score is 0 due to unexpected vm shutdown at Fri Jan 12 21:57:48 2018</d= iv><div>MainThread::INFO::2018-01-12 22:01:59,299::hosted_engine::4<wbr>53:= :ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(start_= monitoring) Current state EngineUnexpectedlyDown (score: 0)</div><div>MainT= hread::INFO::2018-01-12 22:02:09,659::config::493::ovi<wbr>rt_hosted_engine= _ha.agent.host<wbr>ed_engine.HostedEngine.config:<wbr>:(refresh_vm_conf) Re= loading vm.conf from the shared storage domain</div><div>MainThread::INFO::= 2018-01-12 22:02:09,659::config::416::ovi<wbr>rt_hosted_engine_ha.agent.hos= t<wbr>ed_engine.HostedEngine.config:<wbr>:(_get_vm_conf_content_from_ov<wbr= RE</div><div>MainThread::INFO::2018-01-12 22:02:09,660::ovf_store::132::<wb= r>ovirt_hosted_engine_ha.lib.ovf<wbr>.ovf_store.OVFStore::(getEngin<wbr>eVM= OVF) Extracting Engine VM OVF from the OVF_STORE</div><div>MainThread::INFO= ::2018-01-12 22:02:09,663::ovf_store::134::<wbr>ovirt_hosted_engine_ha.lib.= ovf<wbr>.ovf_store.OVFStore::(getEngin<wbr>eVMOVF) OVF_STORE volume path: /= var/run/vdsm/storage/4a7f8717<wbr>-9bb0-4d80-8016-498fa4b88162/5<wbr>cabd8e= 1-5f4b-469e-becc-227469e<wbr>03f5c/8048cbd7-77e2-4805-9af4-<wbr>d109fa36dfc= f=C2=A0</div><div>MainThread::INFO::2018-01-12 22:02:09,683::config::435::o= vi<wbr>rt_hosted_engine_ha.agent.host<wbr>ed_engine.HostedEngine.config:<wb= r>:(_get_vm_conf_content_from_ov<wbr>f_store) Found an OVF for HE VM, tryin= g to convert</div><div>MainThread::INFO::2018-01-12 22:02:09,688::config::4= 40::ovi<wbr>rt_hosted_engine_ha.agent.host<wbr>ed_engine.HostedEngine.confi= g:<wbr>:(_get_vm_conf_content_from_ov<wbr>f_store) Got vm.conf from OVF_STO= RE</div><div>MainThread::INFO::2018-01-12 22:02:09,698::hosted_engine::6<wb= r>04::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_= initialize_vdsm) Initializing VDSM</div><div>MainThread::INFO::2018-01-12 2= 2:02:12,112::hosted_engine::6<wbr>30::ovirt_hosted_engine_ha.age<wbr>nt.hos= ted_engine.HostedEngine:<wbr>:(_initialize_storage_images) Connecting the s= torage</div><div>MainThread::INFO::2018-01-12 22:02:12,113::storage_server:= :<wbr>220::<a href=3D"http://ovirt_hosted_engine_ha.li" target=3D"_blank">o= virt_hosted_engine_ha.li</a><wbr>b.storage_server.StorageServer<wbr>::(vali= date_storage_server) Validating storage server</div><div>MainThread::INFO::= 2018-01-12 22:02:14,444::hosted_engine::6<wbr>39::ovirt_hosted_engine_ha.ag= e<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_initialize_storage_images) Stor= age domain reported as valid and reconnect is not forced.</div><div>MainThr= ead::INFO::2018-01-12 22:02:16,859::states::682::ovi<wbr>rt_hosted_engine_h= a.agent.host<wbr>ed_engine.HostedEngine::(score<wbr>) Score is 0 due to une= xpected vm shutdown at Fri Jan 12 21:57:47 2018</div><div>MainThread::INFO:= :2018-01-12 22:02:16,859::hosted_engine::4<wbr>53::ovirt_hosted_engine_ha.a= ge<wbr>nt.hosted_engine.HostedEngine:<wbr>:(start_monitoring) Current state= EngineUnexpectedlyDown (score: 0)</div><div>MainThread::INFO::2018-01-12 2= 2:02:27,100::config::493::ovi<wbr>rt_hosted_engine_ha.agent.host<wbr>ed_eng= ine.HostedEngine.config:<wbr>:(refresh_vm_conf) Reloading vm.conf from the = shared storage domain</div><div>MainThread::INFO::2018-01-12 22:02:27,100::= config::416::ovi<wbr>rt_hosted_engine_ha.agent.host<wbr>ed_engine.HostedEng= ine.config:<wbr>:(_get_vm_conf_content_from_ov<wbr>f_store) Trying to get a= fresher copy of vm configuration from the OVF_STORE</div><div>MainThread::= INFO::2018-01-12 22:02:27,101::ovf_store::132::<wbr>ovirt_hosted_engine_ha.= lib.ovf<wbr>.ovf_store.OVFStore::(getEngin<wbr>eVMOVF) Extracting Engine VM= OVF from the OVF_STORE</div><div>MainThread::INFO::2018-01-12 22:02:27,103= ::ovf_store::134::<wbr>ovirt_hosted_engine_ha.lib.ovf<wbr>.ovf_store.OVFSto= re::(getEngin<wbr>eVMOVF) OVF_STORE volume path: /var/run/vdsm/storage/4a7f= 8717<wbr>-9bb0-4d80-8016-498fa4b88162/5<wbr>cabd8e1-5f4b-469e-becc-227469e<= wbr>03f5c/8048cbd7-77e2-4805-9af4-<wbr>d109fa36dfcf=C2=A0</div><div>MainThr= ead::INFO::2018-01-12 22:02:27,125::config::435::ovi<wbr>rt_hosted_engine_h= a.agent.host<wbr>ed_engine.HostedEngine.config:<wbr>:(_get_vm_conf_content_= from_ov<wbr>f_store) Found an OVF for HE VM, trying to convert</div><div>Ma= inThread::INFO::2018-01-12 22:02:27,129::config::440::ovi<wbr>rt_hosted_eng= ine_ha.agent.host<wbr>ed_engine.HostedEngine.config:<wbr>:(_get_vm_conf_con= tent_from_ov<wbr>f_store) Got vm.conf from OVF_STORE</div><div>MainThread::= INFO::2018-01-12 22:02:27,130::states::667::ovi<wbr>rt_hosted_engine_ha.age= nt.host<wbr>ed_engine.HostedEngine::(consu<wbr>me) Engine down, local host = does not have best score</div><div>MainThread::INFO::2018-01-12 22:02:27,13= 9::hosted_engine::6<wbr>04::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine= HostedEngine:<wbr>:(_initialize_vdsm) Initializing VDSM</div><div>MainThre= ad::INFO::2018-01-12 22:02:29,584::hosted_engine::6<wbr>30::ovirt_hosted_en= gine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_initialize_storage_im= ages) Connecting the storage</div><div>MainThread::INFO::2018-01-12 22:02:2= 9,586::storage_server::<wbr>220::<a href=3D"http://ovirt_hosted_engine_ha.l= i" target=3D"_blank">ovirt_hosted_engine_ha.li</a><wbr>b.storage_server.Sto= rageServer<wbr>::(validate_storage_server) Validating storage server<br><br= v><div>Artem</div><div><br><div class=3D"gmail_extra"><br><div class=3D"gma= il_quote">On Fri, Jan 12, 2018 at 7:08 PM, Artem Tambovskiy <span dir=3D"lt= r">&lt;<a href=3D"mailto:artem.tambovskiy@gmail.com" target=3D"_blank">arte= m.tambovskiy(a)gmail.com&lt;/a&gt;&amp;gt;&lt;/span&gt; wrote:<br><blockquote class=3D"gmail_= quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,= 204);padding-left:1ex"><div dir=3D"ltr">Trying to fix one thing I broke ano= ther :(=C2=A0<br><br>I fixed mnt_options for hosted engine storage domain a= nd installed latest security patches to my hosts and hosted engine. All VM&= #39;s up and running, but =C2=A0hosted_engine --vm-status reports about iss= ues:=C2=A0<br><br><div>[root@ovirt1 ~]# hosted-engine --vm-status</div><div= r></div><div>conf_on_shared_storage =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 : True</div><div>Status up-to-date =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0: False</div><div>Hostname =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : ovi= rt2</div><div>Host ID =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 1</div><div>Engine status = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0: unknown stale-data</div><div>Score =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 0</d= iv><div>stopped =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: False</div><div>Local maintenance = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: False</div>= <div>crc32 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 193164b8</div><div>local_conf_ti= mestamp =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 8350</div><div>H= ost timestamp =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 : 8350</div><div>Extra metadata (valid at timestamp):</div><div>= =C2=A0 =C2=A0 =C2=A0 =C2=A0 metadata_parse_version=3D1</div><div>=C2=A0 =C2= =A0 =C2=A0 =C2=A0 metadata_feature_version=3D1</div><div>=C2=A0 =C2=A0 =C2= =A0 =C2=A0 timestamp=3D8350 (Fri Jan 12 19:03:54 2018)</div><div>=C2=A0 =C2= =A0 =C2=A0 =C2=A0 host-id=3D1</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 score= =3D0</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 vm_conf_refresh_time=3D8350 (Fri= Jan 12 19:03:54 2018)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 conf_on_shared= _storage=3DTrue</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 maintenance=3DFalse</= div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 state=3DEngineUnexpectedlyDown</div><d= iv>=C2=A0 =C2=A0 =C2=A0 =C2=A0 stopped=3DFalse</div><div>=C2=A0 =C2=A0 =C2= =A0 =C2=A0 timeout=3DThu Jan =C2=A01 05:24:43 1970</div><div><br></div><div= nf_on_shared_storage =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : True</div>= <div>Status up-to-date =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0: False</div><div>Hostname =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : <a href=3D"http://ov= irt1.telia.ru" target=3D"_blank">ovirt1.telia.ru</a></div><div>Host ID =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0: 2</div><div>Engine status =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: unknown stale-data</div>= <div>Score =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 0</div><div>stopped =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0: True</div><div>Local maintenance =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: False</div><div>crc32 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0: c7037c03</div><div>local_conf_timestamp =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 7530</div><div>Host timestamp =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 7530</div>= <div>Extra metadata (valid at timestamp):</div><div>=C2=A0 =C2=A0 =C2=A0 = =C2=A0 metadata_parse_version=3D1</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 met= adata_feature_version=3D1</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 timestamp= =3D7530 (Fri Jan 12 16:10:12 2018)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 ho= st-id=3D2</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 score=3D0</div><div>=C2=A0 = =C2=A0 =C2=A0 =C2=A0 vm_conf_refresh_time=3D7530 (Fri Jan 12 16:10:12 2018)= </div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 conf_on_shared_storage=3DTrue</div><= div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 maintenance=3DFalse</div><div>=C2=A0 =C2=A0= =C2=A0 =C2=A0 state=3DAgentStopped</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 s= topped=3DTrue</div><div>[root@ovirt1 ~]#=C2=A0</div><div><br></div><div><br= :<br><br><br><div>[root@ovirt2 ~]# hosted-engine --vm-status</div><div><br>= </div><div><br></div><div>--=3D=3D Host 1 status =3D=3D--</div><div><br></d= iv><div>conf_on_shared_storage =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : = True</div><div>Status up-to-date =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0: True</div><div>Hostname =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : ovirt2</di= v><div>Host ID =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 1</div><div>Engine status =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: {&qu= ot;reason&quot;: &quot;vm not running on this host&quot;, &quot;health&quot= ;: &quot;bad&quot;, &quot;vm&quot;: &quot;down&quot;, &quot;detail&quot;: &= quot;unknown&quot;}</div><div>Score =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 0</div>= <div>stopped =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: False</div><div>Local maintenance =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: False</div><di= v>crc32 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 78eabdb6</div><div>local_conf_times= tamp =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 8403</div><div>Host= timestamp =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 : 8402</div><div>Extra metadata (valid at timestamp):</div><div>=C2= =A0 =C2=A0 =C2=A0 =C2=A0 metadata_parse_version=3D1</div><div>=C2=A0 =C2=A0= =C2=A0 =C2=A0 metadata_feature_version=3D1</div><div>=C2=A0 =C2=A0 =C2=A0 = =C2=A0 timestamp=3D8402 (Fri Jan 12 19:04:47 2018)</div><div>=C2=A0 =C2=A0 = =C2=A0 =C2=A0 host-id=3D1</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 score=3D0</= div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 vm_conf_refresh_time=3D8403 (Fri Jan 1= 2 19:04:47 2018)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 conf_on_shared_stora= ge=3DTrue</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 maintenance=3DFalse</div><d= iv>=C2=A0 =C2=A0 =C2=A0 =C2=A0 state=3DEngineUnexpectedlyDown</div><div>=C2= =A0 =C2=A0 =C2=A0 =C2=A0 stopped=3DFalse</div><div>=C2=A0 =C2=A0 =C2=A0 =C2= =A0 timeout=3DThu Jan =C2=A01 05:24:43 1970</div><div><br></div><div><br></= div><div>--=3D=3D Host 2 status =3D=3D--</div><div><br></div><div>conf_on_s= hared_storage =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : True</div><div>St= atus up-to-date =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0: False</div><div>Hostname =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : <a href=3D"http://ovirt1.te= lia.ru" target=3D"_blank">ovirt1.telia.ru</a></div><div>Host ID =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0: 2</div><div>Engine status =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: unknown stale-data</div><div>Sc= ore =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 0</div><div>stopped =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0: True</div><div>Local maintenance =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0: False</div><div>crc32 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0: c7037c03</div><div>local_conf_timestamp =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 7530</div><div>Host timestamp =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 7530</div><div>Ex= tra metadata (valid at timestamp):</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 me= tadata_parse_version=3D1</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 metadata_fea= ture_version=3D1</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 timestamp=3D7530 (Fr= i Jan 12 16:10:12 2018)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 host-id=3D2</= div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 score=3D0</div><div>=C2=A0 =C2=A0 =C2= =A0 =C2=A0 vm_conf_refresh_time=3D7530 (Fri Jan 12 16:10:12 2018)</div><div= =C2=A0 =C2=A0 =C2=A0 maintenance=3DFalse</div><div>=C2=A0 =C2=A0 =C2=A0 = =C2=A0 state=3DAgentStopped</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 stopped= =3DTrue</div></div><div><br></div><div><br></div><div>WebGUI shows that eng= ine running on host ovirt1.=C2=A0<br>Gluster looks fine=C2=A0<br><div>[root= @ovirt1 ~]# gluster volume status engine</div><div>Status of volume: engine= </div><div>Gluster process =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 TCP Port =C2=A0RDMA Port = =C2=A0Online =C2=A0Pid</div><div>------------------------------<wbr>-------= -----------------------<wbr>------------------</div><div>Brick ovirt1.telia= ru:/oVirt/engine =C2=A0 =C2=A0 =C2=A0 =C2=A0 49169 =C2=A0 =C2=A0 0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0Y =C2=A0 =C2=A0 =C2=A0 3244=C2=A0</div><div>Bric= k ovirt2.telia.ru:/oVirt/engine =C2=A0 =C2=A0 =C2=A0 =C2=A0 49179 =C2=A0 = =C2=A0 0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Y =C2=A0 =C2=A0 =C2=A0 20372</di= v><div>Brick ovirt3.telia.ru:/oVirt/engine =C2=A0 =C2=A0 =C2=A0 =C2=A0 4920= 6 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Y =C2=A0 =C2=A0 =C2=A0 = 16609</div><div>Self-heal Daemon on localhost =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 N/A =C2=A0 =C2=A0 =C2=A0 N/A =C2=A0 =C2=A0 =C2=A0 =C2= =A0Y =C2=A0 =C2=A0 =C2=A0 117868</div><div>Self-heal Daemon on <a href=3D"h= ttp://ovirt2.telia.ru" target=3D"_blank">ovirt2.telia.ru</a> =C2=A0 =C2=A0 = =C2=A0 =C2=A0 N/A =C2=A0 =C2=A0 =C2=A0 N/A =C2=A0 =C2=A0 =C2=A0 =C2=A0Y =C2= =A0 =C2=A0 =C2=A0 20521</div><div>Self-heal Daemon on ovirt3 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0N/A =C2=A0 =C2=A0 =C2=A0 N/= A =C2=A0 =C2=A0 =C2=A0 =C2=A0Y =C2=A0 =C2=A0 =C2=A0 25093</div><div>=C2=A0<= /div><div>Task Status of Volume engine</div><div>--------------------------= ----<wbr>------------------------------<wbr>------------------</div><div>Th= ere are no active volume tasks<br><br>How to resolve this issue?</div><br><= /div></div<br>______________________________<wbr>_________________<brUsers mailing list<br<a href=3D"mailto:Users@ovirt.org" target=3D&quot;_blank&quot;&gt;Users(a)ovirt.org&lt;/a&gt;&lt;br= <a href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer= " target=3D"_blank">http://lists.ovirt.org/mailman<wbr>/... br<br></blockquote></div><br></div></div></div<br>______________________________<wbr>_________________<brUsers mailing list<br<a href=3D"mailto:Users@ovirt.org" target=3D&quot;_blank&quot;&gt;Users(a)ovirt.org&lt;/a&gt;&lt;br= <a href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer= " target=3D"_blank">http://lists.ovirt.org/mailman<wbr>/... br<br></blockquote></div><br></div</div></div></blockquote></div><br></div</div></div></blockquote></div><br></div</div></div></blockquote></div><br></div></div _______________________________________________<brUsers mailing list<br<a class=3D"aqm-autolink aqm-autowrap" href=3D"mailto:Users%40ovirt.org">Us= ers(a)ovirt.org&lt;/a&gt;&lt;br<a class=3D"aqm-autolink aqm-autowrap" href=3D"http://lists.ovirt.org/mailm= an/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users&...<br</blockquote</div</div</body</html ------------160fee5d54b636727eac37286c--